JP7609302B2 - Action interval evaluation program, device, and method - Google Patents

Description

The disclosed technology relates to an action interval evaluation program, an action interval evaluation device, and an action interval evaluation method.

Advances in deep learning technology have made it possible to recognize human posture and other aspects with a high degree of accuracy from images captured by ordinary RGB cameras, and various research and development efforts are being conducted using this recognition information to estimate human behavior, such as specific tasks.

For example, a work analysis system has been proposed that estimates a worker's work based on work information and calculates a reliability indicating the accuracy of the estimated work. This system calculates the start and end times of each estimated work based on time information. This system also divides the acquired work video by the start and end times of the estimated work, and links the section video from the start to end times of the estimated work to the estimated work and the reliability of the work.

In addition, for example, a work analysis device has been proposed in which a first monitoring area for detecting the start of a unit task and a second monitoring area for detecting the completion of the unit task are set for each unit task within the imaging range of the imaging unit. In this device, a reliability is set for each monitoring area such that the reliability increases as the likelihood that an action related to the unit task is determined to be performed in the monitoring area increases.

JP 2020-91801 A JP 2020-166471 A

We consider estimating the time interval in the video in which a specified action occurred based on time-series features such as a person's posture extracted from the video, and further segmenting this time interval into time intervals of component actions contained in the specified action. In this case, even if the boundaries between the component actions deviate from the actual boundaries, the reliability of the segmentation of the component actions may be evaluated as high.

In one aspect, the disclosed technology aims to accurately evaluate the reliability of specified action segments in human video.

In one aspect, the disclosed technology obtains an estimation result of segmentation of an elemental action section, which is a time section of the elemental action, based on time series feature amounts extracted from a video of a human action and an estimation model including a probability model for each of a plurality of elemental actions included in the action. The disclosed technology also calculates the conformance of the time series feature amounts corresponding to each of the elemental action sections before and after the boundary when the boundary between adjacent elemental action sections is changed, to the probability model corresponding to the elemental action. The disclosed technology then evaluates the estimation result of the segmentation of the elemental action section based on the change in the conformance according to the degree of change in the boundary.

One aspect is that it has the effect of being able to accurately evaluate the reliability of specified action segments in human video.

FIG. 13 is a diagram for explaining a comparative example of the present embodiment. FIG. 11 is a diagram for explaining another comparative example of the present embodiment. FIG. 11 is a diagram for explaining another comparative example of the present embodiment. FIG. 13 is a diagram for explaining a problem in the comparative example. FIG. 2 is a functional block diagram of an action interval evaluation device. FIG. 1 is a conceptual diagram of a hidden semi-Markov model, which is an example of an estimation model. FIG. 2 is a conceptual diagram illustrating a state of a first hidden Markov model. 13 is a diagram for explaining a change in the degree of conformance of the feature amounts of element action sections before and after a boundary to a probabilistic model when the boundary is shifted. FIG. 11 is a graph showing an example of the relationship between time t and the degree of conformance E _j (t) of the boundary j. 11 is a graph showing an example of the relationship between time t and the degree of conformance E _j (t) of the boundary j. FIG. 13 is a diagram for explaining the maximum value of the time width. FIG. 2 is a block diagram showing a schematic configuration of a computer functioning as an action interval evaluation device. 1 is a flowchart illustrating an example of a machine learning process. 13 is a flowchart illustrating an example of a segmentation process.

Below, an example of an embodiment of the disclosed technology is described with reference to the drawings.

The action interval evaluation device according to this embodiment segments the time interval during which the target action detected from the video occurred (hereinafter referred to as "action interval") into component actions based on time series features of human actions extracted from the video of the person. That is, the action interval evaluation device estimates the segmentation of the time intervals of the component actions within the action interval (hereinafter referred to as "component action intervals"). In this embodiment, for example, the action of a person manufacturing a product is taken as the target action, and the combination of the person's actions when performing each manufacturing process is taken as the component actions. That is, the target action is an action that includes multiple component actions with restrictions on the order in which they occur, such as work in a factory that includes multiple processes in a fixed order. Furthermore, the action interval evaluation device evaluates the reliability of the segmentation of the estimated component action intervals.

As a comparative example of this embodiment, a method of manually segmenting video into sections to identify element action sections from video is considered. This comparative example is a method of acquiring video of a series of tasks captured by a camera as shown in the left diagram of FIG. 1, and manually segmenting the acquired video into element action sections by visually inspecting the acquired video as shown in the right diagram of FIG. 1. In the example of FIG. 1, "fitting part A", "screwing part A", and "attaching the cover" are each examples of element actions. In this way, manually segmenting each acquired video into element action sections takes time and effort.

As another comparative example of this embodiment, as shown in the upper diagram of Figure 2, it is possible to manually segment element action intervals for one video, and then use this segmentation result as teaching information to automatically segment element action intervals for other videos, as shown in the lower diagram of Figure 2. In this case, it is possible to save time and effort compared to manually segmenting element action intervals for all videos.

As in the example of Figure 2, when automatically segmenting element action intervals using teacher information, even if the human actions indicated by the teacher information do not match the human actions appearing in the target video, the boundaries of all element action intervals are output in the same way as the teacher information. The boundaries of element action intervals in time periods that do not match the teacher information are likely to be incorrect, but which element actions are not in accordance with the teacher information cannot be determined from the element action interval segmentation estimation result alone. Therefore, it is desirable to evaluate the reliability of the segmentation result and output it separately from the estimated element action interval segmentation result. This reliability has the property that it may not necessarily match the accuracy of the element action interval segmentation estimation result. For example, even time periods that do not match the teacher information may be correctly segmented from the context. In this case, the certainty of the segmentation estimation result is high, but unlike when it matches the teacher information and the boundaries are clear, its reliability is low.

As another comparative example of this embodiment, it is possible to use, as the reliability, the relative compatibility expressed by the ratio of the likelihood (observation probability) according to the probabilistic model of the estimated element behavior to the likelihood according to the probabilistic models of other element behaviors for the observation information of the estimated element behavior section. It is considered that the greater the relative compatibility, the more likely the estimation result is to be correct. For example, it is possible to calculate the likelihood P(X|M _i ) output by the probabilistic model M _i of each element behavior for the observation information X in element behavior section j of element behavior j, and calculate the reliability C _j of element behavior j by the following formula (1). Note that the observation information is a feature amount such as an image or a person's posture information extracted from the image.
C _j = P (X | M _j )/Σ _i P (X | M _i ) ... (1)

For example, as shown in Fig. 3, the comparative example estimates element action intervals by segmenting time-series feature amounts ( _x1 , _x2 , ..., _x10 ) included in an action interval based on an estimation model that models the duration of each element action interval. Fig. 3 shows an example in which the interval of feature amounts _x1 to _x3 is estimated as the element action interval corresponding to element action A, the interval of feature amounts _x4 to _x8 is estimated as the element action interval corresponding to element action B, and the interval of feature amounts _x9 to _x10 is estimated as the element action interval corresponding to element action C. Then, the comparative example calculates the likelihood of all element action intervals based on the respective probability models for the feature amounts assigned to each element action interval, and calculates the relative compatibility using the above formula (1).

However, in the comparative example shown in FIG. 3, even if the boundaries of the element action sections are shifted, the reliability may be calculated to be high. Specifically, as shown in FIG. 4, in the observation information, a time period with high compatibility with the probability model of element action 1 and a time period with high compatibility with the probability model of element action 2 occur at an interval. In this case, even if there is a shift between the correct boundary between element action section 1 and element action section 2 and the boundary in the estimation result, each element action section includes a time period with high compatibility with the probability model of each element action, so the relative compatibility of each element action section is high. Therefore, as in the example of FIG. 4, when the relative compatibility is used as the reliability, the reliability of the estimation result of the segmentation of the element action section is evaluated as high, even though it is desired to evaluate the reliability as low.

In this embodiment, we propose a method that can deal with partial deviations in the boundaries of element action sections. Below, we will explain the details of the action section evaluation device related to this embodiment.

As shown in FIG. 5, the action interval evaluation device 10 functionally includes an extraction unit 11, a machine learning unit 20, and a segmentation unit 30. The machine learning unit 20 further includes an observation probability learning unit 21, a transition probability learning unit 22, and a construction unit 23. The segmentation unit 30 further includes an estimation unit 31, a calculation unit 32, and an evaluation unit 33. In addition, an estimation model 41 is stored in a specified memory area of the action interval evaluation device 10.

The extraction unit 11 acquires a learning video during machine learning. The learning video is a video of a person's behavior, to which teacher information indicating the division of the elemental behavior interval indicating the time interval of the target behavior and the time interval of each of the elemental behaviors included in the target behavior is added. The extraction unit 11 calculates features related to the person's movements from the video of the behavior interval of the learning video, and extracts time-series features. The extraction unit 11 also acquires a segmentation video during segmentation. The segmentation video is a video of a person's behavior, in which the division of the elemental behavior interval in the target behavior interval is unknown. In this embodiment, the segmentation video itself is a time interval of the behavior interval, or the start time and end time of the behavior interval are added to the segmentation video, so that the behavior interval in the segmentation video is known. The extraction unit 11 also extracts time-series features from the behavior interval of the segmentation video in the same manner.

An example of a method for extracting time-series features from a video by the extraction unit 11 will be specifically described. The extraction unit 11 applies a person detection technique to detect a human region (e.g., a bounding box) from each frame constituting a video (learning video or segmentation video), and tracks the detected regions of the same person by associating them between frames. When multiple human regions are detected from one frame, the extraction unit 11 identifies the region of the person to be judged based on the size of the region, the position of the region within the frame, etc. The extraction unit 11 performs image processing on the image within the human region detected from each frame, and calculates posture information based on the joint positions of the person and their connection relationships, etc. The extraction unit 11 creates time-series posture information by associating the posture information calculated for each frame with the time information associated with the frame and arranging them.

The extraction unit 11 also calculates time-series movement information for each part of the body from the time-series posture information. The movement information may be, for example, the degree of bending of each part, the bending speed, etc. Each part may be, for example, an elbow, a knee, etc. The extraction unit 11 also calculates a feature vector whose elements are values obtained by averaging the movement information within the window in the time direction at regular time intervals using a sliding time window.

During machine learning, the extraction unit 11 passes the extracted time series features and teacher information indicating the divisions of the action intervals and element action intervals in the learning video to the machine learning unit 20 as supervised data, and during segmentation, passes the extracted time series features to the segmentation unit 30.

The machine learning unit 20 generates an estimation model 41 by performing machine learning using the supervised data passed from the extraction unit 11.

In this embodiment, a hidden semi-Markov model (hereinafter referred to as "HSMM (Hidden semi-Markov model)") as shown in FIG. 6 is constructed as an example of an estimation model 41 that estimates the segmentation of element action intervals in a target action interval. In addition to the parameters of a hidden Markov model (hereinafter referred to as "HMM (Hidden Markov model)"), the HSMM has a probability distribution of the duration of each state as a parameter.

The HSMM of this embodiment includes a plurality of first HMMs that represent each of the human actions as a state, and a second HMM that represents elemental actions as states. In FIG. 6, m1, m2, and m3 are states corresponding to each action, and a1, a2, and a3 are states corresponding to each elemental action. An elemental action is a combination of a plurality of actions, and an action is a combination of a plurality of postures. When a time series of features related to a human action extracted from a video is given to the HSMM constructed by setting parameters, the HSMM estimates the optimal segmentation of elemental action sections. In FIG. 6, d1, d2, and d3 are examples of elemental action sections.

The parameters of the HMM include observation probability and transition probability. In FIG. 6, O1, O2, ..., O8 are examples of observation probability, and the transition probability is associated with the arrows connecting the states. The observation probability is the probability that certain observation information is observed in each state, and the transition probability is the probability of transitioning from one state to another. If the order of transitions is fixed, the transition probability is not necessary. Note that the number of actions and the number of element actions, i.e., the number of first HMMs and second HMMs, are examples and are not limited to the numbers exemplified in FIG. 6. Below, the observation probability learning unit 21, the transition probability learning unit 22, and the construction unit 23 included in the machine learning unit 20 will be described in detail.

As described below, the observation probability learning unit 21 learns the observation probability of each action that constitutes the HSMM, which is an example of the estimation model 41, using time series features (hereinafter also referred to as "unsupervised data") obtained by removing the supervised information from the supervised data.

In this embodiment, the target behavior is a limited behavior for achieving a certain task goal. For example, such a behavior is a behavior in a routine task performed on a factory line, and has the following characteristics.
Property 1: The difference between each of the component actions that make up a behavior is the difference in the combination of a limited number of actions.
Property 2: Multiple postures observed when performing the same action are similar.

In this embodiment, based on property 1, all component actions are composed of actions included in one action group. For example, as shown in FIG. 7, the action group includes three actions m11, m12, and m13. For example, action m11 may be "raising the arm", action m12 may be "lowering the arm", and action m13 may be "stretching the arm forward". The number of actions included in the action group is not limited to the example in FIG. 7. Furthermore, the number of actions included in each component action is not limited to the example in FIG. 7.

For example, the observation probability learning unit 21 calculates the observation probability of each action using a Gaussian mixture model (hereinafter referred to as "GMM (Gaussian Mixture Model)"). Specifically, the observation probability learning unit 21 estimates the parameters of a GMM that is a mixture of Gaussian distributions for the number of actions, by clustering the features passed from the extraction unit 11. Then, the observation probability learning unit 21 assigns each Gaussian distribution that constitutes the GMM with estimated parameters as a probability distribution representing the observation probability of each action.

The transition probability learning unit 22 calculates the transition probability between actions represented by the first HMM based on the supervised data, as described below. Specifically, the transition probability learning unit 22 allocates the time series feature amounts to each element action interval based on the supervised information contained in the supervised data. The transition probability learning unit 22 then uses the time series feature amounts allocated to each element action interval as observation information, fixes the observation probability of each action calculated by the observation probability learning unit 21, and calculates the transition probability between actions using, for example, maximum likelihood estimation or an Expectation-Maximization algorithm (EM algorithm).

In addition, since creating supervised data takes time and effort, the transition probability learning unit 22 may pad the supervised data by adding noise to the seed supervised data, for example.

The construction unit 23 sets a probability distribution of duration for each element action based on the duration of each element action section given in the teacher information. For example, the construction unit 23 sets a uniform distribution within a predetermined range for the duration of each element action section given in the teacher information as the probability distribution of the duration of the element action.

The construction unit 23 constructs an HSMM as shown in FIG. 6 as an estimation model 41 using the observation probability of each action calculated by the observation probability learning unit 21, the transition probability between actions calculated by the transition probability learning unit 22, and the duration of each set element action. The estimation model 41 is an HSMM in which the second HMM corresponding to each element action transitions in the order of each element action given by the teacher information after the set duration. In FIG. 6, O1, O2, ..., O8 represent the observation probability calculated by the observation probability learning unit 21. In addition, the transition probability associated with the arrow between the actions m1, m2, and m3 included in each of the element actions a1, a2, and a3 corresponds to the transition probability calculated by the transition probability learning unit 22. In addition, d1, d2, and d3 represent the duration of each element action. The construction unit 23 stores the constructed estimation model 41 in a predetermined storage area.

The segmentation unit 30 estimates the segmentation of element action sections in the target action section of the segmentation video based on the time-series features transferred from the extraction unit 11, and evaluates the estimation result of the segmentation of the element action sections. Each of the estimation unit 31, calculation unit 32, and evaluation unit 33 included in the segmentation unit 30 will be described in detail below.

The estimation unit 31 estimates the segmentation of element action sections in an action section by inputting time series feature quantities corresponding to the action sections of the segmentation video to the estimation model 41. The estimation unit 31 passes the estimation result of the segmentation of the element action sections to the calculation unit 32.

The calculation unit 32 calculates the fit of the time series features corresponding to each of the element action intervals before and after the boundary when the boundary between adjacent element action intervals is changed to the second HMM, which is a probabilistic model corresponding to the corresponding element action.

Specifically, the calculation unit 32 calculates the degree of conformance as shown below, using boundary j of an element action in the estimation result of segmentation of the element action section as a reference, while shifting the time corresponding to boundary j. The calculation unit 32 calculates the degree of conformance P(Xj _{, Xj} +1 | _Mj , _Mj+1 ) for a time series feature amount { _Xj , Xj _{+ 1} } which is observation information of a section combining two element action sections before and after boundary j. P( _Xj , Xj ₊₁ | _Mj , Mj ₊₁ )=P( _Xj | _Mj )P(Xj ₊₁ |Mj ₊₁ ), where _Mj and Mj ₊₁ are probabilistic models (second HMMs) of element actions before and after boundary j. The calculation unit 32 expresses P( _Xj , _Xj+1 | _Mj , _Mj+1 ) as a function _Ej (t) of time t corresponding to the boundary j.

For example, as shown in FIG. 8, the feature amount at time t is x _t (in the example of FIG. 8, t=1, 2, ..., 8). In addition, when the time T _j corresponding to the boundary j in the estimation result is t, the feature amount up to x _t is set as the observation information of the j-th element behavior. When _{T j} = 4, the calculation unit 32 inputs the feature amounts x ₁ to x ₄ into the probability model M _j to calculate P (X _j | M _j ), and inputs the feature amounts x ₅ to x ₈ into the probability model M _j+1 to calculate P (X _j+1 | M _j+1 ). Then, the calculation unit 32 calculates E _j (T _j ) from P (X _j | M _j ) and P (X _j+1 | M _j+1 ). Similarly, the calculation unit 32 calculates E _j (t) when the boundary j is shifted to each time. 8 is an example in which the boundary j is time T _j -1, which is one time before time T _j , and the goodness of fit in this case is E _j (T _j -1). The bottom diagram in the left diagram of Fig. 8 is an example in which the boundary j is time T _j +1, which is one time after time T _j , and the goodness of fit in this case is E _j (T _j +1). The calculation unit 32 passes the calculated goodness of fit function E _j (t) to the evaluation unit 33.

Here, the segmentation of the element action intervals should be estimated so that E _j (t) is maximum at the time T _j corresponding to the boundary in the estimated result of the segmentation of the element action intervals. Therefore, as shown in the right diagram of FIG. 8, when the time corresponding to the boundary j is shifted, E _j (t) should decrease, and the larger the time width of the shift, the greater the decrease in E _j (t) should be. When the time corresponding to the boundary j is clear from the feature amount, that is, when the possibility of other boundaries is almost unthinkable, as shown in FIG. 9, when the time corresponding to the boundary j is changed from T _j , E _j (t) drops sharply. On the other hand, when the boundary is unclear, as shown in FIG. 10, even if the time corresponding to the boundary j is changed from T _j , E _j (t) does not drop significantly immediately, and a certain high value continues. In other words, when the value of E _j (t) continues to be high, it can be said that the reliability of the estimated result of the segmentation of the element action intervals is low.

Therefore, the evaluation unit 33 evaluates the reliability of the estimation result of the segmentation of the element action section based on the change in the degree of conformance according to the degree of boundary change. Specifically, when the boundary is changed based on the time _Tj corresponding to the boundary j indicated by the estimation result, the evaluation unit 33 calculates an evaluation value that decreases as the maximum value of the time width in which the change in conformance falls within a predetermined range increases.

More specifically, the evaluation unit 33 calculates the maximum time for which the logarithm of the goodness of fit falls within a certain range when the boundary _j is changed, based on the logarithm log(E _j (T _j )) of the goodness of fit for time T j corresponding to the boundary j indicated by the estimation result. For example, as shown in FIG 11, the evaluation unit 33 sets the maximum value of ||t-T _j || to Δt _j within the range satisfying log(E _j (t))>log(E _j (T _j ))-C (C is a constant determined experimentally), and sets t at that time to T _j '.

Furthermore, the evaluation unit 33 calculates the reliability of the element action section j when focusing on the boundary j so that the reliability decreases as Δt _j becomes larger relative to the length of the element action section j. For example, the evaluation unit 33 sets the length L _j of the estimated time section of the element action section j as L _j =T _j -T _j -1. Then, the evaluation unit 33 calculates the reliability as L _j /(L j +Δt j ) when the length of the element action section j becomes longer when the time corresponding to the boundary _j is changed from T _j to T _j ', that is, when T _j ≦T _j '. On the other hand, the evaluation unit 33 calculates the reliability as (L _j -Δt _j )/L _j when the length of the element action section j becomes shorter when the time corresponding to the boundary j is changed from T _j to T _{j '} , that is, when T _j >T _j '.

Focusing on one element action section, the reliability is calculated for each of the two boundaries, the start and end of the section. The evaluation unit 33 integrates the two reliability to calculate the final reliability of the element action section. The two reliability may be integrated by selecting the minimum value of the two reliability, or by averaging the two reliability. For the first element action section of the action section, since there is no element action section prior to it, the evaluation unit 33 adopts the reliability calculated for the boundary at the end of the section. For the last element action section of the action section, since there is no element action section subsequent to it, the evaluation unit 33 adopts the reliability calculated for the boundary at the start of the section. The evaluation unit 33 assigns the reliability calculated for each element action section to the estimation result of the segmentation of the element action section, and outputs it as an estimated result with evaluation.

The action interval evaluation device 10 may be realized, for example, by a computer 50 shown in FIG. 12. The computer 50 includes a CPU (Central Processing Unit) 51, a memory 52 as a temporary storage area, and a non-volatile storage unit 53. The computer 50 also includes an input/output device 54 such as an input unit and a display unit, and an R/W (Read/Write) unit 55 that controls the reading and writing of data from and to a storage medium 59. The computer 50 also includes a communication I/F (Interface) 56 that is connected to a network such as the Internet. The CPU 51, memory 52, storage unit 53, input/output device 54, R/W unit 55, and communication I/F 56 are connected to each other via a bus 57.

The memory unit 53 may be realized by a hard disk drive (HDD), a solid state drive (SSD), a flash memory, etc. The memory unit 53 as a storage medium stores an action interval evaluation program 60 for causing the computer 50 to function as the action interval evaluation device 10. The action interval evaluation program 60 has an extraction process 61, a machine learning process 70, and a segmentation process 80. The memory unit 53 also has an information storage area 90 in which information constituting the estimation model 41 is stored.

The CPU 51 reads out the action section evaluation program 60 from the storage unit 53, expands it in the memory 52, and sequentially executes the processes of the action section evaluation program 60. The CPU 51 operates as the extraction unit 11 shown in FIG. 5 by executing the extraction process 61. The CPU 51 also operates as the machine learning unit 20 shown in FIG. 5 by executing the machine learning process 70. The CPU 51 also operates as the segmentation unit 30 shown in FIG. 5 by executing the segmentation process 80. The CPU 51 also reads out information from the information storage area 90 and expands the estimation model 41 in the memory 52. As a result, the computer 50 that has executed the action section evaluation program 60 functions as the action section evaluation device 10. The CPU 51 that executes the program is hardware.

In addition, the functions realized by the action interval evaluation program 60 can also be realized, for example, by a semiconductor integrated circuit, more specifically, an ASIC (Application Specific Integrated Circuit).

Next, the operation of the action section evaluation device 10 according to this embodiment will be described. When learning video is input to the action section evaluation device 10 and an instruction is given to execute machine learning of the estimation model 41, the action section evaluation device 10 executes the machine learning process shown in FIG. 13. When segmentation video is input to the action section evaluation device 10 and an instruction is given to execute segmentation of the target action section, the action section evaluation device 10 executes the segmentation process shown in FIG. 14. Note that the machine learning process and the segmentation process are examples of the action section evaluation method of the disclosed technology.

First, we will explain the machine learning processing shown in Figure 13.

In step S11, the extraction unit 11 acquires the learning video input to the behavior section evaluation device 10 and extracts time series features related to human movements from the video of the behavior section of the learning video.

Next, in step S12, the observation probability learning unit 21 estimates the parameters of a GMM that contains a mixture of Gaussian distributions for the number of movements by clustering the features extracted in step S11. Then, the observation probability learning unit 21 assigns each Gaussian distribution constituting the GMM whose parameters have been estimated as a probability distribution representing the observation probability of each movement.

Next, in step S13, the transition probability learning unit 22 distributes the time series feature values extracted in step S11 to each element action interval indicated by the teacher information of the supervised data. Next, in step S14, the transition probability learning unit 22 uses the time series feature values distributed to each element action interval as observed data, fixes the observation probability of each action calculated in step S12, and calculates the transition probability between actions.

Next, in step S15, the construction unit 23 sets a probability distribution of duration for each element action based on the duration of each element action interval given in the teacher information. Next, in step S16, the construction unit 23 constructs an HSMM as an estimation model 41 using the observation probability of each action calculated in step S12, the transition probability between actions calculated in step S14, and the duration of each element action set in step S15. Then, the construction unit 23 stores the constructed estimation model 41 in a specified storage area, and the machine learning process ends.

Next, we will explain the segmentation process shown in Figure 14.

In step S21, the extraction unit 11 acquires the segmentation video input to the action section evaluation device 10, and extracts time series features related to human movements from the action section of the segmentation video. Next, in step S22, the estimation unit 31 inputs the time series features corresponding to the action section of the segmentation video to the estimation model 41, thereby estimating the segmentation of element action sections in the action section.

Next, in step S23, the calculation unit 32 extracts a boundary j for each element action section j included in the action section. Then, the calculation unit 32 calculates the conformance of the feature amount {X _j , X _j+1 } of the section combining the two element action sections before and after the boundary j to the probabilistic models M _j and M _j+1 of the two element actions as a function E _j (t) when the boundary j is shifted to each time.

Next, in step S24, the evaluation unit 33 calculates the maximum time _Δtj at which the logarithm of the goodness of fit when boundary j is changed falls within a predetermined range, based on the logarithm log( _Ej ( _Tj )) of the goodness of fit for the time _Tj corresponding to boundary j indicated by the estimation result. Next, in step S25, the evaluation unit 33 calculates the reliability of element activity section j so that the reliability decreases as the maximum time _Δtj is longer relative to the length of element activity section j. The above steps S23 to S25 are executed for all boundaries included in the activity section.

Next, in step S26, the evaluation unit 33 integrates the two reliability values calculated for the two boundaries, the start and end of the element action section, to calculate a final reliability value for each element action section. Next, in step S27, the evaluation unit 33 assigns the calculated reliability value for each element action section to the estimation result of the segmentation of the element action section, outputs it as an estimated result with an evaluation, and the segmentation process ends.

As described above, the activity section evaluation device of this embodiment estimates the segmentation of element activity sections based on time series feature values extracted from video of a person's activity and an estimation model including a probability model for each of multiple element activity sections included in the activity. Furthermore, when the boundary between adjacent element activity sections is changed, the activity section evaluation device calculates the conformance of the time series feature values corresponding to each element activity section before and after the boundary to the probability model corresponding to the element activity. The activity section evaluation device then evaluates the estimation result of the segmentation of element activity sections based on the change in conformance according to the degree of change in the boundary. This makes it possible to accurately evaluate the reliability of the segmentation of a specified activity in a person's video. In other words, the activity section evaluation device of this embodiment improves the functionality of a computer.

In the above embodiment, the estimation model is described as HSMM, but this is not limited to this and other machine learning models, such as a model using a neural network, may also be applied.

In addition, in the above embodiment, in addition to the evaluation value based on the change in conformity according to the degree of boundary change, the final reliability may be calculated by taking into account the reliability calculated from the relative conformity of the comparative example described in Figure 3 and taking the minimum value of these.

In the above embodiment, the action section evaluation device estimates the segmentation of element action sections in the action section of the segmentation video, but the estimation result of the segmentation of the element action section may be acquired instead of the segmentation video. In this case, the segmentation unit 30 may include an acquisition unit that acquires the estimation result of the segmentation of the element action section, and the estimation unit 31 may be omitted.

In addition, in the above embodiment, a case has been described in which an action interval evaluation device including a machine learning unit and a segmentation unit is realized by a single computer, but each may also be realized by a separate computer.

In the above embodiment, the behavior interval evaluation program is pre-stored (installed) in the storage unit, but this is not limiting. The program according to the disclosed technology can also be provided in a form stored in a storage medium such as a CD-ROM, DVD-ROM, or USB memory.

REFERENCE SIGNS LIST 10 Action interval evaluation device 11 Extraction unit 20 Machine learning unit 21 Observation probability learning unit 22 Transition probability learning unit 23 Construction unit 30 Segmentation unit 31 Estimation unit 32 Calculation unit 33 Evaluation unit 41 Estimation model 50 Computer 51 CPU
52 Memory 53 Storage unit 54 Input/output device 55 R/W unit 56 Communication I/F
57 Bus 59 Storage medium 60 Action interval evaluation program 61 Extraction process 70 Machine learning process 80 Segmentation process 90 Information storage area

Claims (9)

obtaining an estimation result of segmentation of an element behavior interval, which is a time interval of the element behavior, based on a time series feature amount extracted from a video of a human behavior and an estimation model including a probabilistic model for each of a plurality of element behaviors included in the behavior;
calculating a degree of conformity of the feature quantity of the time series corresponding to each of the element action sections before and after the boundary, when a boundary between adjacent element action sections is changed, to the probabilistic model corresponding to the element action;
evaluating an estimation result of the segmentation of the element action interval based on a change in the degree of the conformance according to the degree of the change in the boundary. The action interval evaluation program according to claim 1, wherein the process of evaluating the estimation result includes evaluating the reliability of the estimation result as being lower the greater the maximum value of the time span in which the change in the degree of conformance falls within a predetermined range when the boundary is changed based on the time that is the boundary indicated by the estimation result. The behavior interval evaluation program according to claim 2, wherein the process of evaluating the estimation result includes evaluating the reliability of the estimation result as being lower the greater the maximum value of the time width relative to the length of the element behavior interval. The activity interval evaluation program according to claim 3, wherein the process of evaluating the estimation result includes calculating a reliability based on the maximum value of the time width relative to the length of the element activity interval, and integrating the reliability calculated when the boundary corresponding to the start time of the element activity interval is changed with the reliability calculated when the boundary corresponding to the end time of the element activity interval is changed to calculate a final reliability. Extracting the time series of features from a video of the person's behavior;
5. The action interval evaluation program according to claim 1, further comprising: estimating a segmentation of the elemental action interval based on the extracted time-series feature amount and the estimation model. The action interval evaluation program according to any one of claims 1 to 5, which causes the computer to execute a process further including generating the estimation model by executing machine learning using as training information time-series features extracted from video of an action interval in which the segmentation of element action intervals is known. the action being an action of a person manufacturing a product;
7. The action interval evaluation program according to claim 1, wherein the element actions are a combination of the actions of the person when performing each process of the manufacturing. an acquisition unit that acquires an estimation result of a segmentation of an element behavior section, which is a time section of the element behavior, based on a time series feature amount extracted from a video of a human behavior and an estimation model including a probabilistic model for each of a plurality of element behaviors included in the behavior;
a calculation unit that calculates a degree of conformity of the feature quantity of the time series corresponding to each of the element action sections before and after a boundary between adjacent element action sections when the boundary is changed, to the probabilistic model corresponding to the element action;
an evaluation unit that evaluates an estimation result of the segmentation of the element action section based on a change in the degree of conformance according to a degree of change in the boundary;
An action interval evaluation device comprising: obtaining an estimation result of segmentation of an element behavior interval, which is a time interval of the element behavior, based on a time series feature amount extracted from a video of a human behavior and an estimation model including a probabilistic model for each of a plurality of element behaviors included in the behavior;
calculating a degree of conformity of the feature quantity of the time series corresponding to each of the element action sections before and after the boundary, when a boundary between adjacent element action sections is changed, to the probabilistic model corresponding to the element action;
evaluating an estimation result of the segmentation of the elemental action interval based on a change in the goodness of fit according to a degree of change in the boundary.