JP7633588B2 - Motion recognition device, motion recognition method, and motion recognition program - Google Patents

Description

The present invention relates to a motion recognition device, a motion recognition method, and a motion recognition program.

Conventionally, cameras and sensors are installed on production lines in factories, etc., and the movements of workers are recognized and evaluated based on the information on the movements of the workers obtained by these cameras and sensors. For example, in the following Patent Document 1, the presence or absence of non-standard work that deviates from standard work is determined based on information obtained by motion capture, etc.

In Patent Document 1, an action to be performed in a certain work process is assumed, and it is determined whether or not that action is present and whether or not there is an abnormality in that action.

WO 2018/131630

However, the actions performed in each work process often differ even if they belong to the same action type. For example, the process of tightening a screw and the process of fitting a case share the common action of moving the hand to the work object and grabbing (grabbing) it, but the grasping motions are different in each process. To be more specific, in the process of tightening a screw, for example, one hand will grab a part placed in a parts box, and the other hand will grab an electric screwdriver suspended by a cable from above and in front of the worker. In contrast, in the process of fitting a case, for example, both hands will grab a case placed in a parts box.

When targeting common actions that differ for each work process, the technology in Patent Document 1 can recognize common actions performed in one of the work processes. However, in order to recognize common actions performed in other work processes, it is necessary to carry out separate design work so that common actions performed in other work processes can be recognized.

Therefore, the present invention provides an action recognition device, an action recognition method, and an action recognition program that can recognize common actions performed in different work processes.

The motion recognition device according to one aspect of the present disclosure includes an acquisition unit that acquires motion information including elapsed time of work, common motions common to different work processes to be recognized, and skeletal data of a worker, a conversion unit that calculates the distance between each coordinate and a starting coordinate corresponding to a starting point of a body part determined as a starting point among the body parts based on the coordinates corresponding to each of a plurality of body parts included in the skeletal data, and converts each coordinate included in the skeletal data into the calculated distance, and a learning unit that learns a model for motion recognition that outputs information indicating a motion of a worker that corresponds to one of the common motions based on the motion information after the coordinates are converted into distances by the conversion unit.

According to this aspect, the distance between the coordinates corresponding to each of the multiple body parts included in the skeletal data and the starting coordinates corresponding to the starting part can be calculated, and each coordinate of the skeletal data can be replaced with the calculated distance. Then, based on the movement information after each coordinate of the skeletal data has been replaced with distance data, a model for action recognition can be trained that outputs information indicating the worker's actions that correspond to any of the common actions.

A motion recognition device according to another aspect of the present disclosure includes an acquisition unit that acquires time-series information regarding a motion of a worker, a conversion unit that calculates, based on coordinates corresponding to each of a plurality of body parts included in skeletal data extracted from the time-series information, a distance between each coordinate and a starting coordinate corresponding to a starting part of the body part that is set as a starting point, and converts each coordinate included in the skeletal data into the calculated distance, and a motion recognition unit that inputs the distance converted by the conversion unit into a trained model and recognizes the motion of the worker based on information indicating a motion of the worker that corresponds to one of the common motions commonly included in different work processes to be recognized, which is output from the trained model.

According to this aspect, the distance between the coordinates corresponding to each of the multiple body parts included in the skeletal data extracted from the time-series information of the worker to be recognized and the starting coordinates corresponding to the starting part can be calculated, and each coordinate of the skeletal data can be replaced with the calculated distance. Then, by inputting the replaced distance data into the trained model, it is possible to recognize the worker's movements that correspond to any of the common movements.

In each of the above aspects, the starting point may be either the neck or the lower back.

This allows the starting point to be determined near the center of the movement of the worker being recognized.

A motion recognition method according to another aspect of the present disclosure is a motion recognition method executed by a processor, and includes acquiring motion information including elapsed time of work, common motions included in common to different work processes to be recognized, and skeletal data of a worker, calculating a distance between each coordinate and a starting coordinate corresponding to a starting point of a body part determined as a starting point among the body parts based on the coordinates corresponding to each of a plurality of body parts included in the skeletal data, and converting each coordinate included in the skeletal data to the calculated distance, and training a model that outputs information indicating a motion of a worker that corresponds to one of the common motions based on the motion information after the coordinates have been converted to distance, as a model for motion recognition.

According to this aspect, the distance between the coordinates corresponding to each of the multiple body parts included in the skeletal data and the starting coordinates corresponding to the starting part can be calculated, and each coordinate of the skeletal data can be replaced with the calculated distance. Then, based on the movement information after each coordinate of the skeletal data has been replaced with distance data, a model for action recognition can be trained that outputs information indicating the worker's actions that correspond to any of the common actions.

A motion recognition method according to another aspect of the present disclosure is a motion recognition method executed by a processor, and includes: acquiring time-series information on a motion of a worker; calculating a distance between each coordinate and an origin coordinate corresponding to an origin part of the body part determined as an origin based on coordinates corresponding to each of a plurality of body parts included in skeletal data extracted from the time-series information; converting each coordinate included in the skeletal data to the calculated distance; inputting the converted distance into a trained model; and recognizing the motion of the worker based on information output from the trained model indicating a motion of the worker that corresponds to one of the common motions commonly included in different work processes to be recognized.

According to this aspect, the distance between the coordinates corresponding to each of the multiple body parts included in the skeletal data extracted from the time-series information of the worker to be recognized and the starting coordinates corresponding to the starting part can be calculated, and each coordinate of the skeletal data can be replaced with the calculated distance. Then, by inputting the replaced distance data into the trained model, it is possible to recognize the worker's movements that correspond to any of the common movements.

A motion recognition program according to another aspect of the present disclosure causes a computer to function as an acquisition unit that acquires motion information including elapsed time of work, common motions common to different work processes to be recognized, and skeletal data of a worker, a conversion unit that calculates the distance between each coordinate and a starting coordinate corresponding to a starting point among the body parts defined as a starting point based on the coordinates corresponding to each of a plurality of body parts included in the skeletal data, and converts each coordinate included in the skeletal data to the calculated distance, and a learning unit that learns a model for motion recognition that outputs information indicating a motion of a worker corresponding to one of the common motions based on the motion information after the coordinates have been converted into distances by the conversion unit.

According to this aspect, the distance between the coordinates corresponding to each of the multiple body parts included in the skeletal data and the starting coordinates corresponding to the starting part can be calculated, and each coordinate of the skeletal data can be replaced with the calculated distance. Then, based on the movement information after each coordinate of the skeletal data has been replaced with distance data, a model for action recognition can be trained that outputs information indicating the worker's actions that correspond to any of the common actions.

A motion recognition program according to another aspect of the present disclosure causes a computer to function as an acquisition unit that acquires time-series information regarding a worker's motion, a conversion unit that calculates the distance between each coordinate and an origin coordinate corresponding to an origin part of the body part that is determined as an origin based on the coordinates corresponding to each of the multiple body parts included in the skeletal data extracted from the time-series information, and converts each coordinate included in the skeletal data into the calculated distance, and a motion recognition unit that inputs the distances converted by the conversion unit into a trained model and recognizes the motion of the worker based on information indicating a motion of the worker that corresponds to one of the common motions commonly included in the different work processes to be recognized, which is output from the trained model.

According to this aspect, the distance between the coordinates corresponding to each of the multiple body parts included in the skeletal data extracted from the time-series information of the worker to be recognized and the starting coordinates corresponding to the starting part can be calculated, and each coordinate of the skeletal data can be replaced with the calculated distance. Then, by inputting the replaced distance data into the trained model, it is possible to recognize the worker's movements that correspond to any of the common movements.

The present invention provides an action recognition device, an action recognition method, and an action recognition program that can recognize common actions performed in different work processes.

1 is a diagram illustrating an example of an overview of an action recognition system according to an embodiment of the present invention. 11A to 11C are schematic diagrams illustrating an example of a gripping operation in a process of tightening a screw. 11A to 11C are schematic diagrams illustrating an example of a gripping operation in a case fitting process. FIG. 2 is a diagram illustrating a functional configuration of an action recognition system and an action recognition device. FIG. 2 is a diagram illustrating an example of action information stored in the action recognition device. 13 is a diagram showing a schematic diagram of a manner in which distances between coordinates corresponding to the neck in the skeletal data and each of the other coordinates are calculated. FIG. FIG. 2 is a diagram illustrating an example of a hardware configuration of the action recognition device. 10 is a flowchart illustrating an example of an operation of the action recognition device in a learning mode. 9 is a flowchart for explaining the procedure of the conversion process shown in FIG. 8 . 10 is a flowchart illustrating an example of an operation of the action recognition device in a action recognition mode.

Below, an embodiment of one aspect of the present invention (hereinafter, referred to as "this embodiment") will be described with reference to the drawings. Note that in each drawing, parts with the same reference numerals have the same or similar configurations.

§1 Application Example First, an example of a scene to which the present invention is applied will be described with reference to FIG. 1. In the action recognition system 100 according to this embodiment, the action of a worker A performed in a certain work area R is captured by the image sensors 20a, 20b, and 20c, and the action recognition device 10 that acquires the captured video recognizes a common action of the worker A using a trained model. Here, the common action is an action that is commonly included in different work processes (action that belongs to the same action type). Such a common action is, for example, an action that is commonly performed in different work processes, such as grasping, carrying, and adjustment.

Grasping is defined as the action of moving the hand to the work object and grabbing it, for example, the action of going to grab a part or tool. Transporting is defined as the action of moving the work object to the desired location, for example, the action of carrying a part or tool toward a product being assembled. Adjusting is defined as the action of moving the work to the target state, for example, the action of assembling parts.

In this embodiment, as an example, a case where the common action is grasping is described, but the same can be applied to other common actions such as carrying and adjusting.

The trained model is a model for motion recognition that is trained to receive motion information, including distance data, generated based on the skeletal data of a worker, as input, and to output information indicating the motion of the worker. The distance data input during training is data indicating the distance between the coordinates corresponding to each of the multiple body parts included in the skeletal data of the worker and the starting coordinates corresponding to a starting part of the body part that is set as the starting point.

For example, the neck, waist, etc. can be set as the starting point. Here, if the upper body of the worker is the recognition target, it is preferable to set the neck as the starting point, and if the entire body of the worker is the recognition target, it is preferable to set the waist as the starting point. In this way, it is preferable to set the starting point near the center of the movement of the worker to be recognized.

The distance data will be further explained with reference to Figures 2 and 3. Figure 2 is a diagram illustrating the movement of worker Aa stretching out his right hand to grab an electric screwdriver D that is suspended from above and in front of worker Aa by a cable during the process of tightening a screw. Figure 3 is a diagram illustrating the movement of worker Ab stretching out both hands to grab a case C placed in a parts box during the process of fitting a case.

In both Figures 2 and 3, workers Aa and Ab are performing a gripping motion. However, for example, the gripping motion of the right hand of worker Aa and the right hand of worker Ab are completely different. In the past, when recognizing such a gripping motion, it was necessary to generate a learning model or the like for each work process and recognize the gripping motion performed in each work process.

In response to this, the present invention replaces each coordinate included in the worker's skeletal data with a distance starting from a certain body part, and learns the characteristics of these distances in association with the worker's grasping motion, making it possible to recognize each of the grasping motions performed in different work processes. This is explained in detail below.

In Figures 2 and 3, when the neck of each worker Aa and Ab is used as the starting point and the right hand movements of each worker Aa and Ab are analyzed, it can be characterized that in both gripping movements, the right hand moves in a direction away from the neck. In the process of tightening a screw, as shown in Figure 2, the movement of worker Aa gripping electric screwdriver D with his right hand is such that it is desirable for all workers to move equally efficiently in the process of tightening a screw. Similarly, in the process of fitting the case, as shown in Figure 3, the movement of worker Ab gripping case C with his right hand is such that it is desirable for all workers to move equally efficiently in the process of fitting the case.

Therefore, by learning the distance data of the right hand corresponding to the grasping action performed in each work process in association with the grasping action in each work process, it becomes possible to accurately recognize the movement of the worker grasping the work object with his/her right hand in each work process.

In this way, the action recognition device 10 according to this embodiment can recognize common actions, such as gripping actions, performed in different work processes.

§2 Configuration example [Functional configuration]
Next, an example of the functional configuration of the action recognition system 100 and the action recognition device 10 according to the present embodiment will be described with reference to Fig. 4. The action recognition system 100 includes three image sensors 20a, 20b, and 20c, and the action recognition device 10. In the following, when there is no need to distinguish between the three image sensors 20a, 20b, and 20c, they will be referred to as the image sensor 20. The action recognition device 10 has, as its functional configuration, for example, an acquisition unit 11, a conversion unit 12, a learning unit 13, an action recognition unit 14, and a storage unit 19. The storage unit 19 stores, for example, a video 19a, action information 19b, and a learned model 19c.

In this embodiment, the case will be described where the motion recognition device 10 has a function of learning a model for motion recognition (learning mode) and a function of recognizing the motion of a worker (motion recognition mode), but each function may be distributed and provided in an independent, separate device.

The details of each functional configuration of the action recognition system 100 and the action recognition device 10 are explained below.

<Image sensor>
The image sensor 20 is, for example, a general-purpose camera, and captures a video including a scene in which the worker A is performing an action in the working area R. The image sensor 20 has, for example, a detection unit as a functional configuration. The detection unit detects the action of the worker A, and outputs a video showing the action as time-series information.

Here, the time series information is not limited to video. For example, it may be information about coordinates indicating the movements of worker A measured by a motion capture device that can be provided in place of the image sensor 20.

Each image sensor 20a, 20b, 20c is positioned so that it can capture the entire work area R and the entire body of worker A. In this case, for example, each image sensor 20a, 20b, 20c may be positioned so that it can capture the entire work area R and the entire body of worker A, or each image sensor 20a, 20b, 20c may be positioned so that it can capture a portion of the work area R and worker A and cover the entire work area R and the entire body of worker A by combining the respective videos. Also, each image sensor 20a, 20b, 20c may capture the work area R and worker A at different magnifications. There is no need to provide three image sensors 20; at least one or more may be provided.

<Acquisition Department>
The acquisition unit 11 acquires time-series information (video in this embodiment) related to the actions performed by the worker A from the image sensor 20. The time-series information acquired by the acquisition unit 11 is transmitted to the storage unit 19 and stored as a video 19a. The acquisition unit 11 also acquires the video 19a stored in the storage unit 19.

The acquisition unit 11 extracts skeletal data indicating the movement of the worker's skeleton from the images of the video 19a. The skeletal data can be represented by coordinates (x, y) corresponding to each of a plurality of body parts. In this embodiment, the case where the coordinates are two-dimensional coordinates (x, y) is described, but the same can be applied to the case where the coordinates are three-dimensional coordinates (x, y, z). In addition to the coordinate values, information indicating the accuracy of the coordinate values may be added.

The skeletal data is transmitted to the storage unit 19 and stored as part of the motion information 19b. The acquisition unit 11 also acquires the motion information 19b stored in the storage unit 19.

Movement information 19b will be described with reference to FIG. 5. Movement information 19b is configured to include, as data items, an elapsed time item, a right hand movement item, a left hand movement item, and a skeleton data item, for example. The elapsed time item stores the elapsed time from the reference time, which is the time when the first process of all the processes to be worked on was started. The interval of the elapsed time can be set arbitrarily, and may be set, for example, in units of frames of a video, or may be set at a predetermined interval such as every second.

The right hand action item stores information indicating which of the common actions to be recognized corresponds to a right hand action. The left hand action item stores information indicating which of the common actions to be recognized corresponds to a left hand action. The information stored in the right hand action item and left hand action item can be registered, for example, by referring to a video, checking each of the right and left hand actions at each elapsed time, and inputting the confirmed action content.

The skeleton data item of the motion information 19b stores skeleton data extracted from the video corresponding to the elapsed time. Note that the skeleton data stored in this skeleton data item is replaced with distance data calculated by the conversion process described below.

The motion information in the first row shown in Figure 5 is based on the video at one second after the start of the first process. This motion information contains information indicating that the motion of the worker's right hand does not fall into any of the common motions and that the motion of the worker's left hand falls into the common motion of grasping, as well as skeletal data extracted from the video at that time. The motion information in the last row shown in Figure 5 is based on the video at two minutes and fifty-three seconds after the start of the first process. This motion information contains information indicating that the motions of the worker's right and left hands fall into the common motion of adjusting, as well as skeletal data extracted from the video at that time.

<Conversion section>
4 executes a conversion process for converting each coordinate included in the skeletal data into a distance from a starting coordinate. The conversion process is a process for calculating a distance between each coordinate and a starting coordinate corresponding to a starting point based on coordinates corresponding to each of a plurality of body parts included in the skeletal data, and replacing each coordinate included in the skeletal data with the calculated distance.

The conversion unit 12 has a common function in both the learning mode and the motion recognition mode, but the input skeletal data is different. In the learning mode, the distance is calculated based on the skeletal data included in the motion information generated using the video of the target of learning. On the other hand, in the motion recognition mode, the distance is calculated based on the skeletal data extracted from the video of the worker to be recognized.

Figure 6 shows a schematic diagram of how the neck is set as the starting point a and the distances between the starting point coordinates corresponding to the neck and the coordinates corresponding to each body part are calculated.

By converting each coordinate of the skeleton data shown in Figure 6 into a distance from the origin coordinate, the following recognition becomes possible. For example, if the distance corresponding to the right hand increases over time, it is possible to recognize how much the right hand is being moved (stretched) in the direction away from the neck. On the other hand, if the distance corresponding to the right hand decreases over time, it is possible to recognize how much the right hand is being moved (shortened) in the direction toward the neck.

The distance d _i from the origin coordinate can be calculated, for example, according to the following formula (1). In formula (1), the origin coordinate corresponding to the origin part a is (x _a , y _a ), and the coordinate corresponding to an arbitrary body part i is (x _i , y _i ) (i is variable). The origin coordinate may be included in the coordinate corresponding to the arbitrary body part i.

d _i =(|x _i- x _a | ^p +|y _i- y _a | ^p ) ^1/p ... Formula (1)
The value of p in formula (1) can be, for example, 1 or 2, but is not limited thereto.

The conversion unit 12 replaces the coordinates (x _i , y _i ) corresponding to an arbitrary body part i with, for example, (d _i , d _i ) based on the distance d _i calculated by the formula (1).

Here, the conversion unit 12 may further include an adjustment unit that performs an adjustment process to adjust the skeletal data extracted from the video. The adjustment unit included in the conversion unit 12 performs the adjustment process before performing the above-mentioned conversion process.

The adjustment process includes, for example, a time series complementation process for skeletal data, a normalization process for height (physique), a time series smoothing process for skeletal data, a shift process for skeletal data, and a noise addition process. Each process included in the adjustment process is described below.

The time series completion process of skeletal data is a process that completes missing data that occurs in skeletal data based on other skeletal data located before and after in time. Missing data may occur, for example, in areas that cannot be estimated because they are hidden by the worker's posture, etc.

Height (physique) normalization is a process that normalizes skeletal data based on body shape to absorb physical differences that occur between men and women, for example. As a normalization process, for example, normalized data can be generated by dividing the skeletal data by the length of the torso (for example, the length from the nose to the waist).

The time series smoothing process of the skeletal data is a process that removes noise from changes in the time axis direction of the skeletal data. As a smoothing process, for example, Gaussian filter processing can be performed on the skeletal data to generate smoothed data.

Skeletal data shifting is a process that aligns the origin of skeletal data and standardizes the starting point to reduce variation in movement and make movement characteristics easier to recognize. For example, the entire skeletal data is translated so that the neck joint is located at the origin, and shifted skeletal data is generated.

The noise addition process is a process that virtually increases the skeletal data by adding noise to the skeletal data. The noise added to the skeletal data can be generated, for example, by randomly generating values within a range that can be used as skeletal data.

Here, it is preferable that the adjustment unit executes each step of the adjustment process described above in the learning mode, and executes each step of the adjustment process described above excluding the noise addition process in the action recognition mode.

<Study Section>
The learning unit 13 shown in Fig. 4 is a function in the learning mode. The learning unit 13 generates (learns) a model for action recognition that outputs information indicating an action of a worker that corresponds to any of the common actions, based on the action information 19b after each coordinate of the skeleton data is converted into a distance by the conversion unit 12.

The model trained by the training unit 13 is transmitted to the memory unit 19 and stored as a trained model 19c.

<Action Recognition Unit>
The motion recognition unit 14 is a function in the motion recognition mode. The motion recognition unit 14 inputs motion information 19b including distance data converted from each coordinate of the skeletal data corresponding to the worker to be recognized by the conversion unit 12 to the trained model 19c, and recognizes the motion of the worker based on information indicating the motion of the worker corresponding to any of the common motions output from the trained model 19c. When recognizing the motion of the worker, for example, a known motion recognition method such as ST-GCN (Spatial Temporal Graph Convolutional Networks) can be used.

[Hardware configuration]
Next, an example of the hardware configuration of the action recognition device 10 according to the present embodiment will be described with reference to FIG. 7. The action recognition device 10 includes a CPU (Central Processing Unit) 10a corresponding to a calculation device, a RAM (Random Access Memory) 10b corresponding to a storage unit 19, a ROM (Read Only Memory) 10c corresponding to the storage unit 19, a communication device 10d, an input device 10e, and a display device 10f. These components are connected to each other via a bus so that they can transmit and receive data to each other. Note that, although the present embodiment describes a case where the action recognition device 10 is configured by one computer, the action recognition device 10 may be realized using multiple computers.

The CPU 10a executes programs stored in the RAM 10b or ROM 10c and functions as a control unit that performs calculations and processing of data. The CPU 10a receives various input data from the input device 10e or communication device 10d, and displays the results of calculations on the input data on the display device 10f or stores them in the RAM 10b or ROM 10c.

RAM 10b is composed of, for example, a semiconductor memory element, and stores rewritable data. ROM 10c is composed of, for example, a semiconductor memory element, and stores readable but non-rewritable data.

The communication device 10d is an interface that connects the action recognition device 10 to an external device. The communication device 10d is connected to the image sensor 20 via a communication network such as a LAN (Local Area Network) or the Internet, for example, and receives video from the image sensor 20.

The input device 10e is an interface that accepts data input from a user and may include, for example, a keyboard, a mouse, and a touch panel.

The display device 10f is an interface that visually displays the results of calculations performed by the CPU 10a, and can be configured, for example, with an LCD (Liquid Crystal Display).

The program may be provided by being stored in a computer-readable storage medium such as RAM 10b or ROM 10c, or may be provided via a communication network connected by communication device 10d. In the action recognition device 10, the CPU 10a executes the program to perform the operations of the acquisition unit 11, conversion unit 12, learning unit 13, and action recognition unit 14 shown in FIG. 4. Note that these physical configurations are merely examples and do not necessarily have to be independent configurations. For example, the action recognition device 10 may be provided with an LSI (Large-Scale Integration) in which the CPU 10a is integrated with the RAM 10b and ROM 10c.

8 is a flow chart showing an example of an operation in the learning mode of the action recognition device 10 according to this embodiment. This operation is an operation for learning an action recognition model using the action information 19b obtained after each coordinate included in the skeletal data extracted from the learning video is converted into a distance from the starting point coordinate.

First, the acquisition unit 11 acquires from the storage unit 19 the motion information 19b generated based on the learning video 19a (step S101). This motion information 19b is generated based on a video of a worker performing a work process including a common motion. In addition, the skeleton data item of the motion information 19b stores skeleton data of the worker corresponding to the video.

Next, the conversion unit 12 executes a conversion process to convert each coordinate included in the skeleton data into a distance from the origin coordinate (step S102). The procedure for this conversion process will be described later.

Then, the learning unit 13 trains a model for action recognition that outputs information indicating an action of the worker that corresponds to one of the common actions, based on the action information 19b including the distance data after conversion in step S102 (step S103). Then, this operation ends.

The procedure for the conversion process executed in step S102 above will be described with reference to FIG. 9. In this conversion process, it is assumed that the neck is set in advance as the starting point.

First, the acquisition unit 11 extracts skeletal data for one body per frame from an image of a video 19a having, for example, F frames (step S201). In other words, the acquisition unit 11 extracts skeletal data for F bodies.

Next, the conversion unit 12 calculates the distance between the coordinates corresponding to each of the multiple body parts included in the skeletal data of one body and the starting coordinates corresponding to the starting part using the above formula (1) (step S202).

Then, the conversion unit 12 converts each coordinate included in the skeleton data into the distance calculated in step S202 (step S203). As a result, the distance data is stored in the skeleton data item of the motion information 19b.

Next, the conversion unit 12 determines whether the coordinates of the skeletal data of all F bodies have been converted into distances (step S204). If the determination is NO, the process proceeds to step S202, whereas if the determination is YES, the conversion process ends.

Figure 10 is a flowchart showing an example of the operation of the action recognition device 10 according to this embodiment in the action recognition mode. This operation is based on the premise that a video 19a capturing the action of a worker to be recognized has already been stored in the storage unit 19.

First, the acquisition unit 11 acquires a video 19a for motion recognition from the storage unit 19 (step S301). This video 19a is a video of a worker performing a work process including a common motion. In addition, the skeleton data item of the motion information 19b stores skeleton data of the worker corresponding to the video.

Next, the conversion unit 12 executes a conversion process to convert each coordinate included in the skeleton data into a distance from the origin coordinate (step S302). The procedure of this conversion process is similar to the procedure of FIG. 9 described above, and therefore will not be described.

Next, the action recognition unit 14 inputs the action information 19b including the distance data after the conversion in the above step S302 to the trained model 19c, and recognizes the action of the worker based on the information indicating the action of the worker that corresponds to any of the common actions output from the trained model 19c (step S303). Then, this operation ends.

As described above, the motion recognition device 10 according to this embodiment can calculate the distance between the coordinates corresponding to each of the multiple body parts included in the skeletal data extracted from the video of the worker to be learned and the starting coordinates corresponding to the starting part, and replace each coordinate of the skeletal data with the calculated distance. Then, based on the motion information after replacing each coordinate of the skeletal data with distance data, it is possible to train a motion recognition model that outputs information indicating the motion of the worker that corresponds to one of the common motions.

In addition, the system can calculate the distance between the coordinates corresponding to each of the multiple body parts included in the skeletal data extracted from the video of the worker to be recognized and the starting coordinates corresponding to the starting part, and replace each coordinate of the skeletal data with the calculated distance.The replaced distance data can then be input into the trained model, making it possible to recognize the worker's movements that correspond to any of the common movements.

Therefore, the action recognition device 10 according to this embodiment can recognize common actions performed in different work processes.

The present invention is not limited to the above-described embodiment, and can be implemented in various other forms without departing from the scope of the present invention. For example, the embodiment of the present invention can be described as in the following appendix. However, the embodiment of the present invention is not limited to the form described in the following appendix. Furthermore, the embodiment of the present invention may be in a form in which the descriptions between the appendixes are substituted or combined.

[Appendix 1]
an acquisition unit (11) for acquiring motion information (19b) including an elapsed time of a task, a common motion included in common to different task processes to be recognized, and skeletal data of a worker;
a conversion unit (12) that calculates a distance between each of the coordinates corresponding to a plurality of body parts included in the skeletal data and a starting point coordinate corresponding to a starting point part of the body parts that is determined as a starting point, based on the coordinates corresponding to each of the body parts included in the skeletal data, and converts each of the coordinates included in the skeletal data into the calculated distance;
a learning unit (13) that learns, as a model for action recognition, a model that outputs information indicating an action of the worker that corresponds to any one of the common actions, based on the action information (19b) after the coordinates have been converted into the distances by the conversion unit (12); and
An action recognition device (10) comprising:

[Appendix 2]
An acquisition unit (11) that acquires time-series information (19a) regarding a worker's motion;
a conversion unit (12) that calculates a distance between each of a plurality of body parts included in the skeleton data extracted from the time-series information (19a) and a starting point coordinate corresponding to a starting point part of the body parts that is determined as a starting point, and converts each of the coordinates included in the skeleton data into the calculated distance;
an action recognition unit (14) that inputs the distance converted by the conversion unit (12) into a trained model (19c) and recognizes the action of the worker based on information indicating any of the actions of the worker that corresponds to any of the common actions commonly included in different work processes that are targets of recognition, the information being output from the trained model (19c);
An action recognition device (10) comprising:

[Appendix 3]
The starting site is either the neck or the lower back.
3. The action recognition device (10) according to claim 1 or 2.

[Appendix 4]
A motion recognition method executed by a processor (10a), comprising:
Acquiring motion information (19b) including an elapsed time of a task, a common motion included in common to different task processes to be recognized, and skeletal data of a worker;
calculating a distance between each of the coordinates corresponding to a plurality of body parts included in the skeletal data and a starting point coordinate corresponding to a starting point part of the body parts that is determined as a starting point, based on the coordinates corresponding to each of the body parts included in the skeletal data, and converting each of the coordinates included in the skeletal data into the calculated distance;
a model that outputs information indicating an action of the worker that corresponds to any one of the common actions based on the action information (19b) after the coordinates are converted into the distance is trained as a model for action recognition;
The action recognition method includes:

[Appendix 5]
A motion recognition method executed by a processor (10a), comprising:
Obtaining time series information regarding the worker's movements;
Calculating a distance between each of the coordinates corresponding to a plurality of body parts included in the skeleton data extracted from the time-series information (19a) and a starting point coordinate corresponding to a starting point part of the body parts that is determined as a starting point, and converting each of the coordinates included in the skeleton data into the calculated distance;
inputting the converted distance into a trained model (19c), and recognizing the motion of the worker based on information indicating any of the motions of the worker that correspond to any of the common motions commonly included in different work processes that are targets of recognition, outputted from the trained model (19c);
The action recognition method includes:

[Appendix 6]
Computer,
an acquisition unit (11) for acquiring motion information (19b) including an elapsed time of a task, a common motion included in common to different task processes to be recognized, and skeletal data of a worker;
a conversion unit (12) that calculates a distance between each of the coordinates corresponding to a plurality of body parts included in the skeleton data and a starting point coordinate corresponding to a starting point part defined as a starting point among the body parts, based on the coordinates corresponding to each of the body parts included in the skeleton data, and converts each of the coordinates included in the skeleton data into the calculated distance;
a learning unit (13) that learns, as a model for action recognition, a model that outputs information indicating an action of the worker that corresponds to any one of the common actions, based on the action information (19b) after the coordinates have been converted into the distances by the conversion unit (12);
A motion recognition program that functions as a

[Appendix 7]
Computer,
An acquisition unit (11) for acquiring time-series information (19a) relating to the motion of a worker;
a conversion unit (12) that calculates a distance between each of the coordinates corresponding to a plurality of body parts included in the skeleton data extracted from the time-series information (19a) and a starting point coordinate corresponding to a starting point part of the body parts that is determined as a starting point, and converts each of the coordinates included in the skeleton data into the calculated distance;
an action recognition unit (14) that inputs the distance converted by the conversion unit (12) into a trained model (19c) and recognizes the action of the worker based on information output from the trained model (19c) indicating any of the actions of the worker that correspond to any of the common actions commonly included in different work processes that are targets of recognition;
A motion recognition program that functions as a

10...motion recognition device, 10a...CPU, 10b...RAM, 10c...ROM, 10d...communication device, 10e...input device, 10f...display device, 11...acquisition unit, 12...conversion unit, 13...learning unit, 14...motion recognition unit, 19...storage unit, 19a...video, 19b...motion information, 19c...trained model, 20a, 20b, 20c...image sensor, 100...motion recognition system, A...worker, R...work area, a...starting point

Claims (4)

an acquisition unit that acquires from a storage unit motion information generated based on time-series information on a motion of a first worker in a task performed by the first worker to be recognized, the motion information including an elapsed time of the task performed by the first worker, information indicating a common motion commonly performed by all workers although the motion differs in each task process, and skeletal data of the first worker, in association with each other;
a conversion unit that calculates, based on coordinates of a coordinate system that represents the skeletal data corresponding to each of a plurality of body parts included in the skeletal data of the acquired motion information, a distance between each of the coordinates and a starting point coordinate corresponding to a starting point part of the body part that is defined as a starting point near the center of the worker's movement, and replaces each of the coordinates included in the skeletal data with the calculated distance;
an action recognition unit that inputs action information including the distance after the replacement by the conversion unit into a trained model, and recognizes the action of the first worker based on information indicating a worker's action corresponding to any of the common actions output from the trained model; and
Equipped with
The trained model is a model trained to receive input of motion information after the coordinates included in the skeletal data of the motion information generated based on time-series information on the motion of the second worker in the work performed by the second worker to be trained are replaced with the distance by the conversion unit, and to output information indicating a motion of the worker corresponding to any of the common motions.
Motion recognition device. The starting site is either the neck or the lower back.
The action recognition device according to claim 1. 1. A processor-implemented method for action recognition, comprising:
acquiring from a storage unit motion information generated based on time-series information on motions of a first worker in a task performed by the first worker to be recognized, the motion information storing, in association with each other, information on an elapsed time of the task performed by the first worker, information on common motions commonly performed by all workers although the movements are different in different task steps, and skeletal data of the first worker;
calculating a distance between each of the coordinates of a coordinate system representing the skeletal data corresponding to each of a plurality of body parts included in the skeletal data of the acquired motion information and a starting point coordinate corresponding to a starting point part of the body part that is defined as a starting point near the center of the worker's movement, and replacing each of the coordinates included in the skeletal data with the calculated distance;
inputting motion information including the replaced distance into a trained model, and recognizing the motion of the first worker based on information indicating a motion of the worker corresponding to any of the common motions output from the trained model;
Including,
The trained model is a model trained to receive input of motion information after the coordinates included in the skeletal data of the motion information generated based on time-series information on the motion of the second worker in the work performed by the second worker to be trained are replaced with the distance, and to output information indicating a motion of the worker corresponding to any of the common motions.
Action recognition method. Computer,
an acquisition unit that acquires from a storage unit motion information generated based on time-series information on a motion of a first worker in a task performed by the first worker to be recognized, the motion information storing, in association with each other, information on an elapsed time of the task performed by the first worker, information on common motions commonly performed by all workers although the movements are different in different task steps, and skeletal data of the first worker;
a conversion unit that calculates, based on coordinates of a coordinate system that represents the skeletal data corresponding to each of a plurality of body parts included in the skeletal data of the acquired motion information, a distance between each of the coordinates and a starting point coordinate corresponding to a starting point part of the body parts that is defined as a starting point near the center of the worker's movement, and replaces each of the coordinates included in the skeletal data with the calculated distance;
an action recognition unit that inputs action information including the distance after the replacement by the conversion unit into a trained model, and recognizes the action of the first worker based on information indicating a worker's action corresponding to any of the common actions output from the trained model;
Function as a
The trained model is a model trained to receive input of motion information after the coordinates included in the skeletal data of the motion information generated based on time-series information on the motion of the second worker in the work performed by the second worker to be trained are replaced with the distance by the conversion unit, and to output information indicating a motion of the worker corresponding to any of the common motions.
Motion recognition program.