JP7700511B2 - Human image data analysis system - Google Patents

Description

The present invention relates to a human image data analysis system.

In recent years, image data of a person has been acquired and the posture of the person has been evaluated. For example, Patent Document 1 describes determining the posture of a worker in order to measure the working time of the worker at a manufacturing site. The working situation is captured by a camera, and skeletal data including feature point data indicating the joint positions of the worker appearing in the acquired image data is acquired. A posture model in which a posture label is associated with each piece of skeletal data is stored in advance. Then, based on the acquired skeletal data, the posture of the person appearing in the image data is determined from the posture label predetermined for the posture model.

In addition to determining a person's posture to measure the worker's working time, it is also important to evaluate the person's posture itself. For example, an evaluation may be made to see if a person is walking with the correct posture. It is also possible to evaluate whether a person using a care device such as a walker is using the walker with the correct posture. There are also various walking support devices known that assist walking or operate to promote independent walking. It is also important to evaluate the posture of a person using a walking support device.

In addition, when workers in factories or other places wear active power assist suits to reduce their workload, it is also important to evaluate the posture of the worker. By evaluating the worker's posture, it is possible to evaluate whether the worker is using the assist suit appropriately and whether the assist suit is functioning properly.

JP 2020-201772 A

As mentioned above, it is very important to evaluate a person's posture. In the method described in Patent Document 1, the posture is determined from the person's skeletal data. When a person in the image data is facing forward or backward, the posture of the person can be easily determined from the person's skeletal data.

However, when the torso is oriented sideways, for example, it may not be possible to determine the posture of a person from skeletal data alone. For example, when the torso is oriented sideways, it is not easy to determine whether the right foot or the left foot is positioned forward. Similarly, when the torso is oriented sideways, it is not easy to determine whether the right arm or the left arm is positioned forward. Also, when a person's upper and lower body are twisted, it is not easy to determine how each part of the person is positioned.

The present invention has been made in view of this background, and aims to provide a human image data analysis system that can determine the posture of a person depicted in human image data with high accuracy.

One aspect of the present invention is
A human image data analysis system including a computer device having a processor and a memory device,
The storage device includes:
storing a trained model for feature extraction generated by performing machine learning using first person image data including a person as an explanatory variable and a feature in the first person image data as a target variable;
storing a trained model for behavior analysis generated by performing machine learning using the feature amount extracted based on the first person image data as an explanatory variable and a behavior type of the person in the multiple first person image data in a time series as a target variable;
storing a trained model for keypoint extraction generated by performing machine learning using the feature amount and the behavior type as explanatory variables and a keypoint representing a posture of the person in the first person image data as a target variable;
The arithmetic processing device includes:
a feature extraction unit that extracts the feature from second person image data including a person by inputting the second person image data using the trained model related to the feature extraction stored in the storage device; and
a behavior type output unit that outputs the behavior type of the person in the plurality of time-series images of the second person image data by inputting the feature amount extracted by the feature amount extraction unit using a trained model related to the behavior analysis stored in the storage device;
a keypoint output unit that outputs the keypoints of the person in the second person image data by inputting the feature extracted by the feature extraction unit and the behavior type output by the behavior type output unit using a trained model related to the keypoint extraction stored in the storage device;
Equipped with
The trained model for feature extraction, the trained model for behavioral analysis, and the trained model for keypoint extraction are in a human image data analysis system, which is trained in a learning phase using a loss function that includes elements of the keypoints and elements of the behavior types .
Another aspect of the present invention is
A human image data analysis system including a computer device having a processor and a memory device,
The storage device includes:
storing a trained model for feature extraction generated by performing machine learning using first person image data including a person as an explanatory variable and a feature in the first person image data as a target variable;
storing a trained model for behavior analysis generated by performing machine learning using the feature amount extracted based on the first person image data as an explanatory variable and a behavior type of the person in the multiple first person image data in a time series as a target variable;
storing a trained model for keypoint extraction generated by performing machine learning using the feature amount and the behavior type as explanatory variables and a keypoint representing a posture of the person in the first person image data as a target variable;
The arithmetic processing device includes:
a feature extraction unit that extracts the feature from each of a plurality of second person image data by inputting a plurality of second person image data in a time series including a person, using the trained model related to the feature extraction stored in the storage device;
a behavior type output unit that outputs the behavior type of the person in the plurality of time-series second person image data by inputting the feature amounts for the plurality of images extracted by the feature amount extracting unit based on each of the plurality of time-series second person image data, using a trained model for the behavior analysis stored in the storage device;
a key point output unit that uses a trained model for the key point extraction stored in the storage device to input the feature amounts for one image selected from the plurality of second person image data in time series, the feature amounts being extracted by the feature amount extractor based on the one image of the second person image data selected from the plurality of second person image data in time series, and the behavior type being output by the behavior type output unit, and outputs the key points of the person in the selected one image of the second person image data;
The present invention relates to a human image data analysis system comprising:
Another aspect of the present invention is
A human image data analysis system including a computer device having a processor and a memory device,
The storage device includes:
storing a trained model for feature extraction generated by performing machine learning using first person image data including a person as an explanatory variable and a feature in the first person image data as a target variable;
storing a learned model for behavior analysis generated by performing machine learning using the feature amount for one image extracted based on the first person image data as an explanatory variable and a behavior type of the person in the multiple first person image data in a time series as a target variable;
storing a trained model for keypoint extraction generated by performing machine learning using the feature amount and the behavior type as explanatory variables and a keypoint representing a posture of the person in the first person image data as a target variable;
The arithmetic processing device includes:
a feature extraction unit that sequentially inputs a plurality of second person image data in a time series including a person, by using the trained model related to the feature extraction stored in the storage device, and sequentially extracts the feature from each of the plurality of second person image data;
a behavior type output unit that uses a trained model for the behavior analysis stored in the storage device to sequentially input the feature amounts for one image extracted by the feature amount extraction unit and to perform a recursive calculation using a result of a previous calculation process, thereby outputting the behavior type of the person in the one image of the second person image data that is the target of a current calculation process;
a keypoint output unit that uses a trained model for the keypoint extraction stored in the storage device to input the feature amounts for one image extracted by the feature amount extraction unit and the behavior type output by the behavior type output unit, and outputs the keypoints of the person in the second person image data for one image that is the subject of a current calculation process;
The present invention relates to a human image data analysis system comprising:

The keypoint output unit does not output keypoints for a person using only the features in the person's image data. The keypoint output unit inputs the type of behavior of the person in addition to the features in the person's image data, and outputs keypoints for the person.

In this way, the key point output unit can output a person's key points with high accuracy by grasping the type of behavior of the person before outputting the person's key points. For example, if a person has a posture in which the upper and lower halves of the body are twisted, there is a possibility that a connection relationship connecting adjacent joint positions as one of the key points will be erroneously output. However, by grasping the type of behavior of the person, it is possible to output a connection relationship as one of the key points with high accuracy, even in the case of a twisted posture. Therefore, the posture of a person can be analyzed with high accuracy.

FIG. 1 is a diagram illustrating a configuration of a person image data analysis system. A diagram showing input and output regarding a trained model A in the inference phase in the human image data analysis system of the first embodiment. This figure shows the relationship between trained models B, C, and D in the inference phase, and the inputs and outputs of trained models B, C, and D in the human image data analysis system of the first embodiment. 13 is a graph showing output information of trained model C, representing scores for each behavior type. FIG. 13 is a diagram explaining the output information of the trained model D, showing key points of a person. 13A to 13D are diagrams illustrating inputs and outputs of models B, C, and D in the learning phase in the human image data analysis system of the first embodiment. FIG. 13 is a diagram showing key points of a person as a comparative example. A diagram showing the relationship between trained models B, C, and D in the inference phase, and the inputs and outputs of trained models B, C, and D in the human image data analysis system of the second embodiment.

(1. Overview of human image data analysis system)
The human image data analysis system acquires human image data and analyzes the posture of the person included in the acquired human image data. The posture of the person is classified into, for example, a standing posture, a sitting posture, a lying posture, a kneeling posture, and the like, and each of these is further classified in detail. Furthermore, the posture of the person also differs depending on whether the person is in a stationary state or in a moving state. In other words, the human image data analysis system analyzes the posture of the person depicted in the human image data.

The posture information of a person analyzed by the human image data analysis system is used, for example, as follows. The posture of the person is evaluated when the person is stationary. For example, when the person is standing, an evaluation is made as to whether this is an appropriate standing posture, and the person can be improved to have an appropriate standing posture. Also, when the person is sitting or lying down, an evaluation is made as to whether this is an appropriate sitting or lying down posture, and this can be used to select appropriate seating or bedding, or to develop seating or bedding.

It can also be used to evaluate a person's posture during movement. It can evaluate posture when moving from a standing position to a sitting position, or vice versa, or from a sitting position to a lying position, or vice versa. It can also evaluate posture when walking, running, jumping, etc., and can also evaluate various postures of a person when playing sports.

Furthermore, it is also possible to evaluate whether a person using a nursing care device such as a walker is using the walker with the correct posture. Also, in a walking support device that is driven to assist walking or promote independent walking, it is also possible to evaluate the posture of the person using the walking support device. Using the evaluation results of the person's posture, it is possible to evaluate whether the walking support device is functioning properly. Furthermore, it is also possible to analyze the posture of the person using the walking support device, and use the analysis results to control the walking support device.

In addition, when a care recipient or a worker in a factory wears an active power assist suit to reduce the load of movement, the posture of the wearer can also be evaluated. The evaluation results of the wearer's posture can be used to evaluate whether the assist suit is functioning properly. Furthermore, the posture of the wearer can be analyzed, and the analysis results can be used to control the assist suit. Furthermore, by analyzing the posture of a worker in a factory, etc., the working time of the worker can be evaluated. Furthermore, the working time of each type of work performed by the worker can be evaluated.

2. First Embodiment
(2-1. Configuration of human image data analysis system 1 in inference phase)
The configuration of the person image data analysis system 1 will be described with reference to Figures 1 to 6. In particular, the configuration of the person image data analysis system 1 in the inference phase will be described below. As shown in Figure 1, the person image data analysis system 1 is made up of an imaging device 2 and a computer device used for analysis. The computer device includes a storage device 3 and an arithmetic processing device 4.

The imaging device 2 is, for example, a moving image imaging device capable of capturing continuous moving images in a time series, or a still image imaging device capable of capturing still images in a time series. The imaging device 2 is used to capture an image including a person who is the subject of posture analysis. The storage device 3 stores trained models A, B, C, and D generated by machine learning. The arithmetic processing device 4 includes a person image data generation unit 11, a feature extraction unit 12, a behavior type output unit 13, and a key point output unit 14.

As shown in FIG. 2, trained model A is a machine learning model for extracting human image data that has been generated by performing machine learning. When trained model A receives image data D1 captured by imaging device 2 (hereinafter referred to as "original image data"), it extracts a human area D1a from the original image data D1. The original image data D1 includes a human area D1a and a surrounding area D1b located around the human area D1a. In addition to the person, the human area D1a may also include an object held by the person.

When the original image data D1 is input, the trained model A outputs person image data D2, which is image data of the extracted person region D1a. The trained model A applies, for example, R-CNN (Regions with Convolutional Neural Networks) or the like. The trained model A extracts the person region D1a, for example, using a rectangular region (bounding box) or the like.

As shown in FIG. 3, trained model B is a machine learning model for feature extraction generated by performing machine learning. Trained model B is preferably a machine learning algorithm (including deep learning) including a neural network, but other machine learning algorithms may also be applied. Trained model B is a machine learning model generated by performing machine learning using person image data D2 including a person output by trained model A as an explanatory variable and feature values in person image data D2 as objective variables. In other words, trained model B outputs feature values in person image data D2 by inputting person image data D2.

The types of features extracted by trained model B may be set in advance, or may be automatically extracted by machine learning. Of course, the types of features may be automatically extracted by machine learning and set by the setter. For example, the types of features may be automatically extracted by machine learning and then modified by the setter.

The trained model C is a machine learning model for behavioral analysis generated by performing machine learning. For example, a machine learning algorithm (including deep learning) including a neural network is suitable for the trained model C, but other machine learning algorithms may also be applied. The trained model C is a machine learning model generated by performing machine learning using the feature amounts of multiple images extracted by the trained model B based on each of the multiple images of person image data D2 in a time series as explanatory variables, and the behavior type of the person in the multiple images of person image data D2 in a time series as a target variable. Here, the number of images and the time series time for the feature amounts of multiple images as explanatory variables can be set arbitrarily.

The types of a person's behavior can be broadly categorized into, for example, standing, sitting, lying, and kneeling postures in a stationary state, and walking, running, jumping, and postures when performing various sports in a moving state. The types of a person's behavior are further categorized into these broad categories. For example, sitting postures are categorized into cross-legged, sitting comfortably, sitting upright, sitting long, sitting on the edge of the bed, and half-sitting. Furthermore, lying postures are categorized into supine, lateral, and prone positions. Other postures are also categorized in more detail.

When trained model C receives the feature amounts for multiple images extracted by trained model B based on multiple time-series images of person image data D2, trained model C generates a score for the behavior type of the person as shown in FIG. 4. Trained model C then recognizes the behavior type with the highest score value as the behavior type of the person and outputs that behavior type.

The trained model D is a machine learning model for keypoint extraction generated by performing machine learning. For example, a machine learning algorithm (including deep learning) including a neural network is suitable for the trained model D, but other machine learning algorithms may also be applied. The trained model D is a machine learning model generated by performing machine learning using feature amounts and behavior types as explanatory variables and keypoints expressing the posture of a person in the person image data D2 as objective variables. The feature amounts are information output by the trained model B. The behavior types are information output by the trained model C.

Key points will be described with reference to FIG. 5. Key points include the joint positions of a person, as shown by some of the black circles in FIG. 5. In this embodiment, key points include the positions of the person's eyes, as shown by some of the black circles in FIG. 5. Furthermore, key points include connections shown by lines connecting the black circles in FIG. 5. For example, key points include connections connecting adjacent joint positions, connections connecting adjacent eye positions, and connections connecting eye positions and joint positions close to eye positions. In other words, key points are feature data including parts for expressing a person's posture and connections between each part. When features and behavior types are input, the trained model D outputs the key points shown in FIG. 5.

As shown in Figs. 1 and 2, the person image data generation unit 11 acquires original image data D1 from the imaging device 2. The person image data generation unit 11 uses a trained model A stored in the storage device 3 to input the original image data D1, thereby generating person image data D2 in which a person area D1a is extracted from the original image data D1.

When the original image data D1 is moving image data, the person image data generation unit 11 generates multiple still image data frames consisting of a time series from the acquired moving image data. Then, the person image data generation unit 11 inputs each of the multiple still image data frames consisting of the generated time series (e.g., times T1 to T10) into the trained model A, and generates person image data D2 for each of the multiple still image data frames. In other words, the person image data generation unit 11 generates multiple person image data D2 for times T1 to T10, for example.

When the original image data D1 is still image data, the person image data generation unit 11 inputs each of the multiple still image data frames comprising the acquired time series (e.g., T1 to T10) into the trained model A, and generates person image data D2 for each of the multiple still image data frames. In other words, in this case as well, the person image data generation unit 11 generates multiple person image data D2 for times T1 to T10, for example.

The person image data D2 includes various image data, such as image data of a person with at least their torso facing directly to the imaging device 2, image data of a person with at least their torso facing away from the imaging device 2, and image data of a person with at least their torso facing sideways. Sideways here does not necessarily mean a position at 90° to the imaging device 2, but excludes the positions of completely facing directly to the imaging device 2 and completely facing away from the imaging device 2, and includes the positions of the person facing diagonally.

The person image data D2 also includes image data of a person in a posture in which the upper and lower halves of the body are not twisted, as well as image data of a person in a twisted posture. When a person is walking, the left hand and right foot may be positioned forward, and the right hand and left foot may be positioned backward. In such a case, the person's upper and lower body are in a twisted posture.

As shown in Figs. 1 and 3, the feature extraction unit 12 acquires multiple images of person image data D2 consisting of a time series (T1 to T10) generated by the person image data generation unit 11. The feature extraction unit 12 inputs multiple images of person image data D2 consisting of a time series (T1 to T10) using a trained model B stored in the storage device 3. The feature extraction unit 12 then extracts features for each of the multiple images of person image data D2, i.e., the feature amounts for multiple images, as the output of the trained model B.

As shown in Figs. 1 and 3, the behavior type output unit 13 acquires the feature amounts for multiple images extracted by the feature amount extraction unit 12. Using the trained model C stored in the storage device 3, the behavior type output unit 13 performs processing to input the feature amounts for multiple images extracted by the feature amount extraction unit 12 based on multiple images of person image data D2 consisting of a time series (T1 to T10) to the trained model B.

Then, the behavior type output unit 13 uses the feature amounts for multiple images in the time series (T1 to T10) to output the behavior type of the person in the multiple images of the person image data D2 in a time series. Specifically, as shown in FIG. 4, the behavior type output unit 13 generates a score for each behavior type, and outputs the behavior type with the highest score value as the behavior type of the person.

In this embodiment, the behavior type output unit 13 inputs the feature amounts in multiple images of person image data D2, i.e., the feature amounts for multiple images, rather than the feature amounts in one image of person image data D2. In other words, the behavior type is identified by determining the change in the position of the person in the multiple images of person image data D2 in a time series.

As shown in Figures 1 and 3, the key point output unit 14 acquires the features extracted by the feature extraction unit 12. As described above, the feature extraction unit 12 extracts features for each of the multiple pieces of person image data D2 consisting of a time series (T1 to T10), i.e., the feature amounts for multiple pieces.

However, the keypoint output unit 14 does not need to use feature amounts for multiple images consisting of a time series (T1 to T10). In this embodiment, the keypoint output unit 14 acquires feature amounts for one image extracted based on one image of person image data D2 selected from multiple images of person image data D2 consisting of a time series (T1 to T10). For example, the keypoint output unit 14 acquires feature amounts extracted based on person image data D2 at intermediate time T5 between times T1 to T10. Note that the time selected by the keypoint output unit 14 can be determined arbitrarily.

Furthermore, the key point output unit 14 acquires the behavior type output by the behavior type output unit 13. Using the trained model C stored in the storage device 3, the key point output unit 14 performs a process of inputting the acquired features and behavior type into the trained model C, thereby outputting the key points of the person in the person image data D2 at time T5.

As shown in FIG. 5, the key point output unit 14 outputs the joint positions, eye positions, and the connection relationships connecting each position as the key points of the person in the person image data D2 at time T5.

(2-2. Configuration of human image data analysis system 1 in learning phase)
The configuration of the human image data analysis system 1 in the learning phase will be described with reference to Fig. 6. In particular, the learning phase for models B, C, and D will be described.

First, a training dataset to be used for learning is prepared. As the training dataset, many units each consisting of multiple time-series images of person image data D2 are prepared. For example, multiple video image data sets are suitable as a training dataset because they contain many units each consisting of multiple time-series images of person image data D2. Furthermore, the training dataset includes label information on key points of people in the person image data D2 and types of behavior of people.

The loss function F(x, y) used in learning includes a keypoint element x and an action type element y. Models B, C, and D input a training dataset and learn to reduce the loss function F(x, y). As the loss function F(x, y) has keypoint elements and action type elements, models B, C, and D are trained to output correct answers for keypoints and action types. The trained models B, C, and D trained in this way are stored in the storage device 3.

When learning using the loss function F(x, y) as described above, models B, C, and D are not trained independently, but rather models B, C, and D are trained as a unified model. Therefore, the parts of models B, C, and D that are affected by the loss function (x, y) are trained effectively.

(2-3. Effects)
In the person image data analysis system 1, the key point output unit 14 does not output the key points of a person by using only the feature amounts in the person image data D2. The key point output unit 14 inputs the type of behavior of the person in addition to the feature amounts in the person image data D2, and outputs the key points of the person.

In this way, the keypoint output unit 14 can output a person's keypoints with high accuracy by grasping the type of behavior of the person and then outputting the person's keypoints. This will be explained by comparing Figure 5, which shows the output result of keypoints in this embodiment, with Figure 7, which shows the output result of keypoints as a comparative example.

Figure 5 shows key points output by the key point output unit 14 in this embodiment. On the other hand, Figure 7 shows key points output based only on the feature amounts in the person image data D2, without taking into account the type of activity. The person image data D2 used for the key points shown in Figures 5 and 7 is a posture in which the person's upper and lower body are twisted. Furthermore, the person image data D2 is image data of a posture in which at least the torso of the person is turned sideways.

In the lower body of the person shown in Figure 5, the right hip joint is connected to the right knee joint, and the left hip joint is connected to the left knee joint. Thus, in Figure 5, the joints are correctly connected. On the other hand, in the lower body of the person shown in Figure 7, the right hip joint is connected to the left knee joint, and the left hip joint is connected to the right knee joint. In other words, in Figure 7, the joints are incorrectly connected.

In the lower body of the person shown in Figures 5 and 7, the right hip joint is closer to the left knee joint than the right knee joint, and the left hip joint is closer to the right knee joint than the left knee joint. And because the person's torso is posed sideways in the person's image data, the left and right hip joints and the left and right knee joints are in reversed front-to-back positions. In Figure 7, it appears that joints located close to each other are connected.

As shown in FIG. 7, when a person has a posture in which the upper and lower halves of the body are twisted, there is a possibility that the connection relationship connecting adjacent joint positions as one of the key points may be erroneously output. If the connections of the joint positions are not correctly recognized, the person's posture cannot be correctly recognized. However, in this embodiment, as shown in FIG. 5, by grasping the type of behavior of the person, it is possible to output the connection relationship as one of the key points with high accuracy even if the person is in a twisted posture and lying sideways. Therefore, the person's posture can be analyzed with high accuracy.

The behavior type output unit 13 outputs the behavior type of a person by inputting feature amounts for multiple time-series images. Therefore, the behavior type output unit 13 can identify the behavior type of a person with high accuracy by using multiple time-series images of person image data D2. As a result, it is possible to output key points of a person with high accuracy.

In addition, in the learning phase, the trained models B, C, and D are trained using a loss function F(x, y) that includes keypoint elements and behavior type elements. In other words, the trained models B, C, and D are trained so that the keypoint output unit 14 can output keypoints that take behavior type into account with high accuracy. The trained models B, C, and D trained in this way are used to output person keypoints, making it possible to output highly accurate keypoints.

In addition, the person image data analysis system 1 does not input the original image data D1 captured by the imaging device 2 itself to the feature extraction unit 12, but inputs person image data D2 in which the person area D1a has been extracted from the original image data D1 to the feature extraction unit 12. In this way, generating person image data D2 in which the person area D1a has been extracted leads to highly accurate output of key points of people in the person image data D2.

3. Second Embodiment
The configuration of the inference phase of the human image data analysis system 1 according to the second embodiment will be described with reference to FIGS.

As shown in FIG. 1, the human image data analysis system 1 includes a storage device 3 that stores trained models A, B, C, and D, and a processing device 4. The processing device 4 includes a human image data generation unit 11, a feature extraction unit 12, a behavior type output unit 13, and a key point output unit 14.

Trained models A, B, and D are the same as trained models A, B, and D in the first embodiment. Trained model C applies a recursive algorithm. For example, trained model C applies a recurrent neural network (RNN), long short term memory (LSTM), etc.

In other words, the trained model C is a machine learning model that, when the feature amounts for one image extracted by the feature extraction unit 12 based on one image of person image data D2 are input sequentially, performs a recursive calculation using the result of the previous calculation process, and outputs the behavior type of the person in the image of person image data D2 that is the subject of the current calculation process.

In this embodiment, the processing of each part constituting the calculation processing device 4 is as follows. The feature extraction unit 12 uses the trained model A to sequentially input a plurality of time-series images of person image data D2, thereby sequentially extracting features from each of the plurality of images of person image data D2. In other words, the feature extraction unit 12 sequentially extracts features from one image of person image data D2 that is the subject of calculation processing.

The behavior type output unit 13 uses the trained model C to sequentially input the feature amounts for one image extracted by the feature extraction unit 12 based on one image of the person D2, and performs a recursive calculation using the result of the previous calculation process, thereby outputting the behavior type of the person in the image of the person D2 that is the subject of the current calculation process.

The key point output unit 14 uses the trained model D to input the feature values of the one image that is the subject of the current calculation process and the type of behavior of the person in the person image data D2 that includes the subject of the current calculation process, and outputs the key points of the person in the one image of person image data D2 that is the subject of the current calculation process.

By performing recursive calculations, the behavior type output unit 13 can output the behavior type of a person using the feature amounts for one image that is the subject of the current calculation process. Therefore, the processing in the feature amount extraction unit 12, behavior type output unit 13, and key point output unit 14 is executed by inputting one image of person image data D2 that is the subject of the current calculation process. Therefore, each time time-series person image data is input sequentially, the key points of the person in the person image data can be output. In other words, the key points of the person can be output in real time. As a result, the posture of the person can be analyzed in real time.

REFERENCE SIGNS LIST 1 Person image data analysis system 3 Storage device 4 Processing device 11 Person image data generation unit 12 Feature extraction unit 13 Behavior type output unit 14 Key point output unit D1 Original image data D2 Person image data

Claims (8)

A human image data analysis system including a computer device having a processor and a memory device,
The storage device includes:
storing a trained model for feature extraction generated by performing machine learning using first person image data including a person as an explanatory variable and a feature in the first person image data as a target variable;
storing a trained model for behavior analysis generated by performing machine learning using the feature amount extracted based on the first person image data as an explanatory variable and a behavior type of the person in the multiple first person image data in a time series as a target variable;
storing a trained model for keypoint extraction generated by performing machine learning using the feature amount and the behavior type as explanatory variables and a keypoint representing a posture of the person in the first person image data as a target variable;
The arithmetic processing device includes:
a feature extraction unit that extracts the feature from second person image data including a person by inputting the second person image data using the trained model related to the feature extraction stored in the storage device; and
a behavior type output unit that outputs the behavior type of the person in the plurality of time-series images of the second person image data by inputting the feature amount extracted by the feature amount extraction unit using a trained model related to the behavior analysis stored in the storage device;
a keypoint output unit that outputs the keypoints of the person in the second person image data by inputting the feature extracted by the feature extraction unit and the behavior type output by the behavior type output unit using a trained model related to the keypoint extraction stored in the storage device;
Equipped with
A human image data analysis system, wherein the trained model for feature extraction, the trained model for behavior analysis, and the trained model for keypoint extraction are trained in a learning phase using a loss function that includes elements of the keypoints and elements of the behavior types . the feature extraction unit extracts the feature from each of the second person image data by inputting the second person image data in a time series using a trained model for the feature extraction;
the behavior type output unit outputs the behavior type by inputting the feature amounts for a plurality of images extracted based on each of the plurality of second person image data in time series using a trained model related to the behavior analysis; and
2. The human image data analysis system of claim 1 , wherein the keypoint output unit uses a trained model for the keypoint extraction to input the feature amount for one image extracted based on the one image of the second human image data selected from the plurality of images of the second human image data in a time series, and the behavior type, thereby outputting the keypoints of the person in the one image of the second human image data selected. the feature extraction unit sequentially inputs the plurality of second person image data in time series using a trained model for the feature extraction, thereby sequentially extracting the feature from each of the plurality of second person image data;
the behavior type output unit sequentially inputs the extracted feature amounts for one image using a trained model for the behavior analysis, and performs a recursive calculation using a result of a previous calculation process, thereby outputting the behavior type of the person in the one image of the second person image data that is a target of a current calculation process;
2. The human image data analysis system of claim 1, wherein the keypoint output unit uses a trained model for the keypoint extraction to input the feature amounts and the behavior type for one image, and outputs the keypoints of the person in the second human image data piece that is the subject of a current calculation process. A human image data analysis system including a computer device having a processor and a memory device,
The storage device includes:
storing a trained model for feature extraction generated by performing machine learning using first person image data including a person as an explanatory variable and a feature in the first person image data as a target variable;
storing a trained model for behavior analysis generated by performing machine learning using the feature amount extracted based on the first person image data as an explanatory variable and a behavior type of the person in the multiple first person image data in a time series as a target variable;
storing a trained model for keypoint extraction generated by performing machine learning using the feature amount and the behavior type as explanatory variables and a keypoint representing a posture of the person in the first person image data as a target variable;
The arithmetic processing device includes:
a feature extraction unit that extracts the feature from each of a plurality of second person image data by inputting a plurality of second person image data in a time series including a person , using the trained model related to the feature extraction stored in the storage device;
a behavior type output unit that outputs the behavior type of the person in the plurality of time-series second person image data by inputting the feature amounts for the plurality of images extracted by the feature amount extracting unit based on each of the plurality of time-series second person image data, using a trained model for the behavior analysis stored in the storage device;
a key point output unit that uses a trained model for the key point extraction stored in the storage device to input the feature amounts for one image selected from the plurality of second person image data in time series, the feature amounts being extracted by the feature amount extractor based on the one image of the second person image data selected from the plurality of second person image data in time series , and the behavior type being output by the behavior type output unit, and outputs the key points of the person in the selected one image of the second person image data;
A human image data analysis system comprising: A human image data analysis system including a computer device having a processor and a memory device,
The storage device includes:
storing a trained model for feature extraction generated by performing machine learning using first person image data including a person as an explanatory variable and a feature in the first person image data as a target variable;
storing a learned model for behavior analysis generated by performing machine learning using the feature amount for one image extracted based on the first person image data as an explanatory variable and a behavior type of the person in the multiple first person image data in a time series as a target variable;
storing a trained model for keypoint extraction generated by performing machine learning using the feature amount and the behavior type as explanatory variables and a keypoint representing a posture of the person in the first person image data as a target variable;
The arithmetic processing device includes:
a feature extraction unit that sequentially inputs a plurality of second person image data in a time series including a person , by using the trained model related to the feature extraction stored in the storage device, and sequentially extracts the feature from each of the plurality of second person image data;
a behavior type output unit that uses a trained model for the behavior analysis stored in the storage device to sequentially input the feature amounts for one image extracted by the feature amount extraction unit and to perform a recursive calculation using a result of a previous calculation process , thereby outputting the behavior type of the person in the one image of the second person image data that is the target of a current calculation process ;
a keypoint output unit that uses a trained model for the keypoint extraction stored in the storage device to input the feature amounts for one image extracted by the feature amount extraction unit and the behavior type output by the behavior type output unit, and outputs the keypoints of the person in the second person image data for one image that is the subject of a current calculation process ;
A human image data analysis system comprising: The human image data analysis system according to claim 1 , wherein the first human image data and the second human image data include image data of the human in a pose in which at least a torso of the human is turned sideways. The human image data analysis system according to any one of claims 1 to 6 , wherein the arithmetic processing device further includes a human image data generation unit that inputs original image data including a human area and a surrounding area, and generates the first human image data and the second human image data in which the human area is extracted from the original image data. The human image data analysis system according to claim 1 , wherein the key points include joint positions of the person and connection relationships that connect adjacent joint positions of the person.

Images