JP7635822B2 - Joint point detection device, joint point detection method, and program - Google Patents

Description

The present invention relates to a joint point detection device and a joint point detection method for detecting joint points of a living body from an image, and further to a program for implementing these.

In recent years, systems have been proposed that can estimate a person's posture from an image. Such systems are expected to be used in fields such as video surveillance and user interfaces. For example, if a person's posture can be estimated in an image surveillance system, it would be possible to estimate what a person captured on camera is doing, thereby improving surveillance accuracy. Furthermore, in a user interface, if a person's posture can be estimated, input would be possible through gestures.

For example, Non-Patent Document 1 discloses a system that estimates a person's posture, particularly the posture of a person's hand, from an image. The system disclosed in Non-Patent Document 1 first acquires image data including an image of the hand, inputs the acquired image data into a neural network that has machine-learned image features for each joint point, and outputs a heat map that represents the probability of the existence of each joint point using color and density.

Next, the system disclosed in Non-Patent Document 1 inputs the output heat map to a neural network that has learned the relationship between joint points and the corresponding heat map through machine learning. In addition, multiple such neural networks are prepared, and the output results from one neural network are input to another neural network. As a result, the positions of the joint points on the heat map are refined.

Patent Document 1 also discloses a system for estimating hand posture from an image. Like the system disclosed in Non-Patent Document 1, the system disclosed in Patent Document 1 also uses a neural network to estimate the coordinates of joint points.

JP 2017-191576 A

Christian Zimmermann, Thomas Brox, "Learning to Estimate 3D Hand Pose from Single RGB Images", [online], University of Freiburg, [Retrieved February 8, 2021], Internet <URL: https://openaccess.thecvf.com/content_ICCV_2017/papers/Zimmermann_Learning_to_Estimate_ICCV_2017_paper.pdf>

As described above, by using the systems disclosed in Non-Patent Document 1 or Patent Document 1, it is possible to estimate the coordinates of the joint points of a person's hand from an image. However, these systems have the problem that the estimation accuracy decreases as described below.

First, a living body has many joint points, and some joint points may not be captured in the image. In such cases, in the systems disclosed in Non-Patent Document 1 and Patent Document 1, the positions of the joint points not captured in the image may be incorrect in the heat map. As a result, when the positions of each joint point are refined by the neural network, they are dragged along by the incorrect positions of the joint points not captured in the image, and even the positions of the joint points captured in the image become incorrect.

An example of an object of the present invention is to provide a joint point detection device, a joint point detection method, and a program that can improve the accuracy of estimating the position of a joint point.

In order to achieve the above object, a joint point detection device according to one aspect of the present invention comprises:
a total feature output unit that outputs, for each joint point of the object, a first feature representing the joint point from the image data of the object;
a partial feature output unit that receives, for each of a plurality of specific joint points among the target joint points, the first feature representing joint points other than the specific joint points, and outputs a second feature representing the joint points other than the specific joint points by using a first machine learning model that performs machine learning on positional relationships between the joint points other than the specific joint points;
an integrated feature output unit that receives, for each of the target joint points, the second feature representing the joint point, and outputs a third feature representing the joint point by using a second machine learning model that performs machine learning on a position of the joint point;
The present invention is characterized in that it is provided with:

In order to achieve the above object, a joint point detection method according to one aspect of the present invention includes:
a total feature amount output step of outputting, for each joint point of the object, a first feature amount representing the joint point from the image data of the object;
a partial feature output step of outputting, for each of a plurality of specific joint points among the target joint points, a second feature representing the joint points other than the specific joint points using a first machine learning model that performs machine learning on positional relationships between the joint points other than the specific joint points, using the first feature representing the joint points other than the specific joint points as an input;
an integrated feature output step of outputting, for each of the target joint points, a third feature representing the joint point using a second machine learning model that performs machine learning on a position of the joint point using the second feature representing the joint point as an input;
The present invention is characterized by having the following.

Furthermore, in order to achieve the above object, a program according to one aspect of the present invention comprises:
On the computer,
a total feature amount output step of outputting, for each joint point of the object, a first feature amount representing the joint point from the image data of the object;
a partial feature output step of outputting, for each of a plurality of specific joint points among the target joint points, a second feature representing the joint points other than the specific joint points using a first machine learning model that performs machine learning on positional relationships between the joint points other than the specific joint points, using the first feature representing the joint points other than the specific joint points as an input;
an integrated feature output step of outputting, for each of the target joint points, a third feature representing the joint point using a second machine learning model that performs machine learning on a position of the joint point using the second feature representing the joint point as an input;
The present invention is characterized in that:

As described above, according to the present invention, it is possible to improve the accuracy of estimating the position of joint points.

FIG. 1 is a diagram showing a schematic configuration of a joint point detection device according to an embodiment. FIG. 2 is a diagram showing in more detail the configuration of the joint point detection device 10 according to the embodiment. FIG. 3 is a diagram illustrating the function of the total feature output unit in the embodiment. FIG. 4 is a diagram illustrating the function of the partial feature output unit in the embodiment. FIG. 5 is a diagram illustrating the function of the integrated feature output unit according to the embodiment. FIG. 6 is a flow chart showing the operation of the joint point detection device in the embodiment. FIG. 7 is a diagram showing a configuration of a joint point detection device 10 according to a modified example of the embodiment. FIG. 8 is a diagram illustrating the function of a total feature output unit in the modified example of the embodiment. FIG. 9 is a diagram illustrating the function of a partial feature output unit in the modification of the embodiment. FIG. 10 is a diagram illustrating functions of an integrated feature output unit in the modification of the embodiment. FIG. 11 is a block diagram showing an example of a computer that realizes the joint point detection device according to the embodiment.

(Embodiment)
Hereinafter, a joint point detection device, a joint point detection method, and a program according to an embodiment will be described with reference to FIGS.

[Device configuration]
First, a schematic configuration of a joint point detection device according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing a schematic configuration of a joint point detection device according to an embodiment.

The joint point detection device 10 in the embodiment shown in Figure 1 is a device for detecting joint points of an object, such as a living body, a robot, etc. As shown in Figure 1, the joint point detection device 10 includes a total feature output unit 11, a partial feature output unit 12, and an integrated feature output unit 13.

The total feature output unit 11 outputs a first feature representing each joint point of the target from the image data of the target.

The partial feature output unit 12 receives, for each of a plurality of specific joint points among the target joint points, a first feature representing a joint point other than the specific joint point, and outputs a second feature representing the joint point other than the specific joint point using a first machine learning model. The first machine learning model is a machine learning model that machine-learns the positional relationships between the joint points other than the specific joint points.

The integrated feature output unit 13 receives, for each target joint point, a second feature representing the joint point, and outputs a third feature representing the joint point using a second machine learning model. The second machine learning model is a machine learning model that learns the positions of the joint points by machine learning.

In this manner, in the embodiment, for each specific joint point, the second feature is output using the first machine learning model for the other joint points. Since the first machine learning model learns the positional relationships between joint points other than the specific joint point by machine learning, the second feature can appropriately indicate the positions of the other joint points when the specific joint point is not visible. And since the third feature, which is the final feature for each joint point, is obtained from this second feature, according to the embodiment, it is possible to improve the estimation accuracy of the position of each joint point.

Next, the configuration and functions of the joint point detection device 10 in the embodiment will be specifically described using Figures 2 to 5. Figure 2 is a diagram showing the configuration of the joint point detection device 10 in the embodiment in more detail. Figure 3 is a diagram explaining the functions of the total feature output unit in the embodiment. Figure 4 is a diagram explaining the functions of the partial feature output unit in the embodiment. Figure 5 is a diagram explaining the functions of the integrated feature output unit in the embodiment.

As shown in FIG. 2, in an embodiment, the joint point detection device 10 includes, in addition to the total feature output unit 11, partial feature output unit 12, and integrated feature output unit 13 described above, a joint point detection unit 14 and a memory unit 15.

The memory unit 15 stores a first machine learning model 16 and a second machine learning model 17. In the embodiment, the first machine learning model 16 and the second machine learning model 17 are constructed by a convolutional neural network (CNN). In the embodiment, the memory unit 15 also stores a third machine learning model 18, which will be described later. The third machine learning model 18 is also constructed by a CNN. Note that hereinafter, the machine learning model will also be referred to as "CNN".

Furthermore, hereinafter, an example will be described in which the target of joint point detection is a human hand. Note that in the embodiments, the target of joint point detection is not limited to a human hand, but may be the entire human body or other parts. Also, the target of joint point detection may be anything that has joint points, and may be something other than a human, for example, a robot. Furthermore, in the embodiments, in addition to joint points, parts other than joint points, for example characteristic parts such as fingertips, may also be detected.

In addition, in the embodiment, a heat map is used as the first feature amount, the second feature amount, and the third feature amount. A heat map is a map that represents the possibility of the existence of a joint point on an image, and for example, the possibility of the existence of a joint point can be represented by a shade of color. Note that something other than a heat map may be used as the first feature amount, the second feature amount, and the third feature amount. This point will be described later in a modified example.

In the embodiment, the total feature output unit 11 first acquires the target image data 20. Then, as shown in Fig. 3, the total feature output unit 11 inputs the image data 20 to the third CNN 18, which outputs a first heat map 21 as a first feature of the joint point. Also, in the example of Fig. 3, multiple first heat maps 21 are output for each joint point on the image data 20.

The third CNN 18 is a machine learning model that learns the relationship between joint points on an image and a heat map. In the machine learning of the third CNN 18, image data of the joint points and a heat map that is the correct answer are used as training data. The machine learning of the third CNN 18 is performed by updating parameters so that the difference between the output result (heat map) of the image data that is the training data and the heat map that is the correct answer is reduced.

In the embodiment, the partial feature output unit 12 uses a first CNN 16 constructed for each specific joint point as shown in Fig. 4. In the example of Fig. 4, the specific joint points are the first joint, the second joint, and the base of the finger, and a first CNN 16 is constructed for each of these joint points. Also, in the example of Fig. 4, the first CNN 16 is constructed for the fingertips in addition to the specific joint points.

As described above, the first CNN 16 is a machine learning model that machine-learns the positional relationships between joint points other than specific joint points. In the machine learning of the first CNN 16, the heat map of joint points other than the corresponding specific joint point (or fingertip) and the correct heat map become training data. The machine learning of the first CNN 16 is also performed by updating parameters so that the difference between the output result (heat map) of the image data that is the training data and the correct heat map becomes small.

The partial feature output unit 12 inputs, to each of the first CNNs 16, the first heat map 21 of the joint points other than the corresponding fingertip or specific joint point, out of the first heat map 21 input from the full feature output unit 11. As a result, the second heat map 22 other than the fingertip, the second heat map 22 other than the first joint, the second heat map 22 other than the second joint, and the second heat map 22 other than the base of the finger are output as second features.

In the embodiment, the integrated feature output unit 13 uses a second CNN 17 constructed for each joint point as shown in Figure 5. In the example of Figure 5, the second CNN 17 is constructed for each of the 21 joint points of the hand.

As described above, the second CNN 17 is a machine learning model that machine-learns the positions of corresponding joint points. In the machine learning of the second CNN 17, the heat map of the corresponding joint points and the correct heat map are also used as training data. The machine learning of the second CNN 17 is also performed by updating parameters so that the difference between the output result (heat map) of the image data that is the training data and the correct heat map is reduced.

The integrated feature output unit 13 merges the second heat maps 22 output by each of the first CNNs 16, and inputs only the second heat maps 22 of the corresponding joint points to each of the second CNNs 17. As a result, a third heat map 23 is output for each joint point as a third feature.

Then, the joint point detection unit 14 obtains a third heat map 23 for each joint point of the target hand. The joint point detection unit 14 then uses the third heat map 23 for each joint point to detect the coordinates of each of the target joint points.

Specifically, the joint point detection unit 14 identifies the location of highest density in the third heat map 23 for each joint point, and detects the two-dimensional coordinates of the identified location on the image. Furthermore, if there are multiple third heat maps 23 for each joint point, the joint point detection unit 14 identifies the two-dimensional coordinates of the location of highest density for each third heat map 23, and further calculates the average of each of the identified two-dimensional coordinates, and sets the calculated average coordinates as the final coordinates.

[Device Operation]
Next, the operation of the joint point detection device 10 in the embodiment will be described with reference to Fig. 6. Fig. 6 is a flow diagram showing the operation of the joint point detection device in the embodiment. In the following description, Figs. 1 to 5 will be referred to as appropriate. Also, in the embodiment, a joint point detection method is implemented by operating the joint point detection device 10. Therefore, the description of the joint point detection method in the embodiment will be replaced with the following description of the operation of the joint point detection device 10.

As shown in FIG. 6, first, the full feature output unit 11 acquires target image data 20, inputs the acquired image data 20 to the third machine learning model 18, and outputs a first heat map 21 representing each joint point (step A1).

Next, the partial feature output unit 12 inputs to each first CNN 16 the first heat map 21 of the corresponding fingertip or joint point other than the specific joint point from the first heat map 21 output in step A1, and outputs a second heat map 22 representing the fingertip or joint point other than the specific joint point (step A2).

Next, the integrated feature output unit 13 merges the second heat maps 22 output by each first CNN 16 in step A2, and inputs only the second heat maps 22 of the corresponding articulation points to each second CNN 17, causing them to output a third heat map 23 representing the corresponding articulation points (step A3).

Next, the joint point detection unit 14 detects the coordinates of each target joint point using the third heat map 23 for each joint point output in step A3 (step A4).

As described above, in the embodiment, for example, since there is a high possibility that the fingertip, first joint, second joint, and base will not be visible in the image, the second feature amount is output using the first feature amount of the other parts. Then, the third feature amount of each joint point is obtained from the second feature amount output in this way, so that the finally obtained coordinates of each joint point are accurate values. In other words, according to the embodiment, it is possible to improve the estimation accuracy of the position of each joint point.

[Modification]
Next, a modified example of the embodiment will be described with reference to Figs. 7 to 10. Fig. 7 is a diagram showing the configuration of a joint point detection device 10 in a modified example of the embodiment. Fig. 8 is a diagram explaining the function of a total feature output unit in the modified example of the embodiment. Fig. 9 is a diagram explaining the function of a partial feature output unit in the modified example of the embodiment. Fig. 10 is a diagram explaining the function of an integrated feature output unit in the modified example of the embodiment.

As shown in FIG. 7, in this modification, the first machine learning model 31 and the second machine learning model 32 are constructed by a graph convolution network (GCN). A GCN is a network that takes a graph structure composed of multiple nodes as input and performs convolution processing using adjacent nodes. In this modification, the machine learning model is also written as "GCN". Note that the third machine learning model is the third CNN 18 shown in FIG. 2 and FIG. 3.

In addition, in this modified example, coordinate values indicating the position of the joint point on the image are used as the first feature and the second feature. Specifically, since a GCN is used, the feature output from the GCN has a graph structure in which each node represents the coordinate value of the joint point.

In addition, in this modified example, since coordinate values are output as feature quantities, the joint point detection unit 14 shown in Fig. 2 is omitted in the joint point detection device 10. Other than the points described above, the joint point detection device 10 in this modified example also has the configuration described in the embodiment.

In this modified example, as in the above example, the total feature output unit 11 inputs image data 20 to the third CNN 18 as shown in Figure 8, and outputs a first heat map 21 as the first feature of the joint point.

However, in this modified example, the total feature output unit 11 calculates the coordinate values (first coordinate values) 41 of each target joint point using the output first heat map 21. The calculation of the coordinate values is performed in the same manner as the processing in the joint point detection unit 14 described above. Then, the total feature output unit 11 outputs the first coordinate values 41 for each joint point as a graph structure.

In this modified example, the partial feature output unit 12 uses a first GCN 31 constructed for each specific joint point, as shown in Fig. 9. In the example of Fig. 9, the specific joint points are the first joint, the second joint, and the base of the finger, and a first GCN 31 is constructed for each of these joint points. Furthermore, the first GCN 31 is also constructed for the fingertips in addition to the specific joint points.

The first GCN 31 is a machine learning model that machine-learns the positional relationships between joint points other than specific joint points. In the machine learning of the first GCN 31, the coordinate values of joint points other than the corresponding specific joint point (or fingertip) and the correct coordinate values become training data. The machine learning of the first GCN 31 is also performed by updating parameters so that the difference between the output result (coordinate values) of the image data that becomes the training data and the correct coordinate values becomes small.

The partial feature output unit 12 inputs, to each of the first GCNs 31, the first coordinate values 41 of the corresponding fingertip or joint points other than a specific joint point, among the first coordinate values 41 input from the full feature output unit 11. As a result, the second coordinate values 42 other than the fingertip, the second coordinate values 42 other than the first joint, the second coordinate values 42 other than the second joint, and the second coordinate values 42 other than the base of the finger are output as second features.

In this modified example, the integrated feature output unit 13 uses a second GCN 32 constructed for each joint point, as shown in Fig. 10. In the example of Fig. 10, the second GCN 32 is constructed for each of the 21 joint points of the hand.

As described above, the second GCN 32 is a machine learning model that machine-learns the positions of corresponding joint points. In the machine learning of the second GCN 32, the coordinate values of the corresponding joint points and the coordinate values that are the correct answer become training data. The machine learning of the second GCN 32 is also performed by updating parameters so that the difference between the output result (coordinate values) of the image data that is the training data and the coordinate values that are the correct answer becomes small.

The integrated feature output unit 13 merges the second coordinate values 42 output by each of the first GCNs 31, and inputs only the second coordinate values 42 of the corresponding joint points to each of the second GCNs 32. As a result, a third coordinate value 43 is output for each joint point as a third feature. The output third coordinate value 43 for each joint point is detected as the coordinate of each joint point.

In this way, in this modified example, GCN is used as the machine learning model and coordinate values are used as the features, but in this case too, the second features are output using the first features of parts other than those that are likely to be invisible in the image. Therefore, in this modified example too, it is possible to improve the estimation accuracy of the position of each joint point.

[program]
The program in the embodiment may be a program that causes a computer to execute steps A1 to A4 shown in Fig. 6. By installing and executing this program in a computer, the joint point detection device and joint point detection method in the embodiment can be realized. In this case, the processor of the computer functions as a total feature output unit 11, a partial feature output unit 12, an integrated feature output unit 13, and a joint point detection unit 14, and performs processing.

In addition, in this embodiment, the memory unit 15 may be realized by storing the data files that constitute it in a storage device such as a hard disk provided in the computer, or it may be realized by a storage device of another computer.

In addition to general-purpose PCs, examples of computers include smartphones and tablet terminal devices.

The program in the embodiment may be executed by a computer system constructed by multiple computers. In this case, for example, each computer may function as any one of the total feature output unit 11, the partial feature output unit 12, the integrated feature output unit 13, and the joint point detection unit 14.

[Physical configuration]
Here, a computer that realizes the joint point detection device 10 by executing a program in the embodiment will be described with reference to Fig. 11. Fig. 11 is a block diagram showing an example of a computer that realizes the joint point detection device in the embodiment.

11, the computer 110 includes a CPU (Central Processing Unit) 111, a main memory 112, a storage device 113, an input interface 114, a display controller 115, a data reader/writer 116, and a communication interface 117. These components are connected to each other via a bus 121 so as to be able to communicate data with each other.

Furthermore, the computer 110 may include a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array) in addition to or instead of the CPU 111. In this embodiment, the GPU or FPGA can execute the program in the embodiment.

The CPU 111 loads the program in the embodiment, which is composed of a group of codes stored in the storage device 113, into the main memory 112 and executes each code in a predetermined order to perform various calculations. The main memory 112 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory).

In addition, the program in the embodiment is provided in a state stored in a computer-readable recording medium 120. The program in the embodiment may be distributed over the Internet connected via the communication interface 117.

Specific examples of the storage device 113 include a hard disk drive and a semiconductor storage device such as a flash memory. The input interface 114 mediates data transmission between the CPU 111 and input devices 118 such as a keyboard and a mouse. The display controller 115 is connected to the display device 119 and controls the display on the display device 119.

The data reader/writer 116 mediates data transmission between the CPU 111 and the recording medium 120, reads programs from the recording medium 120, and writes the results of processing in the computer 110 to the recording medium 120. The communication interface 117 mediates data transmission between the CPU 111 and other computers.

Specific examples of recording medium 120 include general-purpose semiconductor storage devices such as CF (Compact Flash (registered trademark)) and SD (Secure Digital), magnetic recording media such as a flexible disk, or optical recording media such as a CD-ROM (Compact Disk Read Only Memory).

In addition, the joint point detection device 10 in the embodiment can be realized by using hardware corresponding to each part, rather than a computer with a program installed. Furthermore, the joint point detection device 10 may be realized in part by a program and the remaining part by hardware.

Some or all of the above-described embodiments can be expressed by (Appendix 1) to (Appendix 12) described below, but are not limited to the following descriptions.

(Appendix 1)
a total feature output unit that outputs, for each joint point of the object, a first feature representing the joint point from the image data of the object;
a partial feature output unit that receives, for each of a plurality of specific joint points among the target joint points, the first feature representing joint points other than the specific joint points, and outputs a second feature representing the joint points other than the specific joint points by using a first machine learning model that performs machine learning on positional relationships between the joint points other than the specific joint points;
an integrated feature output unit that receives, for each of the target joint points, the second feature representing the joint point, and outputs a third feature representing the joint point by using a second machine learning model that performs machine learning on a position of the joint point;
A joint point detection device comprising:

(Appendix 2)
2. The joint point detection device according to claim 1,
The apparatus further includes a joint point detection unit that detects coordinates of each of the joint points of the object by using the third feature amount for each of the joint points of the object.
A joint point detection device characterized by:

(Appendix 3)
2. The joint point detection device according to claim 1,
The first machine learning model and the second machine learning model are constructed by a convolutional neural network;
each of the first feature amount, the second feature amount, and the third feature amount includes a heat map representing a possibility that a joint point exists on the image;
A joint point detection device comprising:

(Appendix 4)
2. The joint point detection device according to claim 1,
The first machine learning model and the second machine learning model are constructed by a graph convolutional network;
each of the first feature amount, the second feature amount, and the third feature amount includes a coordinate value indicating a position of a joint point on an image;
A joint point detection device comprising:

(Appendix 5)
a total feature amount output step of outputting, for each joint point of the object, a first feature amount representing the joint point from the image data of the object;
a partial feature output step of outputting, for each of a plurality of specific joint points among the target joint points, a second feature representing the joint points other than the specific joint points using a first machine learning model that performs machine learning on positional relationships between the joint points other than the specific joint points, using the first feature representing the joint points other than the specific joint points as an input;
an integrated feature output step of outputting, for each of the target joint points, a third feature representing the joint point using a second machine learning model that performs machine learning on a position of the joint point using the second feature representing the joint point as an input;
A joint point detection method comprising:

(Appendix 6)
6. The joint point detection method according to claim 5, further comprising:
The method further includes a joint point detection step of detecting coordinates of each of the joint points of the object by using the third feature amount for each of the joint points of the object.
A joint point detection method comprising:

(Appendix 7)
6. The joint point detection method according to claim 5, further comprising:
The first machine learning model and the second machine learning model are constructed by a convolutional neural network;
each of the first feature amount, the second feature amount, and the third feature amount includes a heat map representing a possibility that a joint point exists on the image;
A joint point detection method comprising:

(Appendix 8)
6. The joint point detection method according to claim 5, further comprising:
The first machine learning model and the second machine learning model are constructed by a graph convolutional network;
each of the first feature amount, the second feature amount, and the third feature amount includes a coordinate value indicating a position of a joint point on an image;
A joint point detection method comprising:

(Appendix 9)
On the computer,
a total feature amount output step of outputting, for each joint point of the object, a first feature amount representing the joint point from the image data of the object;
a partial feature output step of outputting, for each of a plurality of specific joint points among the target joint points, a second feature representing the joint points other than the specific joint points using a first machine learning model that performs machine learning on positional relationships between the joint points other than the specific joint points, using the first feature representing the joint points other than the specific joint points as an input;
an integrated feature output step of outputting, for each of the target joint points, a third feature representing the joint point using a second machine learning model that performs machine learning on a position of the joint point using the second feature representing the joint point as an input;
A program to execute .

(Appendix 10)
10. The program according to claim 9,
The computer includes:
a joint point detection step of detecting coordinates of each of the joint points of the object using the third feature amount for each of the joint points of the object;
A program characterized by:

(Appendix 11)
10. The program according to claim 9,
The first machine learning model and the second machine learning model are constructed by a convolutional neural network;
each of the first feature amount, the second feature amount, and the third feature amount includes a heat map representing a possibility that a joint point exists on the image;
A program characterized by:

(Appendix 12)
10. The program according to claim 9,
The first machine learning model and the second machine learning model are constructed by a graph convolutional network;
each of the first feature amount, the second feature amount, and the third feature amount includes a coordinate value indicating a position of a joint point on an image;
A program characterized by:

Although the present invention has been described above with reference to the embodiment, the present invention is not limited to the above embodiment. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

This application claims priority based on Japanese Patent Application No. 2021-029410, filed on February 26, 2021, the disclosure of which is incorporated herein in its entirety.

As described above, the present invention can improve the accuracy of estimating the position of joint points. The present invention is useful in fields where posture detection of objects with joint points, such as humans and robots, is required. Specific fields include video surveillance and user interfaces.

REFERENCE SIGNS LIST 10 Articulation point detection device 11 Total feature output unit 12 Partial feature output unit 13 Integrated feature output unit 14 Articulation point detection unit 15 Storage unit 16 First machine learning model (CNN)
17. Second Machine Learning Model (CNN)
18. The third machine learning model (CNN)
20 Image data 21 First heat map 22 Second heat map 23 Third heat map 31 First machine learning model (GCN)
32 Second machine learning model (GCN)
41 First coordinate value 42 Second coordinate value 43 Third coordinate value 110 Computer 111 CPU
112 Main memory 113 Storage device 114 Input interface 115 Display controller 116 Data reader/writer 117 Communication interface 118 Input device 119 Display device 120 Recording medium 121 Bus

Claims (6)

a total feature value output means for outputting, for each joint point of the object, a first feature value representing the joint point from the image data of the object;
a partial feature output means for outputting, for each of a plurality of specific joint points among the target joint points, a second feature representing a joint point other than the specific joint point using a first machine learning model that corresponds to the specific joint point and that performs machine learning on a positional relationship between the joint points other than the specific joint point;
an integrated feature output means for outputting, for each of the target joint points, a third feature representing the joint point using a second machine learning model that corresponds to the joint point and that has machine-learned a position of the joint point;
A joint point detection device comprising: The joint point detection device according to claim 1,
The method further includes a joint point detection means for detecting coordinates of each of the joint points of the object by using the third feature amount for each of the joint points of the object.
A joint point detection device characterized by: The joint point detection device according to claim 1,
The first machine learning model and the second machine learning model are constructed by a convolutional neural network;
each of the first feature amount, the second feature amount, and the third feature amount includes a heat map representing a possibility that a joint point exists on the image;
A joint point detection device comprising: The joint point detection device according to claim 1,
The first machine learning model and the second machine learning model are constructed by a graph convolutional network;
each of the first feature amount, the second feature amount, and the third feature amount includes a coordinate value indicating a position of a joint point on an image;
A joint point detection device comprising: outputting, for each joint point of the object, a first feature amount representative of the joint point from the image data of the object;
using the first feature amount representing a joint point other than the specific joint point as an input for each of a plurality of specific joint points among the target joint points, and outputting a second feature amount representing the joint point other than the specific joint point using a first machine learning model that corresponds to the specific joint point and that has machine-learned a positional relationship between the joint points other than the specific joint point;
outputting, for each of the target joint points, a third feature amount representing the joint point using a second machine learning model that corresponds to the joint point and that has machine-learned a position of the joint point, using the second feature amount representing the joint point as an input;
A joint point detection method comprising: On the computer,
outputting, for each joint point of the object, a first feature amount representing the joint point from the image data of the object;
a first machine learning model that receives, as an input, the first feature amount representing a joint point other than the specific joint point for each of a plurality of specific joint points among the target joint points, and that corresponds to the specific joint point and that has machine-learned a positional relationship between the joint points other than the specific joint point, and outputs a second feature amount representing the joint point other than the specific joint point;
outputting, for each of the target joint points, a third feature amount representing the joint point using a second machine learning model that corresponds to the joint point and that has machine-learned a position of the joint point, using the second feature amount representing the joint point as an input;
program.

Images