CN107577451A - More Kinect human skeletons coordinate transformation methods and processing equipment, readable storage medium storing program for executing - Google Patents

Abstract

The invention relates to the field of computer graphics vision, and proposes a multi-Kinect human body skeleton coordinate transformation method, aiming at solving the problem of coordinate transformation between multiple Kinects in a human body tracking environment. The method includes: receiving multi-frame skeleton data of the same human body sent by each terminal device, determining the weighted average skeleton data of the human body skeleton of each terminal device; calculating the credibility of the data of each terminal device, and determining the terminal device with the highest credibility It is the reference terminal equipment, and the coordinate system of the reference terminal equipment is the reference coordinate system; determine the Euler angle between the reference coordinate system and the non-reference coordinate system according to the coordinates of the human skeleton in the reference coordinate system and the coordinates in the non-reference coordinate system and translation variables; determine the transformation matrix between the reference coordinate system and the non-reference coordinate system according to the Euler angle and the translation variable; use the above transformation matrix to transform the skeleton data of each terminal device into the reference coordinate system. This method realizes the continuous and stable transformation of the coordinates between multiple Kinect human skeletons.

Description

Multi-Kinect human skeleton coordinate transformation method, processing equipment, and readable storage medium

technical field

The invention relates to the field of computer graphics vision, in particular to the field of computer graphics processing, in particular to a multi-Kinect human skeleton coordinate transformation method, processing equipment, and storage media.

Background technique

With the development of computer graphics and vision technology and the development of human-computer interaction technology, it is becoming more and more important to display the detected or monitored object or person's posture or action in a timely and correct manner. Based on the Microsoft Kinect sensor, it can obtain the image information of the human body in real time, capture and analyze the human body motion information, and control related equipment or control the action of the robot according to the obtained human body motion information.

Due to the limitation of kinect capture range and angle, a single kinect captures information due to the occlusion relationship between the object itself and objects, resulting in the lack of depth and visualization information, which greatly reduces the accuracy and effect of capture. A motion capture system composed of multiple Kinects can provide a larger field of view. When the human body disappears due to occlusion within the field of view of one Kinect, the system can still continue to capture the movement of the human body through other Kinects. Therefore, it is necessary to transform the skeleton data of the human body captured by multiple kinects into a unified coordinate system, but to convert the data in different coordinate systems into the same coordinate system, due to the different positions and angles of the Kinects that capture the skeleton data, there may be The problem of large error and inconvenient use when converting data in the same reference coordinate system.

At present, the KSCC calibration algorithm proposed by Wei Tao et al. is used for the calibration of users at any position in the remote training system for the application of multi-Kinect sensor coordinate transformation. This method first needs to collect the initial position data of the remote user, and use the initial data to calculate the initial center position coordinates of the user and the initial angle between the user and Kinect, and then all skeletons will be transformed to a unified world coordinate system. However, the current motion capture system composed of multiple Kinects has problems such as large errors in captured human bodies, Kinect sensors can only be placed horizontally, and cannot be used normally when the human body is turned around.

Contents of the invention

In order to solve the above-mentioned problems in the prior art, that is, in order to solve the large error caused by the coordinate conversion between a plurality of Kinect sensors arranged in different positions, the Kinect sensor can only be placed horizontally, and cannot be used normally when the human body turns around, etc. Problem, the present invention adopts following technical scheme to solve the above-mentioned problem:

In the first aspect, the present application provides a multi-Kinect human body skeleton coordinate transformation method, the method comprising: obtaining skeleton data captured by multiple terminal devices; using a trained coordinate transformation matrix to convert the above skeleton data, wherein, training the above The transformation matrix includes: receiving multi-frame skeleton data of the same human body sent by multiple terminal devices, determining the weighted average skeleton data of the skeleton data of each of the above-mentioned terminal devices according to the above-mentioned skeleton data; The reliability of the terminal equipment, determine the terminal equipment with the highest reliability from the above-mentioned terminal equipment as the reference terminal equipment, determine the coordinate system of the above-mentioned reference terminal equipment as the reference coordinate system, and the start-stop designation except the above-mentioned reference terminal equipment The equipment determination fee refers to the terminal equipment, and the coordinate system of the above-mentioned non-reference terminal equipment is a non-reference coordinate system; according to the coordinates of the human skeleton corresponding to the above-mentioned weighted average skeleton data in the above-mentioned reference coordinate system and the coordinates in the above-mentioned non-reference coordinate system Calculate the Euler angle and translation variable between the reference coordinate system and the above-mentioned non-reference coordinate system by the difference between them; the above-mentioned Euler angle between the coordinate axes of the above-mentioned reference coordinate system and the above-mentioned non-reference coordinate system constitutes the rotation Matrix, the above-mentioned translation variable between the coordinate axes of the above-mentioned reference coordinate system and the above-mentioned non-reference coordinate system constitutes a translation matrix, and the product of the above-mentioned rotation matrix and translation matrix is used as the difference between the above-mentioned reference coordinate system and the above-mentioned non-parametric coordinate system transformation matrix between them.

In some examples, the above-mentioned skeleton data includes the coordinates of each joint point constituting the human skeleton and the reliability of the above-mentioned joint point coordinates.

In some examples, the above-mentioned determination of the weighted average skeleton data of the skeleton data of each of the above-mentioned terminal devices according to the above-mentioned skeleton data includes: separately collecting multiple frames of skeleton data of the human body skeleton through multiple terminal devices; Frame the joint point coordinates of the above-mentioned skeleton data and the reliability of the above-mentioned joint point coordinates, take the above-mentioned joint point coordinates as the weight of the above-mentioned joint point coordinates, and calculate the joints of each joint point of the human skeleton detected by the above-mentioned terminal equipment Point weighted average; the human skeleton data determined by taking the weighted average of each joint point as the joint point data of each joint point is the weighted average skeleton data of the human skeleton detected by the terminal device.

In some examples, the calculation of the credibility of each terminal device based on the skeleton data captured by each terminal device includes: obtaining the credibility of each joint point of the multi-frame skeleton data collected by each terminal device; calculating each The average value of the credibility of each joint point of the skeleton data collected by the above-mentioned terminal equipment; the average value of the credibility of each joint point of all the joint points that constitute the human skeleton collected by the operation of the above-mentioned terminal equipment; The result of the summing operation is determined as the reliability of the terminal device.

In some examples, according to the difference between the coordinates of the human skeleton corresponding to the weighted average skeleton data in the above-mentioned reference coordinate system and the coordinates in the above-mentioned non-reference coordinate system, calculate the difference between the reference coordinate system and each non-coordinate system Euler angles and translation variables between, including: calculating the difference between the weighted average skeleton data of the above-mentioned human skeleton collected by the above-mentioned reference terminal device and the weighted average skeleton data of the above-mentioned human skeleton collected by the above-mentioned non-reference terminal device; The difference is the difference skeleton data of the above-mentioned human skeleton; the angle between the coordinates of the above-mentioned difference skeleton data in the above-mentioned reference coordinate system and the reference axes of the above-mentioned reference coordinate system is Euler angle; the calculation of the above-mentioned difference skeleton data in the above-mentioned reference The distances between the coordinates in the coordinate system and the reference axes of the above-mentioned reference coordinate system are translation variables.

In some examples, the determination of the transformation matrix between the reference coordinate system and the coordinate system of each of the above-mentioned non-reference terminal devices according to the above-mentioned Euler angles and translation variables includes: respectively determining the rotation transformation matrix according to the above-mentioned Euler angles and translation variables and a translation matrix; determine the product of the above-mentioned rotation transformation matrix and the above-mentioned translation matrix as a transformation matrix.

In some examples, the above-mentioned determination of the transformation matrix between the reference coordinate system and the coordinate systems of each of the above-mentioned non-reference terminal devices according to the above-mentioned Euler angles and translation variables includes: determining the three axes of the above-mentioned reference coordinate system according to the above-mentioned Euler angles Euler angles θ _x , θ _y , θ _z of xyz, and the rotation transformation matrix obtained is R _i (θ _x , θ _y , θ _z )=R _iz (θ _z )·R _iy (θ _y )·R _ix (θ _x ); Determine the variables x, y, and z of the three axes of the above-mentioned reference coordinate system according to the above-mentioned translation variable, and can draw the translation matrix T _i (x, y, z); with the product of the above-mentioned rotation transformation matrix and the above-mentioned translation matrix is the transformation matrix M _i .

In some examples, the above method further includes: after calculating the weighted average skeleton, judging whether the above-mentioned human skeleton is front or back; if the judging result is negative, exchanging left and right coordinates of joint points on both sides of the above-mentioned human skeleton.

In a second aspect, the application provides a multi-Kinect human skeleton coordinate transformation processing device, which includes: one or more processors, adapted to execute various programs; storage means, used to store one or more programs, the above-mentioned The program is adapted to be loaded and executed by the processor to implement the multi-Kinect human skeleton coordinate transformation method described in any example of the first aspect above.

In a third aspect, the present application provides a readable storage medium, the readable storage medium stores a computer program, and when the program is executed by a processor, it realizes the multi-Kinect human skeleton coordinate transformation described in any example in the above-mentioned first aspect method.

The Kinect human skeleton coordinate transformation method and processing equipment provided by this application determine the terminal device with the highest reliability from the received multi-frame skeleton data, and designate the coordinate system of the terminal device with the highest reliability as the reference coordinate system, Compare the difference between the coordinates of the skeleton data in the reference coordinate system and the coordinates of the non-reference terminal, determine the Euler angle and translation variable between the reference coordinate system and the coordinate system of the non-reference terminal device, and then determine A transformation matrix is used to implement coordinate transformation between the coordinate system of each non-reference terminal and the reference coordinate system.

Description of drawings

FIG. 1 is an exemplary system architecture diagram to which the present application can be applied;

Fig. 2 is the flow chart of an embodiment according to the many Kinect human skeleton coordinate transformation methods of the present application;

Fig. 3 is the flow chart of another embodiment according to the multi-Kinect human skeleton coordinate transformation method of the present application;

Figure 4 is an image of the skeleton data of the same human body captured by terminal devices at two different positions;

Fig. 5 is an image displayed by the skeleton data of the same human skeleton transformed into the reference coordinate system through the transformation matrix.

detailed description

Preferred embodiments of the present invention are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present invention, and are not intended to limit the protection scope of the present invention.

It should be noted that, in the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and embodiments.

FIG. 1 shows an exemplary system architecture of an embodiment in which the multi-Kinect human skeleton coordinate transformation method or the multi-Kinect human skeleton coordinate transformation processing device of the present application can be applied.

As shown in FIG. 1 , the system architecture may include terminal devices 101 , 102 , 103 , a network 104 and a server 105 . The network 104 is used as a medium for providing communication links between the terminal devices 101 , 102 , 103 and the server 105 . Network 104 may include various connection types, such as wires, wireless communication links, or fiber optic cables, among others.

The terminal devices 101, 102, and 103 may perform information interaction with the server 105 through the network 104, so as to receive or send information and the like. The terminal devices 101 , 102 , and 103 can exchange information through the network 104 .

The terminal devices 101, 102, 103 may be various electronic devices with display screens and supporting network communication, including but not limited to smart phones, tablet computers, laptop computers, desktop computers and so on. It should be noted that the terminal device includes sensors for image recognition and voice recognition, such as Kinect 2.0 sensor, wherein the Kinect 2.0 sensor is a 3D somatosensory peripheral peripheral camera released by Microsoft in 2014.

Server 105 may be a processor or server that provides various services, such as an image processing server that performs data analysis on information sent or provided by terminal devices 101, 102, 103, and realizes coordinate transformation between images of different terminal devices. The processing server can analyze and process the received information, and generate processing results (for example, transform each coordinate of the human skeleton obtained by each terminal device into the reference coordinate system, and display the skeleton data in the reference coordinate system) in the The user interface is displayed. It should be noted that, in this application, the server 105 may be a separately set server, or may be selected from the terminal devices 101, 102, 103 or designated as a server.

It should be noted that the multi-Kinect human skeleton coordinate transformation method provided in the embodiment of the present application is generally executed by the server 105 , and correspondingly, the multi-Kinect human skeleton coordinate transformation processing device is generally set in the server 105 .

It should be understood that the numbers of terminal devices, networks and servers in Fig. 1 are only illustrative. According to the implementation needs, there can be any number of terminal devices, networks and servers.

Continuing to refer to FIG. 2 , FIG. 2 shows the flow of an embodiment of the multi-Kinect human skeleton coordinate transformation method according to the present application. This multi-Kinect human skeleton coordinate transformation method comprises the following steps:

Step 201, receiving skeleton data captured by multiple terminal devices.

In this embodiment, the electronic device on which the multi-Kinect human body skeleton coordinate transformation method runs (such as the server shown in Figure 1) can receive the skeleton data of the human body sent by the terminal device from the terminal device through a wired connection or a wireless connection. . Wherein, the terminal device can obtain the skeleton data of the human body through the Kinect sensor connected thereto, and send the obtained skeleton data to the server. The above-mentioned skeleton data refers to the image data of the human body detected by the Kinect sensor, and the data when the above-mentioned Kinect sensor is mapped to the coordinate system of the Kinect sensor according to each frame of image data detected.

Step 202, using the trained coordinate transformation matrix to transform the above skeleton data.

In this embodiment, the trained coordinate change matrix is used to convert the skeleton data captured by each terminal device into coordinate data in the same reference coordinate system, so as to track the human body or the movement of the human body at different terminals. Keeping the positions and angles of the above-mentioned terminal devices unchanged, any image data captured by the above-mentioned terminal devices (skeleton data of human skeletons of one or more people in the tracking area) can be transformed into the reference coordinate system. Through the transformation matrix between each terminal device and the reference terminal device, the skeleton data of the human skeleton captured by the above-mentioned terminal devices are transformed into coordinates in the reference coordinate system.

Wherein, the above steps of training the coordinate transformation matrix include:

Sub-step 2021, receiving multiple frames of skeleton data of the same human body sent by multiple terminal devices, and determining the weighted average skeleton data of the skeleton data of each of the above-mentioned terminal devices according to the above-mentioned skeleton data.

In this sub-step, the server receives the skeleton data of the same human body from multiple terminal devices. The above-mentioned skeleton data refers to the image data of the human body detected by the Kinect sensor, and the above-mentioned Kinect sensor determines the above-mentioned each frame of image data according to the detected data when each frame of image data is mapped to the Kinect sensor coordinate system by weighting the data. The weighted average of the number of skeletons of human skeletons captured by the terminal device. The above-mentioned skeleton data may be coordinate data representing the torso of the above-mentioned human skeleton and each joint point in the Kinect sensor coordinate system when each frame of detected image data is mapped to the Kinect sensor coordinate system.

In some optional implementation manners of this embodiment, the above-mentioned skeleton data includes the coordinates of each joint point composing the human skeleton and the credibility of the above-mentioned joint point coordinates. The above-mentioned skeleton data includes the coordinates of the joint points of the above-mentioned human skeleton and the tracking states of the above-mentioned joint points. The states are respectively tracked, estimated, and untracked, and the joint points in the above three states are allocated with decreasing degrees of reliability. . In some specific examples, the terminal device may pre-process the skeleton data representing the torso and joint points of the human skeleton, and convert it into coordinates in the coordinate system of the terminal device.

As an example, each of the above-mentioned terminal devices is connected to a Kinect sensor, and the above-mentioned server may be a computer provided separately, or any computer of the above-mentioned terminal devices may serve as the server. The TCP/IP protocol can be used to communicate between the above-mentioned server and the above-mentioned terminal equipment, or between multiple above-mentioned terminal equipment. Each frame of data collected by the above-mentioned Kinect sensor can use the BodyFrame class to package the skeleton information of the human body. The Body class encapsulates the coordinates of the 25 skeleton joint points of the target human body and the tracking status of the above-mentioned skeleton joint points. , speculative, and untracked, the system assigns successively decreasing credibility to the joint points of the three states. For example, the tracked joint points can be assigned a reliability of 0.95, the estimated joint points can be assigned a reliability of 0.70, and the untracked joint points can be assigned a reliability of 0.15. Each Kinect sensor can recognize up to six human targets at the same time, so each BodyFrame can contain up to six Body type objects.

It should be noted that, in the above sub-steps, when each Kinect sensor collects the same human skeleton or multiple frames of skeleton data of the same human body, the human skeleton or human body remains still.

In some optional implementation manners of this embodiment, the determination of the weighted average skeleton data of the skeleton data sent by each of the above-mentioned terminal devices according to the above-mentioned skeleton data includes: collecting multi-frame skeleton data of the human skeleton through multiple terminal devices; According to the obtained joint point coordinates of each frame of the above-mentioned skeleton data of each terminal device and the reliability of the above-mentioned joint point coordinates, the reliability of the above-mentioned joint point coordinates is used as the weight of the above-mentioned joint point coordinates to calculate the detection of the above-mentioned terminal equipment. The joint point weighted average value of each joint point of the human skeleton; the joint point data of the above joint points is the joint point weighted average value, and all the joint point data included in the human body skeleton are used as the weight of the human body skeleton detected by the terminal device Average skeleton data. Here, each of the above-mentioned terminal devices accumulates the number of skeletons of N frames from the receipt of the first frame of data. In some specific examples, N is 20 to 30. For the human skeleton of the target person captured by the Kinect of the i-th terminal device The weighted sum S _ij of the coordinates of the jth joint point in the N frames of skeleton data is:

Among them, f refers to the number of frames of the skeleton data, w _fj is the credibility of the jth joint point in the f-th frame of the skeleton data, and v _fj (x, y, z) is the j-th joint point in the f-frame skeleton data coordinate of.

The coordinates of the jth joint point of the human skeleton corresponding to the weighted average skeleton data for:

The weight of the jth joint point of the above human skeleton for:

The weighted average skeleton data corresponding to the multi-frame skeleton data of the target person obtained by the Kinect of the above-mentioned terminal device is:

Wherein, i is the weighted average skeleton data of the i-th terminal device, and the number of joint points of the human skeleton is j, and j can be 25.

Sub-step 2022, calculate the credibility of the skeleton data of each terminal device according to the skeleton data captured by each terminal device, determine the terminal device with the highest reliability from the terminal devices as the reference terminal device, and determine the reference terminal device The coordinate system of is the reference coordinate system.

In this sub-step, the above-mentioned server respectively calculates the credibility of the multi-frame skeleton data collected by each of the above-mentioned terminal devices. Trustworthiness of end devices. The terminal device with the highest reliability is determined as the reference terminal device, and the coordinate system of the reference terminal device is determined as the reference coordinate system. Terminal equipment other than the above-mentioned reference terminal equipment is designated as non-reference terminal equipment, and the coordinate system of the above-mentioned non-reference terminal equipment is a non-reference coordinate system.

In some optional implementation manners in this embodiment, the calculation of the credibility data of the skeleton data of the terminal device includes: obtaining the credibility of each joint point of the multi-frame skeleton data collected by each terminal device; The average value of the credibility of each joint point of the skeleton data collected by the terminal device; the average of the credibility of each joint point of all the joint points that constitute the human skeleton collected by the terminal device value; determine the sum operation result as the terminal device credibility. Or, in some examples, the result obtained by dividing the result of the above summation operation by the number of joint points included in the human body skeleton can also be used as the credibility of the terminal device. Here, for the multi-frame skeleton data of the same human skeleton acquired by each terminal device, the average value of the credibility of each joint point of the human skeleton is first determined, and then the average of the credibility of all joint points of the human skeleton is accumulated. value, and finally, select the terminal device with the largest cumulative result as the reference terminal device. It can be understood that the reliability of each joint point of the reference terminal device (the terminal device with the largest cumulative result) is higher than the reliability of each joint point of other terminal devices, that is, the skeleton obtained by the reference terminal device Most of the joint points in the data are traceable joint points.

Sub-step 2023, according to the difference between the coordinates of the human skeleton corresponding to the above-mentioned weighted average skeleton data in the above-mentioned reference coordinate system and the coordinates in the coordinate system of the non-reference terminal device, calculate the coordinates of the reference coordinate system and each terminal device Euler angles and translation variables between systems.

In this sub-step, based on the reference terminal device and the reference coordinate system determined in sub-step 2022, the above-mentioned server combines the coordinate data of each joint point of the skeleton data obtained by the non-reference terminal device with the above-mentioned human skeleton in the reference coordinate system By comparing the coordinate data of each joint point, the Euler angle and translation variable between the reference coordinate system and the coordinate system of each terminal device are determined. Here, the first coordinate data and the second coordinate data can be determined by determining the coordinates of the central joint point of the human skeleton respectively in the first coordinate data in the reference coordinates and the second coordinate data in the coordinate system of the non-reference terminal device. The vector is used to determine the Euler angle and the translation variable between the reference coordinate system and the coordinate system of each terminal device. Wherein, the central joint point is a joint point which may be pre-designated and located at the center of the human skeleton.

In some optional implementations of this embodiment, the determination of the Euler angles and translation variables between the reference coordinate system and the coordinate systems of each terminal device includes: comparing the above-mentioned human skeleton in the above-mentioned non-reference terminal device coordinate system The difference between the weighted average skeleton data below and the weighted average skeleton data under the above-mentioned reference coordinate system; take the above-mentioned difference as the difference skeleton data of the above-mentioned human skeleton; calculate the coordinates of the above-mentioned difference skeleton data under the above-mentioned reference coordinate system The included angle with each reference axis of the above-mentioned reference coordinate system is Euler angle; the distance between the coordinates of the above-mentioned difference skeleton data in the above-mentioned reference coordinate system and each reference axis of the above-mentioned reference coordinate system is a translation variable. Here, the difference skeleton data is the difference between the weighted average skeleton data of the above-mentioned human skeleton in the reference coordinate system and the data in the non-reference terminal device coordinate system. The angles between the difference skeleton data and each reference axis in the above reference coordinate system are Euler angles, and the distances from each reference axis are translation variables.

In sub-step 2024, a transformation matrix between the reference coordinate system and the coordinate systems of each of the above-mentioned non-reference terminal devices is determined according to the above-mentioned Euler angles and translation variables.

In this embodiment, based on the Euler angles and translation variables determined in sub-step 2023, the above-mentioned server can determine the reference coordinate system and each of the above-mentioned non-reference The transformation matrix between the coordinate systems of the end device. Specifically, the Euler angles between the coordinate axes of the above-mentioned reference coordinate system and the coordinate axes of the above-mentioned non-reference coordinate system constitute a rotation matrix, and the Euler angles between the coordinate axes of the above-mentioned reference coordinate system and the above-mentioned non-reference coordinate system The translation variable constitutes a translation matrix, and the product of the above rotation matrix and the translation matrix is used as a transformation matrix between the above reference coordinate system and the above non-parametric coordinate system.

In some optional implementations of this embodiment, the determination of the transformation matrix between the reference coordinate system and the coordinate system of each non-reference terminal device based on the above Euler angles and translation variables includes: according to the above Euler angles and translation variables The variables determine the rotation matrix and the translation matrix respectively; determine the product of the above rotation transformation matrix and the above translation matrix as the transformation matrix. Here, the above Euler angles and translation variables can be expressed as Euler angle variables θ _x , θ _y , θ _z and translation variables x, y, z, according to the Euler angles θ _x , θ _y , θ _z , the rotation transformation matrix R _i (θ _x , θ _y , θ _z )=R _iz (θ _z )·R _iy (θ _y )·R _ix (θ _x ) can be obtained, which consists of three translation variables x, y, z, the translation matrix T _i (x, y, z) can be obtained, then, the rotation translation transformation matrix is M _i (θ _x , θ _y , θ _z , x, y, z)=T _i ( x,y,z)·R _i (θ _x ,θ _y ,θ _z ). Wherein, i is the number of terminal devices, and R _iz (θ _z ), R _iy (θ _y ), and R _ix (θ _x ) are rotation matrices of Euler angles and coordinate axes of the reference coordinate system, respectively.

In some specific examples, the above-mentioned rotation-translation transformation matrix can be optimized to match the above-mentioned variables θ _x , θ _y , θ _z , x, y, z, so that the function The minimum obtained _Mi is the optimized transformation matrix.

According to the method provided by the above embodiments of the present application, according to the skeleton data acquired by each terminal device, the terminal device with the highest reliability of the skeleton data is determined as the reference terminal device, and the coordinate system of the reference terminal device is the reference coordinate system, and the same human skeleton is determined Euler angles and translation variables of the skeleton data in the reference coordinate system and the non-reference terminal coordinate system, and determine the transformation matrix between the coordinate systems according to the Euler angles and translation variables. Automatic conversion of data coordinates between multiple Kinects has been realized.

Further referring to FIG. 3 , it shows the flow of another embodiment of the training coordinate transformation matrix in the multi-Kinect human skeleton coordinate transformation method. The sub-steps of the above training coordinate transformation matrix include:

Sub-step 3021, receiving multiple frames of skeleton data of the same human body sent by multiple terminal devices, and determining the weighted average skeleton data of the skeleton data of each of the above-mentioned terminal devices according to the above-mentioned skeleton data.

In this embodiment, the electronic device (such as the server shown in Figure 1) on which the multi-Kinect human body skeleton coordinate transformation method runs can receive the same human body information sent by multiple terminal devices from the terminal device through a wired connection or a wireless connection. skeleton data. Wherein, the terminal device can obtain the skeleton data of the human body through the Kinect sensor connected thereto, and send the obtained skeleton data to the server. The above-mentioned skeleton data refers to the image data of the human body detected by the Kinect sensor, and the above-mentioned Kinect sensor determines the above-mentioned each frame of image data according to the detected data when each frame of image data is mapped to the Kinect sensor coordinate system by weighting the data. The weighted average of the number of skeletons of human skeletons captured by the terminal device. The above-mentioned skeleton data may be coordinate data representing the torso of the above-mentioned human skeleton and each joint point in the Kinect sensor coordinate system when each frame of detected image data is mapped to the Kinect sensor coordinate system.

Sub-step 3022, judging whether the above-mentioned human skeleton is front or back, and if the judgment result is negative, exchange the joint point coordinates of the torso of the above-mentioned human skeleton between left and right.

In this embodiment, the above-mentioned server can judge the front and back of the human skeleton through the posture of the human skeleton; it can also judge the front and back of the human skeleton through facial recognition. Flip processing is performed on the skeleton image of the human skeleton judged to be the reverse side, that is, the joint point coordinates on both sides of the torso of the human skeleton are transformed left and right. Here, some joint points can be pre-designated as torso joint points.

Specifically, if it is stipulated that the height of the calibrator’s right hand is higher than that of the left hand during calibration, then calculate the coordinates of the left and right hands of the comparison skeleton. If the height coordinate of the right hand is greater than the height coordinate of the left hand, then the skeleton is the front side, otherwise it is the reverse side, and the left and right coordinates need to be compared. Change.

In some specific examples, the above-mentioned determination of the front and back of the human skeleton through the posture of the human skeleton may be to specify an asymmetric posture used during calibration, and then extract the joint points of the human skeleton from the above-mentioned multi-frame skeleton data. coordinate.

In some examples, it is also possible to determine the front or back of the human skeleton according to the joint point coordinates. Wherein, the coordinates of the above-mentioned joint points include the joint point coordinates of the current frame human skeleton and the joint point coordinates of the synthetic skeleton in the last frame; determine the capture parameters according to the above-mentioned joint point coordinates and the weights of the above-mentioned joint points; The result and the above-mentioned capture parameters determine the front and back of the above-mentioned current frame skeleton. Among them, the above capture parameter C is:

Among them, n represents the number of joint points, vj represents the coordinates of the _{jth joint of the current frame skeleton, v′ j represents} the coordinates of the jth joint of the synthetic skeleton in the previous frame, and w _j represents the weight of the jth joint of the current frame skeleton , w′ _j represents the weight of the jth joint of the synthesized skeleton in the last frame.

Respectively calculate the front capture parameters when the synthetic skeleton of the above frame is the front and the reverse capture parameters when the synthetic skeleton of the above frame is the reverse; compare the size of the above front capture parameters and the above reverse capture parameters; respond to the above positive capture parameters and the above If the difference of the reverse capture parameters is greater than the set threshold, it is determined that the current frame skeleton is the front; in response to the difference between the reverse capture parameters and the front capture parameters is greater than the set threshold, it is determined that the current frame skeleton is the reverse.

Specifically, if the last frame image is the front capture result, calculate C _f , if the last frame image is the reverse capture result, calculate C _b ; if C _f is much smaller than C _b , then the current frame is the front capture result, otherwise it is the result of backside capture. The aforementioned C _f being much smaller than C _b means that the aforementioned C _b is a multiple of C _f , for example, C _b may be 10 times or more than C _f .

Sub-step 3023 calculates the reliability of the skeleton data of the above-mentioned terminal devices, determines the terminal device with the highest reliability among the above-mentioned terminal devices as the reference terminal device, and determines the coordinate system of the above-mentioned reference terminal device as the reference coordinate system.

In this sub-step, the above-mentioned server separately calculates the credibility of the multi-frame skeleton data collected by each of the above-mentioned terminal devices, and determines that the terminal device with the highest average value of credibility is the reference terminal device, and the rest of the terminal devices are non-reference terminals Equipment; determine the coordinate system of the reference terminal equipment as the reference coordinate system. The multi-frame skeleton data collected in the above-mentioned reference terminal device is determined to ensure that there is one and only one human body in the tracking environment or calibration environment, and each terminal device transmits the skeleton data of the human body recognized by different Kinect sensors to the server, and the server from The terminal device acquires the multi-frame skeleton data of the still human body. For example, each Kinect sensor can acquire 20 frames of skeleton data of a still human body.

Sub-step 3024, apply the minimum optimal algorithm according to the average skeleton data to determine the transformation matrix of each non-reference terminal device, the method is: calculate the transformation calibration matrix _Mi corresponding to the minimum F value in the following formula as the reference coordinate system and The coordinate transformation matrix between the coordinate systems of each of the non-reference terminal devices:

Among them, n is the number of joint points of the human skeleton, i is the number of terminal devices, is the coordinates of the j-th joint point of the skeleton data of the i-th terminal device under the reference coordinates, is the coordinate of the jth joint point of the skeleton data of the reference device in the reference coordinate system, yes The weight of the jth joint point of .

In this embodiment, based on the reference terminal device and the reference coordinate system determined in step 303, the above-mentioned server can use the particle swarm optimization algorithm to determine the transformation calibration matrix:

The above-mentioned transformation calibration matrix can be simplified into three rotation variables around the three axes of x, y, and z, and translation variables with the three axes of x, y, and z as directions. The position of particle _i can be expressed as a six-dimensional vector Xi = (θ _x , θ _y , θ _z , x, y, z), and the velocity of particle i can be expressed as a six-dimensional space vector The fitness calculation formula of particles is Set the size of the particle swarm to m, initialize all particles, including random positions and random velocities; through the formula Calculate the fitness value of each particle, where n is the joint point value in the skeleton data, which can be 25; compare the calculated fitness value of each particle with the particle’s historical optimal value pbest _i , if the calculated fitness value If it is less than the historical best value, replace the historical best value; find out the minimum fitness value of all particles in the particle swarm, compare it with the historical optimal value gbest _i of the entire particle swarm, and replace the historical value if it is smaller than the historical optimal value optimal value; according to the formula Update the velocity of each particle, where ω is the inertial parameter of the particle, c ₁ and c ₂ are the "learning" velocity weights of the particle, r ₁ and r ₂ are random numbers between 0 and 1, according to the formula _Xi = X _i +V _i update the position of each particle; when the maximum number of iterations reaches k times, or when the global optimal fitness value is stable around a value for φ consecutive times, the iteration end condition is reached, otherwise jump to the step of updating the particle velocity and continue Find the best location. The coordinate transformation matrix of the coordinate system between each terminal device and the reference terminal device is determined iteratively through the above particle swarm optimization algorithm.

As an example, as shown in Figure 4, it is the skeleton data of the human skeleton of the same character captured by the Kinects of two different terminal devices, the skeleton data shows the outline of the human body and the data of the joint points that form the human skeleton (you can use the respective Kinect coordinate system representation), determine the average skeleton of the skeleton data captured by the above two Kinects, and then determine the coordinate transformation matrix. Figure 5 shows the human skeleton captured by the two Kinects captured by the two Kinects in the reference coordinate system after conversion by the transformation matrix data.

The methods provided in the above embodiments of the present application use the minimum optimal algorithm to determine the optimal conversion matrix between the reference coordinate system and the coordinate system of the non-reference terminal device after multiple zone comparisons. Make each terminal device convert the skeleton data in the reference coordinate system through the conversion matrix continuously and stably.

As another aspect, the application also provides a multi-Kinect human skeleton coordinate transformation device, which includes one or more processors; a storage device for storing one or more programs; when the above-mentioned one or more programs are uploaded by a or a plurality of processors, so that the one or more processors: acquire the skeleton data captured by a plurality of terminal devices; use the trained coordinate transformation matrix to convert the above skeleton data, wherein the training of the transformation matrix includes: receiving multiple The multi-frame skeleton data of the same human body sent by each terminal device, determine the weighted average skeleton data of the skeleton data of each of the above-mentioned terminal devices according to the above-mentioned skeleton data; calculate the credible value of each of the above-mentioned terminal devices according to the skeleton data captured by each terminal device The terminal equipment with the highest reliability is determined from the above-mentioned terminal equipment as the reference terminal equipment, and the coordinate system of the above-mentioned reference terminal equipment is determined as the reference coordinate system. , the coordinate system of the above-mentioned non-reference terminal equipment is a non-reference coordinate system; the reference is calculated according to the difference between the coordinates of the human skeleton corresponding to the above-mentioned weighted average skeleton data in the above-mentioned reference coordinate system and the coordinates in the above-mentioned non-reference coordinate system The Euler angle and translation variable between the coordinate system and the above-mentioned non-reference coordinate system; the above-mentioned Euler angle between the coordinate axis of the above-mentioned reference coordinate system and the coordinate axis of the above-mentioned non-reference coordinate system constitutes a rotation matrix, and the above-mentioned reference coordinate The above-mentioned translation variables between the coordinate axes of the above-mentioned reference coordinate system and the above-mentioned non-reference coordinate system form a translation matrix, and the product of the above-mentioned rotation matrix and the translation matrix is used as the transformation matrix between the above-mentioned reference coordinate system and the above-mentioned non-reference coordinate system.

On the other hand, the present application also provides a computer-readable medium, which may be included in the server described in the above embodiments, or may exist independently without being assembled into the server. The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by the server, the server: acquires the skeleton data captured by multiple terminal devices; uses the trained coordinate transformation matrix to convert The above-mentioned skeleton data, wherein, training the above-mentioned transformation matrix includes: receiving multiple frames of skeleton data of the same human body sent by multiple terminal devices, and determining the weighted average skeleton data of the skeleton data of each of the above-mentioned terminal devices according to the above-mentioned skeleton data; according to each terminal device The captured skeleton data calculates the credibility of each of the above-mentioned terminal devices, determines the terminal device with the highest reliability from the above-mentioned terminal devices as the reference terminal device, and determines the coordinate system of the above-mentioned reference terminal device as the reference coordinate system, except for the above-mentioned Refer to the start-stop designated equipment other than the reference terminal equipment to determine the fee reference terminal equipment, the coordinate system of the above-mentioned non-reference terminal equipment is a non-reference coordinate system; according to the coordinates of the human skeleton corresponding to the above-mentioned weighted average skeleton data in the above-mentioned reference coordinate system and in the above-mentioned The difference between the coordinates in the non-reference coordinate system is used to calculate the Euler angle and translation variable between the reference coordinate system and the above-mentioned non-reference coordinate system; The above-mentioned Euler angles between constitute a rotation matrix, and the above-mentioned translation variables between the coordinate axes of the above-mentioned reference coordinate system and the coordinate axes of the above-mentioned non-reference coordinate system constitute a translation matrix, and the product of the above-mentioned rotation matrix and translation matrix is used as the above-mentioned reference coordinates system and the transformation matrix between the above-mentioned non-parametric coordinate system.

So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings, but those skilled in the art will easily understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to relevant technical features, and the technical solutions after these changes or substitutions will all fall within the protection scope of the present invention.

Claims (10)

1. a kind of more Kinect human skeletons coordinate transformation methods, it is characterised in that methods described includes：

Obtain the skeleton data that multiple terminal devices are captured；

Using the transformation matrix of coordinates trained, the skeleton data is changed, wherein, train the transformation matrix to include：

Receive multiple terminal devices transmitted by same human body multiframe skeleton data, according to the skeleton data determine it is each described in The weighted average skeleton data for the skeleton data that terminal device is sent；

The skeleton data captured according to each terminal device calculates the confidence level of each terminal device, from the terminal device It is reference terminal equipment to determine confidence level highest terminal device, and the coordinate system for determining the reference terminal equipment is with reference to seat Mark system, the other equipment in addition to the reference terminal equipment are appointed as non-reference terminal device, the non-reference terminal device Coordinate system is non-reference coordinate system；

According to coordinate of the human skeleton corresponding to the weighted average skeleton data under the reference frame and described non- The difference between coordinate under reference frame, calculate the Eulerian angles peace between reference frame and the non-reference coordinate system Move variable；

The Eulerian angles between the reference axis of the reference frame and the reference axis of the non-reference coordinate system are formed rotation Torque battle array, the translation variable structure between the reference axis of the reference frame and the reference axis of the non-reference coordinate system Into translation matrix, using the product of the spin matrix and translation matrix as the reference frame and the non-ginseng coordinate system it Between transformation matrix.

2. more Kinect human skeletons coordinate transformation methods according to claim 1, it is characterised in that the skeleton data Including forming each body joint point coordinate of human skeleton and the confidence level of the body joint point coordinate.

3. more Kinect human skeletons coordinate transformation methods according to claim 2, it is characterised in that described in the basis Skeleton data determines the weighted average skeleton data for the skeleton data that each terminal device is sent, including：

Gather the multiframe skeleton data of human skeleton respectively by multiple terminal devices；

According to the body joint point coordinate of skeleton data described in every frame of acquired each terminal device and the body joint point coordinate can Reliability, the weight using the confidence level of the body joint point coordinate as the body joint point coordinate, calculate what the terminal device was detected The artis weighted average of each artis of human skeleton；

Joint point data using the artis weighted average as the artis, with the whole included by the human skeleton The weighted average skeleton data for the human skeleton that joint point data is detected by terminal device.

4. more Kinect human skeletons coordinate transformation methods according to claim 3, it is characterised in that described according to each end The skeleton data that end equipment is captured, the confidence level of each terminal device is calculated, including：

Obtain the confidence level of each artis for the multiframe skeleton data that each terminal device is gathered；

Calculate the average value of the confidence level of each artis for the skeleton data that each terminal device is gathered；

Add and operate the confidence level of each artis of the whole artis for the composition human skeleton that the terminal device is gathered Average value；

It is determined that it is described plus and operating result be the terminal device confidence level.

5. more Kinect human skeletons coordinate transformation methods according to claim 3, it is characterised in that described in the basis Coordinate of the human skeleton corresponding to weighted average skeleton data under the reference frame and under the non-reference coordinate system Coordinate between difference, calculate the Eulerian angles between reference frame and each non-coordinate system and translate variable, including：

Weighted average skeleton data and the non-reference for calculating the human skeleton that the reference terminal equipment is gathered are whole The difference of the weighted average skeleton data for the human skeleton that end equipment is gathered；

Difference skeleton data using the difference as the human skeleton；

Calculate coordinate of the difference skeleton data under the reference frame and each reference axis angle of the reference frame For Eulerian angles；

Calculate coordinate of the difference skeleton data under the reference frame and each reference axis of the reference frame Distance is translation variable.

6. more Kinect human skeletons coordinate transformation methods according to claim 1, it is characterised in that described in the basis Eulerian angles and translation variable determine the transformation matrix between the coordinate system of reference frame and each non-reference terminal device, bag Include：

Three axle xyz of reference frame Eulerian angles θ is determined according to the Eulerian angles_x、θ_y、θ_z, obtain rotation transformation square Battle array is R_i(θ_x, θ_y, θ_z)=R_iz(θ_z)·R_iy(θ_y)·R_ix(θ_x)；

The variable x, y, z of three axles of the reference frame is determined according to the translation variable, it can be deduced that translation matrix T_i (x, y, z)；

Using the product of the rotational transformation matrix and the translation matrix as transformation matrix M_i。

7. more Kinect human skeletons coordinate transformation methods according to claim 6, it is characterised in that methods described is also wrapped Include and minimum optimal algorithm is applied according to average skeleton data, determine the transformation matrix of each non-reference terminal device, its method For：It is calculated as follows corresponding conversion calibration matrix M when F values are minimum in formula_i：

<mrow> <mi>F</mi> <mo>=</mo> <msubsup> <mi>&amp;Sigma;</mi> <mrow> <mi>j</mi> <mo>=</mo> <mn>1</mn> </mrow> <mrow> <mi>j</mi> <mo>=</mo> <mi>n</mi> </mrow> </msubsup> <msub> <mover> <mi>w</mi> <mo>&amp;OverBar;</mo> </mover> <mrow> <mi>i</mi> <mi>j</mi> </mrow> </msub> <msub> <mover> <mi>w</mi> <mo>&amp;OverBar;</mo> </mover> <mrow> <mn>1</mn> <mi>j</mi> </mrow> </msub> <mrow> <mo>|</mo> <mrow> <msub> <mover> <mi>v</mi> <mo>&amp;OverBar;</mo> </mover> <mrow> <mn>1</mn> <mi>j</mi> </mrow> </msub> <mo>-</mo> <msub> <mover> <mi>v</mi> <mo>&amp;OverBar;</mo> </mover> <mrow> <mi>i</mi> <mi>j</mi> </mrow> </msub> <msub> <mi>M</mi> <mi>i</mi> </msub> </mrow> <mo>|</mo> </mrow> </mrow>

Wherein, n is that the joint of human skeleton is counted, and i is the number of terminal device,It is the weighted average of i-th of terminal device Coordinate of j-th of the artis of skeleton data under reference coordinate,It is the jth of the weighted average skeleton data of reference device Coordinate of the individual artis under reference frame,It isJ-th of artis weight.

8. more Kinect human skeletons coordinate transformation methods according to claim 1, it is characterised in that methods described is also wrapped Include：

After weighted average skeleton is calculated, judge the human skeleton for front or back；

If judged result is reverse side, the body joint point coordinate of the human skeleton both sides is subjected to right and left mutually changing.

9. a kind of more Kinect human skeletons coordinate transform processing equipment, including：

One or more processors, it is adapted for carrying out each bar program；And

Storage device, for storing one or more programs,

Characterized in that, described program be suitable to load by processor and is performed with realize in claim 1-8 it is any described in it is more Kinect human skeleton coordinate transformation methods.

10. a kind of computer-readable recording medium, is stored thereon with computer program, it is characterised in that the program is by processor More Kinect human skeletons coordinate transformation methods as described in any in claim 1-8 are realized during execution.