JP3735969B2 - Conducting action judging method and conducting action judging device - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
This invention is vector-quantized sensor output corresponding to the operation, by this hidden Markov model generated label series by (HMM) to compute the probability of output, commanding operation determination method for determining the content of the operation And a command action determination device.
[0002]
[Prior art]
Various devices and methods for determining human movement have been proposed. The method is generally signal processing such as simple filter and size comparison, angle and angle difference of two-dimensional motion signal, and the like.
[0003]
[Problems to be solved by the invention]
However, since human operations are extremely ambiguous and unstable, the accuracy and reliability of detection and discrimination are low only with conventional simple signal processing, and detection errors and discrimination errors frequently occur.
[0004]
For this reason, for example, when trying to control the tempo and dynamics of music using this motion determination,
{Circle around (1)} Since the recognition rate of the commanding action is extremely low, it is necessary for the operator to become familiar with the action that the machine can easily recognize, and it takes time for the user to get used to the system.
{Circle around (2)} A reaction different from the intended operation, that is, misrecognition occurs frequently, and stable music performance is difficult.
Inconvenience such as occurred.
[0005]
An object of the present invention is to provide a command operation determination method and a command operation determination device that improve the accuracy and reliability of operation detection such as a command operation by using an HMM.
[0006]
[Means for Solving the Problems]
The invention of claim 1 of this application is a hidden Markov model in which detection values output from a sensor for detecting motion are labeled at regular intervals, edited into a label sequence, and a label sequence corresponding to a specific motion is learned. Calculates the probability of outputting the label sequence, and determines the content of the operation detected by the sensor when the calculation result exceeds a predetermined value .
[0007]
The invention of claim 2 of this application is characterized in that a label is generated on the basis of the content of the determined operation, and this is inserted into the subsequent label series.
[0008]
The invention of claim 3 of this application is a sensor that detects an operation and outputs a detection value, a state determination unit that labels detection values input from the sensor at regular intervals, and a constant from the state determination unit. A label sequence generation unit that edits labels output every time into a label sequence, an HMM storage unit that stores a hidden Markov model in which a label sequence corresponding to a specific operation is learned, and a label sequence from the label sequence generation unit And a probability calculation unit that calculates the probability that the hidden Markov model stored in the HMM storage unit outputs the label sequence; and the sensor calculates when the calculation result of the probability calculation unit exceeds a predetermined value. An operation determining unit that determines the content of the detected operation.
[0009]
The invention of claim 4 of this application is characterized in that the label sequence generation unit includes means for inserting a label based on the content of the operation determined by the operation determination unit into the subsequent label sequence.
[0010]
DETAILED DESCRIPTION OF THE INVENTION
A hidden Markov model (HMM) used in the present invention will be described with reference to FIG. The HMM outputs various label sequences with predetermined probabilities, has N states (S ₁ , S ₂ ,..., S _N ), changes states one after another at regular intervals, At the time of the transition, labels are output one by one. Next, which state is to be transited and which label is to be output at that time are determined probabilistically by “transition probability” and “output probability”, respectively. Further, the probability that a specific label sequence as viewed from the output side of the HMM is referred to as “likelihood”, and “probability” in the claims of the present application means this “likelihood”.
[0011]
The HMM shown in FIG. 1 is composed of _three states S ₁ , S ₂ , S ₃ and outputs a label sequence consisting of only two types of labels a and b. Transition from the initial state S ₁ to S ₁ itself with a probability of 0.3 (in this case, labels a and b are output with a probability of 0.8 and 0.2, respectively) or with a probability of 0.5 Transition to S ₂ (always outputs a) or transition to final state S ₃ with a probability of 0.2 (always outputs b). From state S ₂ , transition to S ₂ itself with a probability of 0.4 (in which case a or b is output with a probability of 0.3 or 0.7, respectively), or a final probability of 0.6 Transition to the state S ₃ (in which case a or b is output with equal probability).
[0012]
Now consider the probability that this HMM will output the label sequence aab. Since the sequence of state transitions that can be output by the HMM can repeat self-transitions, there are an infinite number without limiting the number of state transitions, but there is a possibility of outputting aab S ₁ S _{There are only 1} S ₂ S ₃ and S ₁ S ₂ S ₂ S ₃ and S ₁ S ₁ S ₁ S ₃ , and the probability of taking each state transition sequence is as follows:
0.3 × 0.8 × 0.5 × 1.0 × 0.6 × 0.5 = 0.036
0.5 × 1.0 × 0.4 × 0.3 × 0.6 × 0.5 = 0.018
0.3 × 0.8 × 0.3 × 0.8 × 0.2 × 1.0 = 0.01152
The total probability that the label sequence aab is output from the HMM, that is, the likelihood of the label sequence aab is
0.036 +0.018 +0.01152 = 0.06552
It is. Note that even if aab is output from the HMM, it is not observed from the output side which of the above three types of state transition series, so this form of Markov model is referred to as a Hidden Markov model. The HMM is conventionally used in the field of speech recognition such as word speech recognition.
[0013]
In speech recognition, input speech is labeled every frame time of about 10 ms, and the output probability of this label sequence is calculated for a plurality of HMMs learned to output pronunciations of different words, and the likelihood is highest. The HMM word is recognized as the word of the input speech. This technology is described in detail in the Journal of the Institute of Electronics, Information and Communication Engineers, April 1987, pages 352 to 358 “Speech Recognition by Markov Model (Masaaki Okouchi)”.
[0014]
Next, a command method for detecting and determining using the HMM in this embodiment, that is, a swinging motion of the command rod will be described. FIG. 2 is a diagram showing the trajectory of the conductor when conducting triple time. FIG. 3 is a diagram showing the trajectory of a conductor when conducting a 4-beat. FIG. 4 is a diagram showing the trajectory of the conductor when conducting a double time signature.
[0015]
2A is a normal (non-legato), FIG. 2B is a legato, FIG. 2C is a light staccato, and FIG. 2D is a trajectory of a baton for a strong staccato. Yes. As is clear from these figures, the instruction operation for the first beat is a swing-down operation from top to bottom, and the lower end of the operation is the beat point of the first beat. Except for a light staccato ((C) in the same figure), a swing-up operation with a short recoil is accompanied. In addition, the instruction operation for the second beat is a swinging operation in the right direction, and when there is a beat point during the operation due to the playing style (non-legato (Fig. (A)), legato (Fig. (B)) ) And when there is a beat point at the right end of the rightward movement (staccato ((C), (D) in the figure)). In addition, the instruction operation for the third beat is a swing-up operation from the lower right to the upper left, and the light staccato ((C) in the figure) has a beat point at its end point (starting point of the first beat swinging down). Except for, there is a beat point in the middle of the operation. In the figure, the beat point is indicated by the number of the beat number, but the one surrounded by □ is the beat point where the baton stops at that point, and the one surrounded by ○ is the speed at which the baton is located. It is a beat point that passes or turns around the point.
[0016]
In FIG. 3, (A) is a normal (non-legato), (B) is a legato, (C) is a light staccato, and (D) is a trajectory of a baton for a strong staccato. Yes. The 4-beat command method is similar to the 3-beat command method, and the 4-beat first beat instruction operation, the 3-beat first beat instruction operation, the 4-beat third beat instruction operation, and the 3-beat second beat. The instruction operation, the fourth beat instruction operation of 4 beats, and the third beat instruction operation of 3 beats substantially correspond. Then, the second beat instructing action of 4 beats is a swinging action in the left direction from the end point of the first beat instructing action, and when there is a beat point during the action due to the rendition style (non-legato (FIG. (A)) ), Legato (FIG. (B))) and a beat point at the left end of the leftward movement (staccato (FIG. (C), (D))).
[0017]
Further, in FIG. 4, a two-beat instruction is almost vertically moved. In non-legato (Fig. (A)), legato (Fig. (B)), and strong staccato (Fig. (D)), the first beat indicating operation is a swing-down operation from top to bottom and its reaction The lower end of the swing-down motion is the beat point of the first beat. In addition, the second beat instruction operation includes a swing-down operation, which is a short preliminary operation, and a swing-up operation from the bottom to the top (starting point of the first beat instruction operation). The lower end of the swing-down operation is the second beat. This is the beat point.
[0018]
FIG. 5 shows a command action analysis apparatus that analyzes the command method from the swing of the command stick using the HMM. FIG. 6 is a diagram for explaining the state determination operation of the state determination unit of the command action analyzer, and FIG. 7 is a diagram illustrating the HMM stored in the HMM storage unit 5 of the command action analyzer.
[0019]
This commanding action analysis apparatus comprises a sensor 1, a state determination unit 2, a register 3, a probability calculation unit 4, an HMM storage unit 5, and a beat determination unit 6. The content determined by the beat determination unit 6 is an automatic performance device 7 Is input. The sensor 1 is a sensor that detects an angular velocity or an acceleration built in a controller that is held by an operator according to the above-described command method. As a controller that swings in accordance with the command method, a command bar type is generally used, but other types such as a hand grip type or a direct stick type can be considered. Since this sensor 1 is used to detect a region of a swing angle as shown in FIG. 6A, for example, a vertical sensor for detecting a vertical component of swing and a horizontal component of swing are used. It consists of a horizontal sensor to detect. The baton having such vertical and horizontal sensors has not been disclosed, but is described in detail in Japanese Patent Application No. 7-110549. The detection value of the sensor 1 is input to the state determination unit 2.
[0020]
The state determination unit 2 performs the following operation. (1) The output of the sensor 1 is divided into 10 ms frames. {Circle around (2)} The region of the swing speed (angular speed) is judged, and each frame is labeled (operation labels: l _{1 to} l ₅ ) according to the section shown in FIG. (3) The label is input to the register 3. This operation is repeatedly executed every 10 ms frame clock. Note that l _{6 to} l _{14 in} the label list of FIG. 6B are beat labels output by the beat determination unit 6.
[0021]
Note that the labeling of the swing motion in FIG. 6A is an example, and the present invention is not limited to this. The sensor output corresponding to the input operation differs depending on the sensing system (type of controller or sensor) used, the operator, the gripping method of the sensor unit, and so on. It is only necessary to collect a large amount of beat instruction motion data in various modes such as the operator and a plurality of sensor gripping methods, and to obtain a representative point of similar motion data for labeling.
[0022]
The register 3 has a configuration as shown in FIG. A beat label register 30 for storing beat determination information (beat label) determined by the beat determination unit 6 described later, a 50-stage shift register 31 for storing 50 labels (operation labels) input from the state determination unit 2; And it consists of the composition part 32 which connects the beat label and the action label and inputs them to the probability calculation part 4. The shift register 31 shifts the stored contents with a frame clock of 10 ms, and stores 50 (500 ms worth) of the latest operation labels input from the state determination unit 2.
[0023]
The register stores the beat label and the action label separately, and connects the beat label and the action label so that the beat label is always at the head of the label series, for the following reason.
When the length of the shift register 31 is 1 beat or more, not only the beat operation of this time but also the label of the beat operation of the previous beat is included, and the analysis becomes complicated. Therefore, although the length of the shift register 31 is limited to 500 ms, in that case, if the beat label is input to the shift register 31 in time series like the operation label, the shift register 31 leaves the shift register 31 at the next beat timing. I'm sorry. Therefore, only beat labels are stored separately.
[0024]
However, as shown in (C) in the figure, the shift register is configured with a length of one or more beats, and beat labels are input in time series in the same way as operation labels. The labels can be included in the label series in time series. In this case, by analyzing including the operation of the beat operation of the previous beat, the analysis becomes complicated, but the beat type can be determined more accurately.
[0025]
However, the number of stages of the shift register and the frequency of the frame clock are not limited to the above embodiment.
[0026]
The probability calculation unit 4 stores in the HMM storage unit 5 the probability of outputting a label sequence composed of 51 labels consisting of one beat label and 50 action labels, that is, the likelihood of the label sequence. Calculate for all existing HMMs. As shown in FIG. 7A, the HMM storage unit 5 stores a plurality of HMMs that output label sequences of various beating operations. These HMMs are modeled with a finite number of state transition probabilities in which the commanded movement that is the object of recognition changes over time. The HMM consists of 3 to 4 states with self-transition paths. The state transition probability and the output probability of each label may be determined. The likelihood calculated by the probability calculation unit 4 is output to the beat determination unit 6.
[0027]
Here, FIG. 7 (B), (C) is, HMM learned the quadruple time of the first beat of the label sequence, i.e., a diagram illustrating an example of M _41. FIG. 5B shows the configuration of the HMM when a label series having a beat label at the head is always input from the register having the configuration shown in FIG. 5B. FIG. 5C shows the configuration of FIG. The configuration of the HMM is shown in the case where a label series consisting of an operation label of the previous beat, a beat label of the previous beat, and an operation label of the current beat is input from the register having the configuration of FIG.
In FIG. 7B, since there is always only one beat label at the head, the HMM always transitions from state S ₁ to state S ₂ (with transition probability = 1), at which time any beat label l ₆ ~ L ₁₄ is output. Only the state S ₂ and state S operation label l ₁ to l ₅ In ₃ outputs.
On the other hand, in FIG. 5C, the number of states is 4 in order to analyze the previous beat beat action label, the previous beat beat label, and the current beat action label, and at all transitions (including self-transition). There is a possibility that all the labels l _{1 to} l ₁₄ are output.
The configuration of the HMM is not limited to the above example.
[0028]
The beat determination unit 6 compares the likelihood of each HMM input from the probability calculation unit 4 and selects the highest one, and when the highest one exceeds a certain threshold, it is a beat timing, The beat type (time signature and beat number) at that time is determined to be the beat type of the HMM that outputs the highest likelihood. If even the highest value of the input likelihood does not exceed the certain threshold, nothing is output as it is not a beat timing.
[0029]
That is, the register 3 outputs 51 label sequences to the probability calculation unit 4 at each frame timing regardless of whether or not it is a beat timing, and the probability calculation unit 4 outputs each HMM based on the input label sequence. Is output at each frame timing. Thus, the likelihood of all HMMs is input to the beat determination unit 6 regardless of whether or not it is a beat timing. However, the likelihood for the label sequence input at the beat timing and other timings are also input. The absolute value differs from the likelihood for the input label series. Therefore, by setting an appropriate threshold value and determining that the likelihood is not the beat timing when the likelihood is less than that, the beat determination unit 6 determines the beat type based on the HMM that outputs the maximum likelihood. Synchronously, beat timing can be detected.
[0030]
When the beat determination unit 6 determines the beat timing and the beat type of the beat, the beat type is output to the automatic performance device 7. The automatic performance device 7 controls the performance tempo based on the input beat timing and the beat type information so that the performance beat timing, the beat type matches the input beat timing, and the beat type. Furthermore, the beat determination unit 6 generates beat labels (l _{6 to} l ₁₄ ) corresponding to the beat type and inputs them to the register 3. The register 3 stores the input beat label in the beat label register 30.
[0031]
With the above configuration, the conducting action detection device can control the automatic performance device 7 by detecting the beat instruction action by the operator. In this device, the beat determination result of the beat determination unit 6 is converted into a beat label, input to the register 3, and stored in a register different from the operation label. You may make it memorize | store in order according to the generation | occurrence | production timing.
[0032]
Note that more advanced learning can be applied to the HMM stored in the HMM storage unit 5 and reflected in the recognition. Note that learning here means that “the likelihood M Pr (L: M) of the model M of the HMM that expresses the action for the label series L given to a certain action is maximized. Adjusting parameters (transition probabilities and output probabilities). As this learning,
(1) Customization for a specific individual: A method for recalculating representative points with only the individual data (2) Generalization: A method for collecting more people's data and recalculating the representative points (3) Performing Fine tuning: When the performer's data is always shifted from the representative value preset in the corresponding label, the representative value is finely adjusted periodically.
[0033]
and so on. Learning is performed by a method in which an appropriate initial value is given to a parameter and convergence is made by iterative calculation based on data.
[0034]
It should be noted that various methods can be considered for modeling the command method using the HMM depending on how the label, the type of parameter to be handled, the structure of the HMM, and other various elements are determined. It is just an example of a modeling technique.
[0035]
It is also possible to increase the number of parameters and status labels to be handled to increase the types of actions to be recognized, the types of music information, and the recognition rate. For example, the dynamics can be recognized from the magnitude and speed of the motion, or legato, staccato, etc. can be recognized by referring to the motion trajectory curvature in the two-dimensional plane. In other words, if the curvature of the trajectory at the time of beat is small (smooth), it can be determined that it is legato (slur / espresso), and if the curvature is large (clear), it is staccato. It can be judged.
[0036]
As the parameters used for labeling, the direction of swing and velocity (angular velocity) were used, but the main motion direction component was obtained from the shape of the command (beat instruction) trajectory, and the axis of the first component was The direction and the axis of the second component may be converted to the horizontal direction to reduce the difference in the player's gripping method and the habitual component.
[0037]
In addition to the swing direction and velocity (angular velocity), the following parameters can be used.
[0038]
Peak, valley, absolute value, etc. of time domain waveforms such as angle, position, velocity, acceleration from reference point in 2D plane or 3D space,
Difference from previous beat type (beat timing) (angle, speed, position, etc.)
Any of these quadrants as seen from the center of the difference operation from the previous beat timing from the previous beat timing may be used, or a plurality of them may be used in combination. Alternatively, a spatial analysis of these multiple parameters may be subjected to cluster analysis to obtain a representative vector as a label.
[0039]
In addition, the command action analysis device is a device for recognizing beat timing and beat type. However, HMM can be applied to recognition of a layer one level higher, that is, to analyze time signatures by chaining beat types. it can. For example, recognition is performed such as “when the transition from the second beat → the third beat → the first beat is performed, the three beats are now being played”. In this case, the time signature can be recognized without hitting all the beats by introducing a null transition that causes a state transition without outputting a label. For example, in the HMM for 3-beat recognition, if the null transition from the 1st beat to the 3rd beat is allowed, the 3-beat can be recognized without necessarily hitting the 2nd beat.
[0040]
As described above, in this embodiment, the present invention is applied to the commanding action determination device that controls the tempo of automatic performance, but the commanding action is a continuous action that is repeated based on a certain rule in time series. The structure determination and learning of the HMM are easy, and high determination accuracy can be expected.
[0041]
Further, the apparatus of FIG. 5 can be used not only for determining a commanding action, but also for determining a general action of a person or an object. For example, the present invention can be applied to an interface of operation in virtual reality (VR) as an application to a multimedia interface. Sensors used in VR include a three-dimensional position / angle sensor that detects the position and angle in three dimensions, a glove-type indirect bending angle detection sensor (data grab) that detects an indirect finger bending angle, and a suit type. There are indirect bending angle detection sensors. It is also possible to recognize an operation image taken by a camera. FIG. 8 labels the motion state detected by the sensor (FIG. 8A), and determines the motion in the game by the HMM (FIG. 5B) that learned the time transition of the label. For example, the HMM corresponding to the punch outputs a high probability for the label sequence in the operating state of grasping the hand (l ₈ ), stretching the elbow (l ₂ ), and fully extending the elbow (l ₄ ). To be learned.
[0042]
It is also possible to use the apparatus shown in FIG. 5 for sign language recognition. Also in this case, the shape of the finger and the position of the hand are detected by a camera, a data grab, etc., and are labeled as shown in FIG. Based on this label sequence, a word expressed by an HMM as shown in FIG. Further, the movement is not limited to the shape of the finger or the position of the hand, and the movement may be detected based on the movement of the whole body.
[0043]
In addition to HMM, fuzzy control and neural network are known as motion recognition methods. However, fuzzy control needs to describe all detection / discrimination rules, whereas HMM does not require rule description. Therefore, it is easy to build a system because rules can be built in by learning. In addition, in the neural network, a very complicated operation is required for the network to learn, but in the HMM, the learning can be performed by a simple operation and the learning is easy. Therefore, it is more effective for motion recognition than these fuzzy controls and neural networks.
[0044]
Further, the HMM can reflect the fluctuation of the actual operation more accurately than the fuzzy control or the neural network. This is because the output probability corresponds to a probabilistic fluctuation in which the state occurs, and the transition probability corresponds to a fluctuation in the time axis direction. Moreover, since the structure is simple, it is easy to carry out theoretical development by statistical theory, information theory, etc., and it is easy to integrate into the recognition of the upper hierarchy level using the concept of probability.
[0045]
【The invention's effect】
As described above, according to the present invention, the HMM learns a specific operation, labels the sensor output based on the operation, and calculates the likelihood of the HMM. By absorbing these fluctuations, it is possible to accurately determine the operation.
[0046]
In addition, by generating a label based on the operation determination result and including it in the label sequence given to the HMM, the past operation content affects the determination of the next operation content, and the continuous operation is determined. In this case, it is possible to determine the operation with high accuracy.
[Brief description of the drawings]
[Fig. 1] A diagram showing a simple example of an HMM. [Fig. 2] A diagram showing a trajectory of a command bar when conducting a three time signature. [Fig. 3] A diagram showing a trajectory of a command rod when conducting a four time signature. [Fig. 4 is a diagram showing the trajectory of a command stick when commanding two beats. FIG. 5 is a diagram showing a configuration of a command action analysis apparatus according to an embodiment of the present invention. FIG. 6 is a diagram of a state determination unit of the command action analysis apparatus. FIG. 7 is a diagram illustrating the state determination operation. FIG. 7 is a diagram illustrating an HMM stored in the HMM storage unit of the command motion analysis device. FIG. 8 is a label and an HMM when the motion analysis device of the present invention is applied to a game. FIG. 9 is a diagram showing labels and HMMs when the motion analysis apparatus of the present invention is applied to sign language recognition.
DESCRIPTION OF SYMBOLS 1 ... Sensor, 2 ... State determination part, 3 ... Register, 4 ... Probability calculation part,
5 ... HMM storage unit, 6 ... beat determination unit, 7 ... automatic performance device

Claims (4)

The detection value output from the motion detection sensor is labeled at regular intervals, edited into a label sequence, and the probability that a hidden Markov model that has learned the label sequence corresponding to a specific motion will output the label sequence is calculated. And a commanding action determination method characterized by determining the content of the action detected by the sensor when the calculation result exceeds a predetermined value . To generate a label based on the contents of the determined operation command operation determination method according to claim 1, characterized in that inserted into a subsequent said label sequence. A sensor that detects movement and outputs a detection value;
A state determination unit that labels detection values input from the sensor at regular intervals;
A label sequence generation unit for editing labels output at regular intervals from the state determination unit into label sequences;
An HMM storage unit that stores a hidden Markov model in which a label sequence corresponding to a specific action is learned;
A probability calculation unit that inputs a label sequence from the label sequence generation unit and calculates a probability that a hidden Markov model stored in the HMM storage unit outputs the label sequence;
An operation determination unit that determines the content of the operation detected by the sensor when the calculation result of the probability calculation unit exceeds a predetermined value ;
A commanding action determination device consisting of The command operation determining apparatus according to claim 3, wherein the label sequence generation unit includes means for inserting a label based on the content of the operation determined by the operation determination unit into the subsequent label sequence.

Images