JP3519672B2 - Motion recognition processing device and motion analysis system for moving object - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a technique for recognizing a motion of a moving body based on a captured moving image, and more particularly to a motion recognizing apparatus suitable for use in monitoring or analyzing the motion of the moving body.

[0002]

2. Description of the Related Art As a conventional technique for recognizing a motion of a person in an image, a method using an optical flow is known. This is, as shown in JP-A-10-334270,
It is a method that recognizes the movement by analyzing the change of the image due to the movement of the object in the image as an optical flow.

Another conventional technique is MIT Medi.
a Laboratory Perceptual Computing Section Technica
l As shown in Report No.402, the silhouette of the object is extracted and given the density value according to the time change,
Motion History Ima by integrating it on the time axis
There is a method of generating an image called ge (MHI) and classifying the motion of an object from the moment feature amount of this MHI.

[0004]

DISCLOSURE OF THE INVENTION Problems to be Solved by the Invention
In No. 70, because it is a method focusing on the density distribution of the image,
The processing was difficult when the feature did not exist as a clear pattern of shades. Especially when targeting a person moving in a wide area, it is difficult to capture a person image with sufficient resolution, so it is difficult to obtain a clear grayscale pattern and it is also difficult to recognize an action.

In addition, the method of classifying motions based on the characteristics of the temporal shape change of a person using a silhouette image cannot identify the degree of the motion. In addition, there is no consideration in the case of targeting a moving body moving in a wide area.

In view of the state of the art, the object of the present invention is to
It is to recognize the motion of a moving body moving in a wide area. Further, it is also necessary to recognize the degree of movement of the moving body.

[0007]

In order to achieve the above object, a parameter input section inputs a control parameter for changing a range and a density value of a specific region in an image at a predetermined ratio, and then diffuses the control parameter. The calculation unit separates a moving image of a moving object into an object image and a background image, assigns a predetermined density value to pixels in an area where the object image exists at a predetermined time, and based on the control parameter, the passage of time. Together with changing the range of the area and the density value, the motion identification unit extracts a feature amount from the distribution of the density value in the region, based on the collation of the feature amount and the default value, the movement of the object recognize.

Further, a trajectory extraction processing section for obtaining a trajectory of a moving object from moving image data from a video camera which has photographed a moving action of a person, and outputting it as trajectory data, and the moving image data individually Identifying the action of a moving object, and identifying the action as a group from the action identification unit that outputs as independent action data and the trajectory data,
A behavior analysis processing unit that outputs as collective action data, and a display processing unit that performs statistical processing and three-dimensional graphics processing from the single action data and the collective action data to output statistical data and CG video data. , Analyze the motion of moving objects.

[0009]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below. FIG. 1 shows a motion recognition method according to the present invention.
It is explanatory drawing of a cellular diffusion model. In the present invention, it is assumed that a diffusion phenomenon occurs in the surrounding fluid due to the motion of a person,
Since the situation is analyzed with a model discretized into cells,
This motion recognition method is called a cellular diffusion model. Further, although there are various kinds of motions to be recognized, here, the target is to identify a moving motion such as “walking” or “running”.

FIG. 1A shows an image of a moving person photographed by a video camera or the like. Although only 4 × 4 pixels are shown in the figure, this is a part of the vicinity of a person in the image. Here, it is assumed that the person is photographed with a size of one pixel, and the person image is at the position as shown at the time t0. Since the present invention does not use the information of the grayscale pattern of the person image, the person image may be a binary image in which the silhouette of the person is known.

(B) shows a human image at time t1. Compared to (a), the human image has moved to the position one pixel to the right. (C) shows the pressure state of the cellular diffusion model at t1. In this model, the portion of the background image other than the human image is considered to be filled with a certain fluid. Now the person is from cell A to cell B
However, in this model, it is assumed that the person disappears in cell A and the person appears in cell B.
At this time, in the cell A, the existing person disappears to generate a negative pressure, and the fluid tries to flow in due to the diffusion phenomenon. Similarly, in cell B, the appearance of a person causes positive pressure, and the fluid tends to flow out due to diffusion. In addition, the pressure is 0 in the part where there is no change.

At this time, a pressure difference of the fluid is generated in the cell as shown in (d). Here, the disappearing portion of the person like cell A is the low pressure portion, and the appearing portion of the person like cell B is the high pressure portion. Will be called. In this figure, the hatching portion on the lower right side indicates the low pressure portion, and the hatching portion on the upper right side indicates the high pressure portion. The thicker the hatching line, the greater the pressure difference with the surroundings. Further, the pressure is 0 in the part without hatching.

(E) shows the diffusion state of the low voltage portion at time t2. In the low pressure part, the fluid flows in due to diffusion, and the pressure difference with the surroundings becomes small (pressure rises).
However, the area of the low pressure part will be expanded.

The pressure of the hatched portion is a value obtained by adding a predetermined value to the pressure value of (d), and the value is stored in the pixel as the density value of the image. The expansion of the area is obtained by applying the expansion operation of image processing. The expansion operation is a process of thickening a person image by one pixel. It is well known that one pixel of the input image P is Pij, one pixel of the output image Q is Qij, the person image is 1 and the background image is 0. Is defined as in Equation 1.

[0015]

[Equation 1] When either Pij or its 8 neighborhoods is 1: Qij
= 1 Other cases: Qij = 0 This expansion operation can be executed at high speed because it can be executed by the hardware of the image processing board connected to the personal computer.

Similarly, (f) shows the diffusion state of the high-voltage portion.
As the fluid flows out by diffusion, the pressure drops and the high pressure region expands. Also in this case, the pressure value of the hatched portion is a value obtained by subtracting a predetermined value from the pressure value of (d), and the pressure value is stored in the pixel. Similarly, the expanded area is also obtained by expansion calculation.

(G) is the result of adding the low pressure part of (e) and the high pressure part of (f), and this is the final pressure distribution at time t2. Since the absolute values of the pressure values are the same in the low pressure portion and the high pressure portion here, the pressure value is 0 in the area where both are present due to the addition. The addition of these pressure values is also performed by the image processing board as the addition of the density value storing the pressure value.

Next, a motion recognition method based on the cellular diffusion model will be described with reference to FIG. Here, it is assumed that the person image at time t0 is (a), t1 is (b), t2 is (c), and t3 is (d), and the person is moving to the right pixel by pixel. .

First, as shown in FIG. 1, the pressure distribution accompanying the state change from (a) to (b) becomes like (e) at time t1, becomes (f) at t2, and Spread and at t3
(g). Similarly, in the change from (b) to (c), the pressure distribution becomes like (h) at time t2, and becomes (i) at t3. Similarly, the change from (c) to (d) is as shown in (j) at time t3.

Here, in order to analyze the operation in the state from time t0 to t3, the pressure distributions from (e) to (j) are added at each time. As a result, (k) is (e) as it is, (l) is the sum of (f) and (h), and (m) is (g) and (i).
(j) is added. In the motion recognition of the present invention, (k)
From (m) to (m), the characteristics of the pressure distribution pattern at each time are used.

FIG. 3 shows an example of pressure distribution when a person moves. Here, the person image is photographed in a state as shown in (a) in a part of the image, and this person is moving in the right-hand direction of the screen. This person image is the silhouette of the person in the image.

The pressure distribution is calculated by the method described with reference to FIG. 2, and the result has a distribution as shown in (b), for example. Then, the part where the positive or negative pressure exists is extracted and the circumscribed rectangle is obtained. Since it is possible that a plurality of persons exist, in that case, each circumscribed rectangle is labeled for distinction. Since the pressure is stored as a density value in the image memory, the circumscribed rectangle and the result of labeling are obtained using the labeling calculation of the image processing board.

(C) is X for the pressure distribution in the circumscribed rectangle.
The pressure projection value in the axial direction is obtained. The pressure projection value is the pressure value of each cell projected in a specific axis direction and added, and the value can be obtained by specifying the label in the image and performing projection calculation on the image processing board. it can.

When a person moves, a convex high pressure projection value is generated on the moving direction side (the right side in this case) in the circumscribed rectangle, and a concave low pressure projection value is generated on the opposite movement direction side. That is the normal pattern. The movement of the person is recognized by matching with the pattern of the pressure projection values.

FIG. 4 describes a method in which Fourier analysis is applied to pattern matching in order to process movement recognition. (a) is a diagram showing a distribution of pressure projection values, and this distribution is considered as a one-dimensional signal, and the amplitude and phase spectra are obtained by performing a discrete Fourier transform on this signal.

If N data of i = 0,1,2, ..., N-1 exists for the discretized one-dimensional signal f (i), its Fourier coefficient Ck is well known. As described above, it is represented by Equation 2 (j is an imaginary unit).

[0027]

[Equation 2]

From this Ck, the amplitude spectrum and the phase spectrum can be obtained. The measured value of (b) is obtained by Fourier-transforming the fluctuation of the pressure projection value of (a) to obtain the spectrum. At this time, the size of the pressure distribution differs depending on the position where the image is captured on the screen, but here, the size of the pressure distribution is normalized so that the size in the X-axis direction is all N pixels.

(C) is the spectrum of a moving person,
This is the reference value. The distribution of this spectrum represents the feature amount for the movement of the person. If it can be determined that the spectrum of the measured value in (b) is similar to the reference value in (c), the person is recognized as moving.

When a signal is judged in a frequency region such as a spectrum, it is possible to cut and compare the higher harmonics which are likely to cause noise during shooting, which is advantageous in that it is less susceptible to noise. . In the above example, the spectrum obtained by the Fourier transform is used for matching the measured value and the reference value. However, in addition to this, for example, for a function indicating the distribution of pressure projection values, it is possible to calculate the residual sum of squares of the function of the measured value and the reference value, and to judge that the smaller this value is, the more similar the two are. Is possible.

FIG. 5 is an explanatory diagram of the motion classification process. Figure 4
Although it is possible to recognize that a person is moving by this method, it is impossible to distinguish how fast the person is moving. Therefore, here, the information of the circumscribing rectangle is used to classify the motion into “standing”, “walking”, or “running”.

Now, assuming that the person moves in the X-axis direction, the circumscribed rectangle becomes longer in the X-axis direction as the moving speed increases. Since the pressure distribution has a shape corresponding to the amount of movement within a fixed time, the size of the circumscribed rectangle has movement information. The aspect ratio of the circumscribed rectangle represents the moving speed.

As shown in the figure, the length of the circumscribed rectangle in the X-axis direction is B
The x and Y axis lengths are By and the aspect ratio is Bx / By. If this aspect ratio is as large as (a), the person is "running", and if it is small as (b), it is "standing", and the middle of the two is You can judge that you are "walking". Even if there is a movement in the Y-axis direction, it is possible to make a similar determination.

Next, an example of the operation recognition processing procedure of the present invention will be described with reference to the flowchart of FIG. Step 100
Then, the user inputs the control parameters for diffusion calculation using the keyboard and mouse. Specifically, it is a parameter that controls the size of the density value assigned to the pixel, the ratio of the density value that decreases due to diffusion, and the ratio that the area expands due to diffusion. Since these need to be changed according to the shooting conditions at the time of recognition, the user makes adjustments here.

Step 101 means repeating the process of step 102 for a predetermined number of frames. Step 102 is a step of forming a background image, and generally, a portion having no change in the image is detected within a predetermined number of frames, and the portion may be used as the background image.

Step 103 means repeating the processing of steps 104 to 116 for the number of frames to be measured. In step 104, an image for one frame is read from the video camera into the image memory of the image processing board. In step 105, a person image is extracted. For this, the difference between the background image created in 102 and the current image read in 104 is calculated, and this is further binarized to obtain the silhouette of the person as a person image. If both the background image and the current image are stored in the image memory, these operations can be executed on the image processing board.

Step 106 means repeating the processing of steps 107 to 111 for a predetermined number of times of diffusion. Here, the number of times of diffusion is the number of times of diffusion required to eliminate the generated pressure difference, and is 3, for example, in the example of FIG.
In this case, assuming that the current time is t3, it is possible to go back three frames and calculate sequentially from the image (a) at t0 to (m).
Calculate the pressure distribution like.

At step 107, the high voltage portion is extracted. Figure 1
In the example, the high voltage part is the part of the background image at time t0, and t1
Then, it is the part that became a portrait. In order to find the high-voltage portion, the image processing board may perform the logical operation of the images (a) and (b). Similarly, in step 108, the low voltage part is extracted. The low-voltage portion is the portion of the human image at t0, and the portion of the background image at t1, which is also calculated by the logical operation.

In step 109, the pressure diffusion state is obtained. As described with reference to FIG. 1, how the pressure is diffused is calculated by the image expansion calculation. The size of spreading by diffusion is set in advance, and for example, it is specified that one pixel expands by one diffusion.

In step 110, the pressure value changed by diffusion is corrected. As also described with reference to FIG. 1, the pressure in the low pressure portion rises and the pressure in the high pressure portion decreases due to diffusion. Here, too, the value set in advance is calculated so that a predetermined pressure value is added or subtracted in one diffusion.

In step 111, the pressure distribution is calculated. In the example of FIG. 2, the image of (m) is obtained, and (e) to (j)
Calculate by adding the pressure images such as

In step 112, the pressure projection value is calculated.
The pressure projection value is as shown in FIG. 3C, and is calculated by the projection calculation of the image processing board from the pressure distribution. In step 113, discrete Fourier transform is performed. In this method, the spectral distribution of the pressure projection values is obtained by the method described with reference to FIG. Step 114 is the step of matching the spectra,
As shown in FIG. 4, the measured value and the reference value are compared, and if the degree of similarity is high, it is determined that the person is moving.

Step 115 is a step of calculating the aspect ratio of the circumscribing rectangle. As shown in FIG. 5, the motion is classified as "running", "walking" or "standing" from the ratio. You can Step 116 outputs the recognition result. Since the movement operation of the person in the image can be recognized by the processing so far, the result is output to a file on the disk or the like.

FIG. 7 is a block diagram of a motion recognition processing apparatus according to an embodiment. The video camera 7 outputs a video signal to the image processing board 6. The image processing board 6 has an image memory and an image operation function such as logical operation and addition / subtraction. By controlling this with the personal computer 5, the processing is executed in the procedure shown in FIG.

Inside the personal computer 5, the motion recognition processing unit 1
Is implemented by software or hardware. Here, the parameter input unit 2 of the motion recognition processing unit 1 executes step 100 in the flow of FIG. Diffusion calculator 3
Executes the processing from step 101 to step 111, processes the expansion of the area to which the density value is assigned by the expansion operation of the image with the passage of time, and changes the absolute value of the density value so as to decrease. . Then, an image created by changing the range of the region and the density value with time is displayed on the display.

The operation identifying section 4 performs the steps 112 to 1
Perform up to 16 processes. Using the projected density value obtained by projecting the density value distribution in a predetermined axis direction as a feature amount, Fourier transform is performed from the density value distribution to calculate a spectrum, and the spectrum is collated with a predetermined value. Further, a circumscribing rectangle circumscribing the area in which the density values are distributed is calculated, and the operation is recognized based on the length ratio of the vertical and horizontal sides of the circumscribing rectangle. Then, with respect to the distribution of the density values, the fluctuation of the density in a predetermined section is investigated, and when a large fluctuation equal to or larger than the predetermined value is detected, it is determined that there is a part where the object moves in the section.

In this embodiment, the image processing part is executed by the hardware called the image processing board 6,
It is also possible to execute everything by the software in the personal computer without using this.

FIG. 8 is a block diagram showing a case where a system for human motion analysis is constructed by using the motion recognition processing according to the present invention. The moving image data from the video camera 7 is input to the motion recognition process 1 and the trajectory extraction process 20.

In the trajectory extracting process 20, the object extracting unit 21 extracts a moving object such as a person or an object in the image, and the labeling unit 22
Identifies and tracks each moving body, and the 3D coordinate conversion unit 23 converts the image coordinates into three-dimensional space coordinates and outputs the data as trajectory data. The content of the locus data is, for example, the position coordinates on the plane where the person was present recorded at each time.

Since the motion recognition result output by the personal computer 1 represents the motion of the moving body alone, it is referred to as independent motion data here. Here, for example, for a person, an identifier indicating the type of motion such as "running" or "standing" is recorded at each time.

The behavior analysis process 30 is a part for analyzing the movement of a group of persons, for example, and comprises a feature quantity estimating section 31 for obtaining a motion feature quantity and a matching section 32 for collating the feature quantity. The behavior analysis processing 30 receives the trajectory data as input and outputs group action data describing the meaning of the movement of the group. Regarding this data, as an example of security application of person monitoring in a situation where many people gather such as commercial facilities and public facilities, an identifier that represents the meaning of collective action such as "avoidance" and "aggregation" is used. , Is recorded at each time.

In the display processing 40, the individual motion data and the collective motion data are input, and the statistical analysis unit 41 first outputs the statistical data. The content of the statistical data is information such as the moving distance of each person and the number of classified operations, and is displayed on a display together with the CG image. Next model motion generator
In 42, the joint angle of each joint is determined in order to move the three-dimensional CG human model. Then, the rendering unit 43 creates a CG image from the human body model and outputs it as CG image data.

The present invention can be applied to applications other than security and game analysis in sports such as soccer. In this case, the individual motion data represents the recognition result of the motion of each player, and the collective motion data represents the recognition result of the team play.

According to the present embodiment, not only the position of the person is displayed but also the identifier of the action independently performed for each person can be provided. Therefore, effective information for the action analysis of the person can be obtained. There is an effect that can be provided.

In the above embodiment, in the movement recognition process, the spectrum is obtained from the pressure projection value as shown in FIG. 4 and is used as the feature amount for recognition, but other feature amounts can be used. .

In the embodiment shown in FIG. 9, the recognition is performed by examining the average value of the pressure projection values. When a person moves from left to right as shown in FIG. 10A, the peak of the distribution of pressure projection values always occurs on the right side, and the valley of the distribution is on the left side, so this property is used. As shown in (b) of the figure, X = Xmin to X
Assuming that the distribution of pressure projection values is obtained in the section of = Xmax, the center of gravity X = G for this distribution shape is obtained. The average value of the pressure projection values on the left side of G is Px0, and the average value on the right side of G is Px1. If the absolute values of Px0 and Px1 are equal to or greater than a predetermined threshold value and Px1> Px0, it is recognized that the person is moving to the right. The same applies to movement in the Y-axis direction.

According to the present embodiment, since the average value of the pressure projection values before and after the center of gravity may be examined, the calculation is simple and the processing can be speeded up.

In the above-mentioned embodiments, the movement of the whole body of the person is recognized, but it is also possible to recognize the part moved in the body.

An example of recognition of a moving part will be described with reference to FIG.
As shown in the figure, the pressure value due to the movement of the person is from X = Xmin to X.
= Xmax and Y = Ymin to Y = Ymax, and there is a pressure projection value obtained by projecting in the X-axis and Y-axis directions. Here, Xmid and Ymid are the midpoints of the X-axis and Y-axis directions, and the moving part of the human image is
Consider division by areas such as four circled numbers shown in FIG. It is assumed that the determination thresholds are Cx and Cy, and that a region to be moved exists in a region where the absolute value of the pressure projection value exceeds these values.

In the example of the figure, only the region of the lower left portion is largely moved, and other regions are hardly moved. In such a case, it can be recognized that only the right foot of the person has moved significantly. According to this embodiment,
There is an advantage that you can know the part that has moved a lot in the body, and from this you can recognize the movement in more detail, for example, "kick with your right foot."

[0061]

The present invention has an effect that a silhouette of a moving object moving in a wide area can be stably obtained and action recognition can be performed. Further, since the variation amount of the silhouette shape is converted into density and stored, density of the motion can be recognized by examining density distribution as well as classification of motion.

[Brief description of drawings]

FIG. 1 is an explanatory diagram of a cellular diffusion model.

FIG. 2 is an explanatory diagram of motion recognition based on a cellular diffusion model.

FIG. 3 is an exemplary view of a pressure distribution of a person image.

FIG. 4 is an explanatory diagram of movement recognition processing.

FIG. 5 is an explanatory diagram of motion classification processing.

FIG. 6 is a flowchart of motion recognition processing according to an embodiment of the present invention.

FIG. 7 is a configuration diagram of a motion recognition processing device according to an embodiment of the present invention.

FIG. 8 is a configuration diagram of a human motion analysis system according to an embodiment of the present invention.

FIG. 9 is an explanatory diagram of motion recognition based on an average value.

FIG. 10 is an explanatory diagram of recognition of a moving part.

[Explanation of symbols]

1 ... Motion recognition processing unit, 2 ... Parameter input unit, 3 ... Diffusion calculation unit, 4 ... Motion identification unit, 5 ... Personal computer, 6 ... Image processing board, 7 ... Video camera, 20 ... Trajectory extraction processing unit, 30 ...
Behavior analysis processing unit, 40 ... Display processing unit.

─────────────────────────────────────────────────── ─── Continuation of the front page (56) Reference JP-A-3-243862 (JP, A) (58) Fields investigated (Int.Cl. ⁷ , DB name) G06T 7/00-7/60 H04N 7 / 18

Claims (9)

(57) [Claims] 1. A parameter input unit for inputting control parameters for controlling a spatial rate of change in a range of a specific region in an image and a temporal rate of change of a density value; The moving image of the object to be separated is divided into an object image and a background image, and at time t0, a predetermined density value is assigned to a pixel in a region where the object image exists, and based on the above control parameters, t1, t2,
... the range of the area and the density value that change depending on the object that moves with the passage of time, the expanded range of the area
A diffusion calculation unit that performs calculation processing to change the absolute value of the density value so as to decrease, and a feature amount is extracted from the distribution of the density value in the region, and based on the comparison between the feature amount and a predetermined value, the object A motion recognition processing device comprising: a motion identification unit that recognizes the motion of the motion recognition device. 2. The diffusion calculation unit according to claim 1, wherein when a pixel which is currently a background image is an object image before, a density value representing a negative value is assigned to the pixel, When a pixel which is an object image is a background image before, a density value expressing a positive value is assigned to the pixel, and a density value expressing a value of 0 is assigned to a portion having no change. Motion recognition processing device. 3. The motion recognition processing device according to claim 1, wherein the diffusion calculation unit processes the expansion of the area to which the density value is assigned by an expansion operation of an image. 4. In any one of claims 1 to 3, the diffusion calculation unit, an image created by changing the range and the density value of the area over time, and output to the display, to display A motion recognition processing device characterized by. 5. The one of claims 1 to 4, the operation identification unit operation, which comprises using a projection density value is projected distribution in a predetermined axial direction of the density value as the feature amount Recognition processing device. In any one of claims 6] Claims 1 to 4, the operation identification unit calculates a circumscribed rectangle which circumscribes the area where the density value is distributed, the axial direction parallel to the circumscribed rectangle of the sides <br / A motion recognition processing device characterized by calculating a density projection value obtained by projecting the above-mentioned density value on <>, Fourier-transforming from the distribution of the density projection value to calculate a spectrum, and matching the spectrum with a predetermined value. 7. In any of claims 1 to 4, the operation identification unit calculates a circumscribed rectangle which circumscribes the area where the density value is distributed, the ratio of the length of the circumscribed rectangle of vertical and horizontal sides A motion recognition processing device characterized by recognizing a motion based on the motion recognition processing device. In any of 8. claims 1 to 4, the operation identification unit calculates a circumscribed rectangle which circumscribes the area where the density value is distributed, the axial direction parallel to the circumscribed rectangle of the sides <br / Calculate the density projection value by projecting the above-mentioned density value on>, calculate the average value of the density in a predetermined section for the distribution of the above-mentioned density value, and recognize the action based on the magnitude relationship of the above average value. A motion recognition processing device characterized by. 9. The motion recognition processing device according to claim 1 , wherein the motion of each moving object is identified from the moving image data obtained by capturing the moving motion of a person or the like, and the motion is output as independent motion data. And a behavior analysis processing unit that identifies the behavior as a group from the trajectory data of the moving object and outputs it as group movement data, and performs statistical processing and three-dimensional graphics processing from the single movement data and the collective movement data. And a display processing unit that outputs statistical data and CG image data.