JP7173535B2 - Motion extraction device, motion extraction method, and program - Google Patents

Description

The present invention relates to a motion extraction device and a motion extraction method for extracting a video of a specific motion from video data, and further to a program for realizing these.

2. Description of the Related Art Conventionally, in the field of sports, the motion of athletes has been analyzed from videos. Motion analysis based on such images is very important from the viewpoint of improvement of technique, prevention of injuries, and the like. Moreover, in recent years, with the spread of smart phones, such motion analysis has become easier to perform.

By the way, usually, recorded video often shows motions that do not need to be analyzed, so when performing motion analysis using video, first, extract the part that shows the motion to be analyzed from the video. There is a need to. However, since such extraction work is usually performed manually while frame-by-frame advancing, the more the number of images to be analyzed increases, the greater the burden on the analyst becomes.

Therefore, Patent Literature 1 discloses an apparatus for automatically extracting a specific action from video. Specifically, the apparatus disclosed in Patent Literature 1 first generates a difference image between frames, calculates a feature quantity indicating the characteristics of a person's motion from the difference image, and calculates the feature quantity in a time series. , and dimensionally compressed, thereby generating reference time-series data. Subsequently, the device disclosed in Patent Document 1 calculates the correlation coefficient between the generated reference time-series data and the reference time-series data stored in advance in the storage device. is equal to or greater than the threshold of is extracted as a section of a specific motion.

Thus, according to the device disclosed in Patent Document 1, it is possible to automatically extract a specific action from the video. In addition, since the device disclosed in Patent Document 1 is realized by a computer, if a program that realizes this device is executed on a smartphone, the smartphone can automatically extract a specific action from the video. It is thought that it will be possible to

JP 2008-287594 A

However, in order to accurately extract a motion using the device disclosed in Patent Document 1, it is necessary to prepare reference time-series data as a standard for each person who performs the motion in advance. In addition, manual editing is required in creating reference time-series data that serves as a standard. In other words, with the device disclosed in Patent Document 1, it is difficult to reduce the burden on the analyst.

An example of the object of the present invention is to provide a motion extraction device, a motion extraction method, and a program capable of solving the above problems and automatically extracting a specific motion from a video while reducing the burden on the analyst. That's what it is.

In order to achieve the above object, a motion extraction device according to one aspect of the present invention includes:
a data acquisition unit that acquires video data to be processed;
a trajectory identifying unit that identifies a trajectory of specific feature points for each frame constituting the acquired video data;
a histogram generating unit that extracts each of a plurality of regions of the frame for each of the frames, and generates a histogram representing the direction of pixel motion for each of the plurality of extracted regions;
For each frame, based on the identified trajectory and the histogram in each of the plurality of regions, it is determined whether or not the frame includes an image of a person performing a specific action, and a motion extracting unit that extracts a frame determined to contain an image of a person performing a motion;
characterized by comprising

Further, in order to achieve the above object, a motion extraction method according to one aspect of the present invention includes:
(a) obtaining video data to be processed;
(b) identifying a trajectory of specific feature points for each frame constituting the acquired video data;
(c) for each of said frames, retrieving each of a plurality of regions of said frame and generating a histogram representing the direction of pixel motion for each of said plurality of retrieved regions;
(d) for each frame, based on the identified trajectory and the histogram in each of the plurality of regions, determining whether the frame includes an image of a person performing a specific action; extracting frames determined to contain video of a person performing the specific action;
characterized by having

Furthermore, in order to achieve the above object, the program in one aspect of the present invention is
to the computer,
(a) obtaining video data to be processed;
(b) identifying a trajectory of specific feature points for each frame constituting the acquired video data;
(c) for each of said frames, retrieving each of a plurality of regions of said frame and generating a histogram representing the direction of pixel motion for each of said plurality of retrieved regions;
(d) for each frame, based on the identified trajectory and the histogram in each of the plurality of regions, determining whether the frame includes an image of a person performing a specific action; extracting frames determined to contain video of a person performing the specific action;
is characterized by executing

As described above, according to the present invention, it is possible to automatically extract a specific action from a video while reducing the burden on the analyst.

FIG. 1 is a block diagram showing a schematic configuration of a motion extraction device according to an embodiment of the present invention. FIG. 2 is a block diagram specifically showing the configuration of the motion extraction device according to the embodiment of the present invention. FIG. 3 is a diagram illustrating processing performed by the skip processing unit in the embodiment of the present invention. FIG. 4 is a diagram explaining the function of the trajectory specifying unit according to the embodiment of the present invention. FIG. 5 is a diagram for explaining the function of the histogram generator in the embodiment of the invention. FIG. 6 is a diagram showing an example of a state transition table used in the embodiment of the invention. FIG. 7 is a diagram showing an example of time-series changes in the intensity of the trajectory created by the release point specifying unit in the embodiment of the present invention. FIG. 8 is a flowchart showing the operation of the motion extraction device according to the embodiment of the present invention. FIG. 9 is a block diagram showing an example of a computer that implements the motion extraction device 10 according to the embodiment of the present invention.

(Embodiment)
A motion extraction device, a motion extraction method, and a program according to embodiments of the present invention will be described below with reference to FIGS. 1 to 9. FIG.

[Device configuration]
First, a schematic configuration of the motion extraction device according to the present embodiment will be described. FIG. 1 is a block diagram showing a schematic configuration of a motion extraction device according to an embodiment of the present invention.

A motion extraction device 10 according to the present embodiment shown in FIG. 1 is a device for extracting a video of a person performing a specific motion from video data. As shown in FIG. 1 , the motion extraction device 10 includes a data acquisition unit 11 , a locus identification unit 12 , a histogram generation unit 13 and a motion extraction unit 14 .

Among these, the data acquisition unit 11 acquires video data to be processed. The trajectory specifying unit 12 specifies the trajectory of specific feature points for each frame that constitutes the acquired video data. The histogram generation unit 13 extracts each of a plurality of areas of each frame, and generates a histogram representing the direction of pixel motion for each of the extracted areas.

First, for each frame, based on the trajectory of specific feature points and histograms in each of a plurality of regions, the motion extraction unit 14 determines whether the frame includes an image of a person performing a specific motion. judge. Next, the action extracting unit 14 extracts the frame determined to contain the video of the person performing the specific action.

In this way, since the motion extraction device 10 can use the trajectory of the feature points and the histogram of each region for each frame, a specific motion can be performed in that frame without imposing any burden on the analyst. Whether or not a person is captured can be determined with high accuracy. According to the motion extraction device 10, it is possible to automatically extract a specific motion from a video while reducing the burden on the analyst.

Next, the configuration and functions of the motion extraction device 10 according to the present embodiment will be described more specifically with reference to FIGS. 2 to 6. FIG. FIG. 2 is a block diagram specifically showing the configuration of the motion extraction device according to the embodiment of the present invention.

As shown in FIG. 2 , in the present embodiment, the motion extraction device 10 is constructed by a program that runs on the operating system 21 of the terminal device 20 . The terminal device 20 includes a camera 22 and a display device 23 such as a liquid crystal display. In addition, video data of video captured by the camera 22 is output to the motion extraction device 10 . Specific examples of the terminal device 20 include smartphones, tablet terminals, notebook PCs (Personal Computers), and the like.

Further, as shown in FIG. 2, in the present embodiment, the motion extraction device 10 includes, in addition to the data acquisition unit 11, the trajectory identification unit 12, the histogram generation unit 13, and the motion extraction unit 14 described above, skip processing It has a unit 15 and a release point specifying unit 16 .

The data acquisition unit 11 acquires video data output from the camera 22 described above in this embodiment. In addition, in the present embodiment, the specific motion to be extracted is not particularly limited. An example will be described. In this case, the data acquisition unit 11 acquires video data of a person who is throwing an object.

The skip processing unit 15 first obtains the difference between each frame and a background image prepared in advance for each frame. Specifically, the background image may be, for example, the frame immediately preceding the current frame being processed. Therefore, assuming that the current time (current frame) is t, the difference to be obtained is G(t), and the difference between the current frame and the previous frame is B(t), the skip processing unit 15 performs, for example, Equation 1 below. can be used to calculate the difference G(t). In Equation 1 below, W1, W2, W3, .

[Number 1]
G(t)=W1*B(t)+W2*G(t-1)+W3*G(t-2)+...

Subsequently, the skip processing unit 15 identifies edges in the vertical direction (y-axis direction) of the image obtained from the obtained difference, and obtains the coordinates of the identified edges in the horizontal direction (x-axis direction). Furthermore, the skip processing unit 15 identifies frames in which a specific action is not performed based on time-series changes in horizontal coordinates in each frame, and deletes the identified frames from the video data.

Here, deletion of unnecessary frames by the skip processing unit 15 will be described with reference to FIG. FIG. 3 is a diagram illustrating processing performed by the skip processing unit in the embodiment of the present invention. In FIG. 3, the horizontal axis indicates frames (time), and the left vertical axis indicates coordinates in the x-axis direction. The vertical axis on the right indicates binary flags indicating the presence or absence of frame deletion (skip). L indicates the coordinate in the x-axis direction with the maximum edge amount in each frame.

In the example of FIG. 3, the image data includes an image of a person performing a pitching motion. In this case, the person moves from left to right in the image while the ball is being pitched, so the value of the coordinate of the edge in the x-axis direction increases. However, the coordinate value of the edge in the x-axis direction does not change before the start of the pitch, and the coordinate value decreases after the end of the pitch.

Therefore, since the frame before the start of the pitching and the frame after the end of the action are unnecessary frames, the skip processing unit 15 skips the frames in the section where the coordinate value of the edge in the x-axis direction increases. leave only On the other hand, the skip processing unit 15 deletes the remaining frames as frames in which no action is performed. In this way, the skip processing unit 15 deletes unnecessary frames that do not show motion from the frames that make up the video data.

When frames are deleted by the skip processing unit 15, the trajectory identification unit 12, the histogram generation unit 13, and the motion extraction unit 14 target the video data from which the frames have been deleted by the skip processing unit 15, that is, the deleted frames. The processing is performed on the frames that did not have any (hereinafter referred to as "target frames").

In the present embodiment, the trajectory identifying unit 12 first extracts specific feature points for each target frame using, for example, Harris corner detection, and further extracts the extracted feature points between frames (time Calculate the optical flow at t and time t+1).

Next, as shown in FIG. 4, the trajectory identifying unit 12 connects the calculated optical flows in the time-series direction to identify the trajectory of the feature points. FIG. 4 is a diagram explaining the function of the trajectory specifying unit according to the embodiment of the present invention. The example of FIG. 4 shows main frames from the start to the end of pitching. Also, the trajectory of the identified feature points is represented by a horizontal line in each frame.

In this embodiment, as shown in FIG. 5, the histogram generation unit 13 continuously extracts each of a plurality of regions of each target frame in the horizontal direction, and calculates the number of pixels for each extracted region. Generate a histogram showing the direction of motion. FIG. 5 is a diagram for explaining the function of the histogram generator in the embodiment of the invention.

Specifically, as shown in the upper and middle parts of FIG. 5, the histogram generation unit 13 first sets a plurality of regions so as to overlap each other in the horizontal direction for each target frame, and sequentially Take out. In FIG. 5, the upper part shows one target frame. The middle row shows a state in which a plurality of areas are set. In the example of FIG. 5, nine areas are set for one target frame. Also, for the sake of explanation, the positions of the regions are shifted in the vertical direction.

Subsequently, as shown in the lower part of FIG. 5, the histogram generator 13 generates a histogram representing the direction of pixel motion for each region. Specifically, the histogram generation unit 13 obtains a HOF (Histogram of Optical Flow) feature amount composed of a histogram for each region, and further normalizes each block of the histogram. The lower part of FIG. 5 shows each normalized block (normalized block). Also, as indicated by arrows, each normalized block corresponds to each direction of pixel movement (trajectory) (up, upper right, right, lower right, lower, lower left, left, upper left). In addition, in the example of FIG. 5, the direction of the pixel is indicated by an arrow. Also, the histogram shown in the lower part of FIG. 5 is the histogram generated for the fifth area.

In the present embodiment, the motion extraction unit 14 stores the trajectory identified by the trajectory identification unit 12 and the histogram generated by the histogram generation unit 13 for each target frame in, for example, the state transition table shown in FIG. match. Based on the collation result, the action extracting unit 14 determines whether or not each frame contains an image of a person performing a specific action, and if so, extracts this frame.

FIG. 6 is a diagram showing an example of a state transition table used in the embodiment of the invention. In the example of FIG. 6, the state transition table defines corresponding state transitions for each combination of each state (initial state, pitching in progress, pitching end) and each event. Also, in the example of FIG. 6, a state transition table used when a right-handed thrower performs a pitching motion is shown. Each "event" is defined by either or both trajectory and histogram conditions.

For example, suppose that "there is no trajectory longer than a certain length" in the frame at time t, but "there is a rightward trajectory longer than the certain length" in the frame at time t+1. In this case, as can be seen from the "initial state" column in FIG. 6, the state transitions from the initial state to the pitching state. Accordingly, at this time, the motion extracting unit 14 determines that the frame at time t+1 includes an image of a person performing a pitching motion, and extracts this frame. In the example of FIG. 6, the motion extracting unit 14 extracts a frame in which the state can be determined to be "balling".

First, the release point identifying unit 16 detects the presence of a person performing a pitching motion based on the histogram values of pixels moving in a specific direction from each region for each frame extracted by the motion extracting unit 14. Select the areas that you consider to be

More specifically, since the motion is a pitching motion, the release point identifying unit 16 determines the value of the normalized block (see the bottom of FIG. 5) corresponding to each of the upper right, lower, and lower right in each of the extracted frames. Identify the area where is above a certain level. In the example of FIG. 5, the release point identifying unit 16 identifies the fourth and fifth regions.

Subsequently, the release point specifying unit 16 obtains the strength of the trajectory in the selected region for each extracted frame, and calculates the moment when the person releases the object (ball) based on the time-series change in the strength of the obtained trajectory. to identify the frame of

More specifically, the release point specifying unit 16 extracts rightward trajectories in each of the fourth region and the fifth region from the trajectory specified by the trajectory specifying unit 12 for each extracted frame. Seek strength. Then, as shown in FIG. 7, the release point specifying unit 16 connects the intensities of the trajectory in the rightward direction obtained for each frame to obtain the time-series change in the trajectory strength. FIG. 7 is a diagram showing an example of time-series changes in the intensity of the trajectory created by the release point specifying unit in the embodiment of the present invention.

Subsequently, as shown in FIG. 7, the release point specifying unit 16 extracts only the frames in the middle portion, for example, the frames from the first half (1/3) to the latter half (2/3) of the extracted frames. . After that, the release point specifying unit 16 specifies a frame in which the strength of the trajectory is higher among the extracted frames, and specifies the specified frame as the frame at the moment when the person released the ball.

[Device operation]
Next, the operation of the motion extraction device 10 according to the embodiment of the present invention will be explained using FIG. FIG. 8 is a flowchart showing the operation of the motion extraction device according to the embodiment of the present invention. 1 to 7 will be appropriately referred to in the following description. Further, in the present embodiment, the motion extraction method is carried out by operating the motion extraction device. Therefore, the description of the motion extraction method in the present embodiment is replaced with the description of the motion extraction device 10 below.

As shown in FIG. 8, first, the data acquisition unit 11 acquires video data output from the camera 22 of the terminal device 20 (step A1).

Next, the skip processing unit 15 identifies frames in which no pitching motion is performed from the video data acquired in step A1, and deletes them (step A2).

Specifically, in step A2, the skip processing unit 15 obtains the difference between each frame and a background image prepared in advance for each frame. An edge is specified, and the coordinate of the specified edge in the horizontal direction (x-axis direction) is obtained. Furthermore, the skip processing unit 15 identifies frames in which a specific action is not performed based on time-series changes in horizontal coordinates in each frame, and deletes the identified frames from the video data.

Next, the trajectory specifying unit 12 selects frames (target frames) that were not deleted in step A2.
For each, specific feature points are extracted, and the trajectory of the extracted feature points is specified (step A3).

Specifically, in step A3, the trajectory identifying unit 12 extracts specific feature points for each target frame using, for example, Harris corner detection, and calculates the optical flow between frames for the extracted feature points. (Optical Flow) is calculated. Then, as shown in FIG. 4, the trajectory identifying unit 12 connects the calculated optical flows in the time-series direction to identify the trajectory of the feature points.

Next, the histogram generation unit 13 continuously extracts each of a plurality of regions of each target frame in the horizontal direction, and generates a histogram indicating the direction of pixel motion for each of the extracted regions ( Step A4).

Specifically, in step A4, the histogram generation unit 13 first sets a plurality of regions for each target frame so as to overlap each other in the horizontal direction, as shown in the upper and middle stages of FIG. Take out the areas in order. Subsequently, as shown in the lower part of FIG. 5, the histogram generator 13 generates a histogram representing the direction of pixel motion for each region.

Next, the motion extraction unit 14 collates the trajectory identified in step A3 and the histogram generated in step A4 for each target frame with the state transition table shown in FIG. is included, and a frame including a pitching motion is extracted (step A5).

Next, the release point specifying unit 16 specifies the frame at the moment when the person releases the object (ball) from among the frames extracted in step A5 (step A6).

Specifically, in step A5, the release point identifying unit 16 determines that there is a person performing a pitching motion based on the value of the histogram of pixels moving in a specific direction in each extracted frame. Select the area to consider.

Subsequently, the release point specifying unit 16 obtains the strength of the trajectory in the selected region for each extracted frame, and calculates the moment when the person releases the object (ball) based on the time-series change in the strength of the obtained trajectory. identify the frame of After that, the release point specifying unit 16 outputs the specified frame to an application program or the like operating on the terminal device 20 .

[Effects of Embodiment]
As described above, in the present embodiment, the motion extraction device 10 automatically identifies the frame in which the pitching motion is performed, and further identifies the frame in which the ball is released from among the identified frames. do. According to the present embodiment, the analyst can analyze in detail a person's motion, especially a motion of throwing an object, without performing any action that would be a burden.

In addition, in the present embodiment, if the pitching motion performed by a specific player is photographed a plurality of times, the release points can be compared for each pitching motion. In addition, since the frame of the release point can be specified in each pitching motion, if the video data of each pitching motion is cut out based on this frame, it becomes easy to compare the pitching motions. Factor analysis becomes possible.

[program]
The program in this embodiment may be any program that causes a computer to execute steps A1 to A6 shown in FIG. By installing this program in a computer and executing it, the motion extraction device 10 and the motion extraction method according to the present embodiment can be realized. In this case, the computer processor functions as a data acquisition unit 11, a trajectory identification unit 12, a histogram generation unit 13, a motion extraction unit 14, a skip processing unit 15, and a release point identification unit 16, and performs processing. Moreover, as a computer, specifically, the terminal device 20 mentioned above, for example, a smart phone, a tablet-type terminal, a notebook PC, etc. are mentioned.

Also, the program in this embodiment may be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer functions as one of the data acquisition unit 11, trajectory identification unit 12, histogram generation unit 13, motion extraction unit 14, skip processing unit 15, and release point identification unit 16. Also good.

Here, an example of a computer that implements the motion extraction device 10 by executing the program according to the present embodiment will be described with reference to FIG. FIG. 9 is a block diagram showing an example of a computer that implements the motion extraction device 10 according to the embodiment of the present invention.

As shown in FIG. 9, computer 110 includes CPU 111 , main memory 112 , storage device 113 , input interface 114 , display controller 115 , data reader/writer 116 and communication interface 117 . These units are connected to each other via a bus 121 so as to be able to communicate with each other. The computer 110 may include a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array) in addition to the CPU 111 or instead of the CPU 111 .

The CPU 111 expands the programs (codes) of the present embodiment stored in the storage device 113 into the main memory 112 and executes them in a predetermined order to perform various calculations. The main memory 112 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory). Also, the program in the present embodiment is provided in a state stored in computer-readable recording medium 120 . It should be noted that the program in this embodiment may be distributed on the Internet connected via communication interface 117 .

Further, as a specific example of the storage device 113, in addition to a hard disk drive, a semiconductor storage device such as a flash memory can be cited. Input interface 114 mediates data transmission between CPU 111 and input devices 118 such as a keyboard and mouse. The display controller 115 is connected to the display device 119 and controls display on the display device 119 .

Data reader/writer 116 mediates data transmission between CPU 111 and recording medium 120 , reads programs from recording medium 120 , and writes processing results in computer 110 to recording medium 120 . Communication interface 117 mediates data transmission between CPU 111 and other computers.

Specific examples of the recording medium 120 include general-purpose semiconductor storage devices such as CF (Compact Flash (registered trademark)) and SD (Secure Digital); magnetic recording media such as flexible disks; An optical recording medium such as a ROM (Compact Disk Read Only Memory) can be used.

Note that the motion extraction device 10 according to the present embodiment can also be realized by using hardware corresponding to each part instead of a computer in which a program is installed. Furthermore, the motion extraction device 10 may be partially realized by a program and the rest by hardware.

Some or all of the above-described embodiments can be expressed by (Appendix 1) to (Appendix 12) described below, but are not limited to the following descriptions.

(Appendix 1)
a data acquisition unit that acquires video data to be processed;
a trajectory identifying unit that identifies a trajectory of specific feature points for each frame constituting the acquired video data;
a histogram generating unit that extracts each of a plurality of regions of the frame for each of the frames, and generates a histogram representing the direction of pixel motion for each of the plurality of extracted regions;
For each frame, based on the identified trajectory and the histogram in each of the plurality of regions, it is determined whether or not the frame includes an image of a person performing a specific action, and a motion extracting unit that extracts a frame determined to contain an image of a person performing a motion;
A motion extraction device characterized by comprising:

(Appendix 2)
The motion extraction device according to appendix 1,
the histogram generation unit extracting each of the plurality of regions continuously in the horizontal direction of the frame for each frame;
The action extracting unit determines whether the frame includes an image of a person performing the specific action based on the specified trajectory and the histogram in each of the plurality of areas for each frame. determine the
A motion extraction device characterized by:

(Appendix 3)
The motion extraction device according to appendix 2,
the specific motion is a motion of throwing an object;
The motion extraction unit determines, for each frame, whether or not the frame includes an image of a person performing a motion of throwing an object;
The motion extraction device is
Selecting from among the plurality of regions in each of the extracted frames, a region considered to contain the person throwing the object based on the histogram values of pixels moving in a specific direction. ,
obtaining the intensity of the trajectory in the selected region for each frame;
further comprising a release point identifying unit that identifies the frame at the moment when the person releases the object based on the time-series change in the intensity of the trajectory obtained;
A motion extraction device characterized by:

(Appendix 4)
The motion extraction device according to any one of Appendices 1 to 3,
For each frame, a difference from a background image prepared in advance is obtained, an edge in the vertical direction of the image obtained from the obtained difference is identified, the coordinates of the identified edge in the horizontal direction are obtained, and the A skip processing unit that identifies frames in which the specific action is not performed based on time-series changes in horizontal coordinates, and deletes the identified frames from the video data,
The trajectory identifying unit identifies the trajectory of the specific feature point for the video data from which frames have been deleted by the skip processing unit,
The histogram generation unit generates the histogram for the video data from which frames have been deleted by the skip processing unit,
The motion extracting unit determines whether or not the video data from which the frame has been deleted by the skip processing unit includes a video of a person performing a specific motion.
A motion extraction device characterized by:

(Appendix 5)
(a) obtaining video data to be processed;
(b) identifying a trajectory of specific feature points for each frame constituting the acquired video data;
(c) for each of said frames, retrieving each of a plurality of regions of said frame and generating a histogram representing the direction of pixel motion for each of said plurality of retrieved regions;
(d) for each frame, based on the identified trajectory and the histogram in each of the plurality of regions, determining whether the frame includes an image of a person performing a specific action; extracting frames determined to contain video of a person performing the specific action;
A motion extraction method characterized by comprising:

(Appendix 6)
The motion extraction method according to appendix 5,
In step (c), each of the plurality of regions is extracted continuously in the horizontal direction of the frame for each frame;
In step (d), the frame includes an image of a person performing the specific action based on the identified trajectory and the histogram in each of the plurality of regions for each frame. determine whether there is
A motion extraction method characterized by:

(Appendix 7)
The motion extraction method according to appendix 6,
the specific motion is a motion of throwing an object;
In step (d), for each frame, determining whether the frame includes an image of a person throwing an object;
The motion extraction method is
(e) the person performing the action of throwing the object based on the histogram values of pixels moving in a specific direction from among the plurality of regions in each of the frames extracted in step (d); choose a region that it considers to exist,
obtaining the intensity of the trajectory in the selected region for each frame;
further comprising identifying a frame at a moment when the person released the object based on the time-series change in intensity of the trajectory obtained;
A motion extraction method characterized by:

(Appendix 8)
The motion extraction method according to any one of Appendices 5 to 7,
(f) for each frame, obtain a difference from a background image prepared in advance, identify an edge in the vertical direction of the image obtained from the obtained difference, obtain the horizontal coordinates of the identified edge, identifying a frame in which the specific action is not performed based on the time-series change of the horizontal coordinate in the frame, and deleting the identified frame from the video data;
In the step (b), identifying the trajectory of the specific feature point for the video data from which the frame has been deleted in the step (f),
In step (c), generating the histogram for the video data from which frames have been deleted in step (f);
In step (d), determining whether or not the image data from which the frame has been deleted in step (f) contains an image of a person performing a specific action;
A motion extraction method characterized by:

(Appendix 9)
to the computer,
(a) obtaining video data to be processed;
(b) identifying a trajectory of specific feature points for each frame constituting the acquired video data;
(c) for each of said frames, retrieving each of a plurality of regions of said frame and generating a histogram representing the direction of pixel motion for each of said plurality of retrieved regions;
(d) for each frame, based on the identified trajectory and the histogram in each of the plurality of regions, determining whether the frame includes an image of a person performing a specific action; extracting frames determined to contain video of a person performing the specific action;
The program that causes the to run.

(Appendix 10)
The program according to Appendix 9,
In step (c), each of the plurality of regions is extracted continuously in the horizontal direction of the frame for each frame;
In step (d), the frame includes an image of a person performing the specific action based on the identified trajectory and the histogram in each of the plurality of regions for each frame. determine whether there is
A program characterized by

(Appendix 11)
The program according to Appendix 10,
the specific motion is a motion of throwing an object;
In step (d), for each frame, determining whether the frame includes an image of a person throwing an object;
to the computer;
(e) the person performing the action of throwing the object based on the histogram values of pixels moving in a specific direction from among the plurality of regions in each of the frames extracted in step (d); choose a region that it considers to exist,
obtaining the intensity of the trajectory in the selected region for each frame;
further executing a step of identifying the frame at the moment when the person released the object based on the time-series change in the intensity of the trajectory obtained;
A program characterized by

(Appendix 12)
The program according to any one of Appendices 9 to 11,
to the computer;
(f) obtaining a difference from a background image prepared in advance for each frame, identifying an edge in the vertical direction of the image obtained from the obtained difference, and obtaining horizontal coordinates of the identified edge;
Furthermore, a step of identifying frames in which the specific action is not performed based on time-series changes in the horizontal coordinates in each frame and deleting the identified frames from the video data is further executed. let
In the step (b), identifying the trajectory of the specific feature point for the video data from which the frame has been deleted in the step (f),
In step (c), generating the histogram for the video data from which frames have been deleted in step (f);
In step (d), determining whether or not the image data from which the frame has been deleted in step (f) contains an image of a person performing a specific action;
A program characterized by

As described above, according to the present invention, it is possible to automatically extract a specific action from a video while reducing the burden on the analyst. INDUSTRIAL APPLICABILITY The present invention is useful in various fields that require analysis of human motion, such as the field of sports.

REFERENCE SIGNS LIST 10 motion extraction device 11 data acquisition unit 12 trajectory identification unit 13 histogram generation unit 14 motion extraction unit 15 skip processing unit 16 release point identification unit 20 terminal device 21 operating system 21
22 camera 23 display device 110 computer 111 CPU
112 main memory 113 storage device 114 input interface 115 display controller 116 data reader/writer 117 communication interface 118 input device 119 display device 120 recording medium 121 bus

Claims (6)

a data acquisition unit that acquires video data to be processed;
a trajectory identifying unit that identifies a trajectory of specific feature points for each frame constituting the acquired video data;
a histogram generation unit configured to continuously extract each of a plurality of regions of the frame in the horizontal direction of the frame for each frame and generate a histogram representing the direction of pixel motion for each of the plurality of extracted regions;
For each frame, based on the identified trajectory and the histogram in each of the plurality of regions, it is determined whether or not the frame includes an image of a person throwing an object, and a motion extracting unit that extracts a frame determined to contain an image of a person performing a motion of throwing a
Selecting from among the plurality of regions in each of the extracted frames, a region considered to contain the person throwing the object based on the histogram values of pixels moving in a specific direction. ,
obtaining the intensity of the trajectory in the selected region for each frame;
a release point identifying unit that identifies a frame at the moment when the person releases the object based on the time-series change in intensity of the obtained trajectory;
A motion extraction device characterized by comprising: The motion extraction device according to claim 1 ,
For each frame, a difference from a background image prepared in advance is obtained, an edge in the vertical direction of the image obtained from the obtained difference is identified, the coordinates of the identified edge in the horizontal direction are obtained, and the A skip processing unit that identifies frames in which the specific action is not performed based on time-series changes in horizontal coordinates, and deletes the identified frames from the video data,
The trajectory identifying unit identifies the trajectory of the specific feature point for the video data from which frames have been deleted by the skip processing unit,
The histogram generation unit generates the histogram for the video data from which frames have been deleted by the skip processing unit,
The motion extracting unit determines whether or not the video data from which the frame has been deleted by the skip processing unit includes a video of a person performing a specific motion.
A motion extraction device characterized by: (a) obtaining video data to be processed;
(b) identifying a trajectory of specific feature points for each frame constituting the acquired video data;
(c) for each frame, extracting each of a plurality of regions of the frame successively in the horizontal direction of the frame, and generating a histogram representing the direction of pixel motion for each of the extracted regions;
(d) for each frame, based on the identified trajectory and the histogram in each of the plurality of regions, determining whether the frame includes an image of a person throwing an object; , extracting a frame determined to contain an image of the person performing the action of throwing the object ;
(e) an area considered to contain the person throwing the object based on the histogram values of pixels moving in a specific direction from among the plurality of areas in each of the extracted frames; elect the
obtaining the intensity of the trajectory in the selected region for each frame;
identifying a frame at a moment when the person releases the object based on the obtained time-series change in intensity of the trajectory;
A motion extraction method characterized by comprising: The motion extraction method according to claim 3 ,
(f) for each frame, obtain a difference from a background image prepared in advance, identify an edge in the vertical direction of the image obtained from the obtained difference, obtain the horizontal coordinates of the identified edge, identifying a frame in which the specific action is not performed based on the time-series change of the horizontal coordinate in the frame, and deleting the identified frame from the video data;
In the step (b), identifying the trajectory of the specific feature point for the video data from which the frame has been deleted in the step (f),
In step (c), generating the histogram for the video data from which frames have been deleted in step (f);
In step (d), determining whether or not the image data from which the frame has been deleted in step (f) contains an image of a person performing a specific action;
A motion extraction method characterized by: to the computer,
(a) obtaining video data to be processed;
(b) identifying a trajectory of specific feature points for each frame constituting the acquired video data;
(c) for each frame, extracting each of a plurality of regions of the frame successively in the horizontal direction of the frame, and generating a histogram representing the direction of pixel motion for each of the extracted regions;
(d) for each frame, based on the identified trajectory and the histogram in each of the plurality of regions, whether or not the frame includes an image of a person performing a specific action of throwing an object ; extracting frames determined to contain video of the person performing the action of throwing the object ;
(e) an area considered to contain the person throwing the object based on the histogram values of pixels moving in a specific direction from among the plurality of areas in each of the extracted frames; elect the
obtaining the intensity of the trajectory in the selected region for each frame;
identifying a frame at a moment when the person releases the object based on the obtained time-series change in intensity of the trajectory;
The program that causes the to run. The program according to claim 5 ,
to the computer;
(f) for each frame, obtain a difference from a background image prepared in advance, identify an edge in the vertical direction of the image obtained from the obtained difference, obtain the horizontal coordinates of the identified edge, further executing a step of identifying a frame in which the specific action is not performed based on the time-series change of the horizontal coordinate in the frame, and deleting the identified frame from the video data;
In the step (b), identifying the trajectory of the specific feature point for the video data from which the frame has been deleted in the step (f),
In step (c), generating the histogram for the video data from which frames have been deleted in step (f);
In step (d), determining whether or not the image data from which the frame has been deleted in step (f) contains an image of a person performing a specific action;
A program characterized by

Images