JP7659158B2 - Motion recognition system, motion recognition method, and motion recognition program - Google Patents

Description

The present invention relates to a motion recognition system, a motion recognition method, and a motion recognition program.

The technology to automatically recognize specific human actions from images is expected to be applied to a variety of fields. For example, cameras could be installed inside a store that sells goods, and customers' actions could be automatically identified from the images they capture. In this case, applications such as realizing an unmanned store by detecting customers' purchasing actions and automatically processing payments, detecting fraudulent or dangerous customer behavior, and optimizing product displays and prices by analyzing customer behavior would become possible.

For automatic recognition of movements, for example, the following image search device has been proposed. This image search device recognizes posture information of the search target from an input image, extracts features from the posture information and the input image, associates them with the input image, and stores them in a database. It also generates a search query from posture information specified by the user, and searches the database for images containing similar postures according to the search query.

There is also technology that automatically recognizes actions by learning from images. For example, a method has been proposed in which a system learns from training images in which the positions and types of actions to be detected within the image are specified, and then the learning results are used to estimate the human actions and their types in unknown test images.

JP 2019-91138 A

Chunhui Gu, 11 others, AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions, IEEE Conference on Computer Vision and Pattern Recognition(CVPR) 2018

When judging whether a specific action is captured in a query image that captures the specific action and an image to be recognized for the action based on the similarity between the latter image, for example, feature amounts are compared between the feature points of each image. Here, the structure of the human body is complex, and even when a person performs a specific action, the entire body may take a variety of postures. For this reason, there is a problem in that it is not possible to improve the accuracy of recognizing the action unless a large number of query images in which the specific action is performed in various postures are used.

In one aspect, the present invention aims to provide an action recognition system, an action recognition method, and an action recognition program that are capable of performing highly accurate action recognition using a small number of query images.

In one proposal, a motion recognition system is provided that has an input receiving unit and a determination unit. In this motion recognition system, the input receiving unit plays and displays a first video of a person performing a specific motion, and receives, during playback and display, an instruction input that indicates a position related to the specific motion on a frame of the first video. The determination unit calculates a similarity between the first video and the second video based on a comparison result of feature amounts between a second feature point group that includes one or more feature points included in the second video and a first feature point group that includes two or more feature points included in the first video, and determines whether or not the second video includes a specific motion based on the calculated similarity. When calculating the similarity, the determination unit assigns a larger weight to the comparison result of feature amounts for feature points in the first feature point group that are closer to the position indicated by the instruction input.

In one proposal, a motion recognition method is provided in which a computer plays and displays a first video of a person performing a specific motion, accepts, during playback and display, a command input indicating a position related to the specific motion on a frame of the first video, calculates a similarity between the first video and the second video based on a comparison result of features between a second feature point group including one or more feature points included in the second video and a first feature point group including two or more feature points included in the first video, and determines whether the second video includes the specific motion based on the calculated similarity. In this motion recognition method, in calculating the similarity, a greater weight is assigned to the comparison result of features of the first feature point group that are closer to the position indicated by the command input.

Furthermore, in one proposal, a motion recognition program is provided that causes a computer to execute processing similar to the motion recognition method described above.

In one aspect, highly accurate action recognition can be performed using a small number of query images.

1 is a diagram illustrating a configuration example and a processing example of an action recognition system according to a first embodiment; FIG. 13 illustrates an example of a configuration of an information processing system according to a second embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device. 2 is a block diagram showing an example of the configuration of processing functions included in an information processing device; 11 is an example of a flowchart showing an overall processing procedure of a motion detection and analysis process performed by an information processing device. FIG. 11 is a diagram for explaining a motion feature amount; 13 is an example of a flowchart showing a process of calculating a motion feature amount from a sample video. FIG. 11 is a diagram showing a first example of motion feature amounts; FIG. 13 is a diagram showing a third example of motion features; FIG. 13 is a diagram showing an example of a time-space instruction input. FIG. 11 is a diagram for explaining a calculation process of an attention level. 13 is an example of a flowchart illustrating a calculation process of an attention level according to a time-space instruction input. 13 is an example of a flowchart showing a process of calculating a motion feature amount from a detection target moving image. 11A and 11B are diagrams for explaining a similarity search process between a sample video and a detection target video. 11 is a diagram for explaining a similarity calculation between an input section video image and a partial video image; FIG. 13 is an example of a flowchart illustrating a motion detection process based on a similarity degree. FIG. 13 is a diagram illustrating an example of an analysis table. 13 illustrates an example of a configuration of processing functions included in an information processing device according to a third embodiment. 13 is a flowchart illustrating an example of an overall processing procedure of a motion detection and analysis process according to a third embodiment. 13 is a flowchart illustrating an example of a learning and re-determination process. FIG. 13 illustrates an example of a configuration of an information processing system according to a fourth embodiment.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
First Embodiment
Fig. 1 is a diagram showing an example of the configuration and processing of a motion recognition system according to a first embodiment. The motion recognition system shown in Fig. 1 is a system that executes motion recognition processing to determine whether a person performing a specific motion is captured in a video image, and is realized, for example, as a computer. The motion recognition system 1 has an input receiving unit 2 and a determination unit 3. The processing of the input receiving unit 2 and the determination unit 3 is realized, for example, by a processor executing a predetermined program. Note that the input receiving unit 2 and the determination unit 3 may each be implemented in a separate device.

The input reception unit 2 plays and displays a first video of a person performing a specific action. This first video is used as a query image in the action recognition process. The input reception unit 2 accepts an instruction input that indicates a position related to the specific action on a frame of the first video that is played and displayed.

In FIG. 1, the first moving image includes frames 10a, 10b, ... from the beginning. In frame 10a, a pointing input is made with, for example, a mouse cursor to point to pointing position 11a, and in frame 10b, a pointing input is made to point to pointing position 11b. In this way, pointing input may be made continuously across multiple frames in the first moving image. Also, it is possible to change the pointing position while the frames are displayed one after the other, in which case the pointing position made by pointing input is recorded for each frame.

As described above, the instruction input is made so as to indicate a position related to a specific action of the recognition target. For example, the instruction input is made so as to indicate the vicinity of a body part of the person appearing in the first video that is highly related to the specific action. In FIG. 1, the specific action detected is the action of the person picking up a product (clothing). In such an action, the movement near the tip of the hand is distinctive in distinguishing it from other actions. For this reason, the operator makes an instruction input so as to indicate the vicinity of the tip of the hand of the person appearing on the screen while viewing the first video being played back and displayed.

The determination unit 3 compares the feature amounts between the group of feature points included in the second video, which is the detection target for the above-mentioned specific action, and the group of feature points included in the first video, and calculates the similarity between the first video and the second video based on the comparison result. The determination unit 3 determines whether the second video contains the specific action based on the calculated similarity. Note that in FIG. 1, the second video includes frames 20a, 20b, ... from the beginning.

For example, assume that body parts of a person are extracted as feature points. In the example of FIG. 1, assume that the positions of the palm and the ankle are extracted as feature points from each frame of the first video. For example, feature point 12a corresponding to the palm and feature point 12b corresponding to the ankle are extracted from frame 10a of the first video. In this case, the positions of the palm and the ankle are also extracted as feature points from each frame of the second video. For example, feature point 22a corresponding to the palm and feature point 22b corresponding to the ankle are extracted from frame 20a of the second video.

When calculating the similarity between frames 10a and 20a, the features are compared between feature points 12a and 22a, and between feature points 12b and 22b. The similarity between frames 10a and 20a is calculated based on the comparison results of the respective features.

Finally, the similarity between the first video and the second video is calculated for all combinations of frames between the first video and the second video, and the similarities are summed to determine the similarity between the first video and the second video. The determination unit 3, for example, compares the similarity thus determined with a predetermined threshold value to determine whether or not a specific action is included in the second video.

However, when calculating the similarity, the determination unit 3 assigns a greater weight to the comparison result of the feature amounts of feature points included in the first video that are closer to the pointing position by the pointing input. This makes the feature amounts of feature points closer to the pointing position more likely to contribute to the similarity calculation.

For example, in FIG. 1, because feature point 12a on frame 10a is close to pointing position 11a, a large weight is assigned to the comparison result of the feature amounts between feature point 12a and feature point 22a. On the other hand, because feature point 12b on frame 10a is far from pointing position 11a, a small weight is assigned to the comparison result of the feature amounts between feature point 12b and feature point 22b.

The structure of the human body is complex, and when the body is photographed, the way the body appears in the captured image changes in various ways. Even when a person performs a specific action that is the target of detection, there is diversity in the way the body appears as a whole. For example, when a person picks up a product, the movements of the body parts near the hands will be similar, but the movements of the feet may vary greatly depending on the person and the situation. For this reason, when trying to detect the action of picking up a product from an image, information on feature points near the hands is highly useful for detecting the action, but information on feature points near the feet is less useful for detecting the action.

By the above-described processing of the determination unit 3, the feature amount of a feature point closer to the position indicated by the instruction input has a higher contribution to the similarity calculation. Therefore, by simply preparing at least one first video image as a query image, it becomes possible to detect a specific movement with high accuracy even when there is diversity in the body movement when the specific movement is performed. In other words, according to the first embodiment, it becomes possible to perform highly accurate movement recognition using a small number of query images.

Note that inputting instructions to the first video can be achieved, for example, by a simple operation using a mouse. For example, by dragging the mouse, it is possible to continuously input instructions across multiple frames included in the first video.

Second Embodiment
Next, as a second embodiment, a system including an information processing device having the action recognition processing function of the action recognition system 1 in FIG. 1 will be described.

Figure 2 is a diagram showing an example of the configuration of an information processing system according to the second embodiment. The information processing system shown in Figure 2 includes an information processing device 100 and a camera 200 connected to the information processing device 100.

The information processing device 100 is realized as a computer such as a personal computer or a server device. Based on a video captured by the camera 200, the information processing device 100 detects that a person captured in the video has performed a specific action. The camera 200 is placed at a predetermined position where a person may be captured, and transmits data of the captured video to the information processing device 100.

As shown in FIG. 2, in this embodiment, as an example, the camera 200 is placed in a store 210 where a product 211 is sold, and captures the actions of a person 212 (customer) standing in front of the product 211. The information processing device 100 detects that the person 212 performs a specific action related to the product 211 based on the captured video image. For example, the information processing device 100 detects the action of the person 212 picking up the product 211 or the action of the person 212 checking the price of the product 211. Furthermore, the information processing device 100 can perform various analyses, such as customer behavior analysis, using the detection results of the actions of the person 212 over a predetermined period of time.

Fig. 3 is a diagram showing an example of the hardware configuration of an information processing device. The information processing device 100 is realized, for example, as a computer as shown in Fig. 3. The information processing device 100 shown in Fig. 3 has a processor 101, a RAM (Random Access Memory) 102, a HDD (Hard Disk Drive) 103, a graphic interface (I/F) 104, an input interface (I/F) 105, a reading device 106, a communication interface (I/F) 107, and a network interface (I/F) 108.

The processor 101 performs overall control of the entire information processing device 100. The processor 101 is, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), or a PLD (Programmable Logic Device). The processor 101 may also be a combination of two or more elements of a CPU, an MPU, a DSP, an ASIC, or a PLD.

RAM 102 is used as the main storage device of information processing device 100. RAM 102 temporarily stores at least a portion of the OS (Operating System) program and application programs to be executed by processor 101. RAM 102 also stores various data necessary for processing by processor 101.

The HDD 103 is used as an auxiliary storage device for the information processing device 100. The HDD 103 stores the OS program, application programs, and various data. Note that other types of non-volatile storage devices, such as a solid state drive (SSD), can also be used as the auxiliary storage device.

A display device 104a is connected to the graphic interface 104. The graphic interface 104 displays an image on the display device 104a in accordance with an instruction from the processor 101. Examples of the display device include a liquid crystal display and an organic EL (ElectroLuminescence) display.

The input interface 105 is connected to the input device 105a. The input interface 105 transmits a signal output from the input device 105a to the processor 101. Examples of the input device 105a include a keyboard and a pointing device. Examples of the pointing device include a mouse, a touch panel, a tablet, a touch pad, and a trackball.

A portable recording medium 106a is detachably attached to the reading device 106. The reading device 106 reads data recorded on the portable recording medium 106a and transmits it to the processor 101. Examples of the portable recording medium 106a include an optical disk and a semiconductor memory.

The communication interface 107 transmits and receives data to and from the camera 200. The network interface 108 transmits and receives data to and from other devices via the network 108a.

The processing functions of the information processing device 100 can be realized by the above hardware configuration.
The information processing device 100 detects a specific motion from a video in the following procedure. Prior to motion detection, the information processing device 100 acquires a motion (sample motion) from the camera 200 that shows a person actually performing the specific motion to be detected, and detects motion feature amounts indicating the characteristics of the motion from this sample motion. The information processing device 100 then acquires a motion that is the subject of motion detection (detection target motion) and detects motion feature amounts from this detection target motion. The information processing device 100 then compares the motion feature amounts between the sample motion and the detection target motion, thereby detecting a motion of a section of the detection target motion in which a person performing a specific motion is captured.

Fig. 4 is a block diagram showing an example of the configuration of processing functions of an information processing device. As shown in Fig. 4, the information processing device 100 includes a storage unit 110, a video input unit 111, a feature amount calculation unit 112, a time-space batch input unit 113, an attention calculation unit 114, a video search unit 115, a similarity calculation unit 116, and an analysis unit 117.

The storage unit 110 is realized by a storage area of a storage device provided in the information processing device 100, such as the RAM 102 and the HDD 103. The storage unit 110 stores sample video data 121 and detection target video data 131. The storage unit 110 also stores motion feature data 122 and attention level data 123 as data corresponding to the sample video data 121. The storage unit 110 also stores motion feature data 132 as data corresponding to the detection target video data 131.

The processing of the video input unit 111, feature amount calculation unit 112, time-space batch input unit 113, attention calculation unit 114, video search unit 115, similarity calculation unit 116 and analysis unit 117 is realized, for example, by the processor 101 executing a predetermined application program.

Video data captured by the camera 200 is input to the video input unit 111. The above-mentioned sample video and the detection target video are input as the video. The feature amount calculation unit 112 calculates motion feature amounts from the video input to the video input unit 111. The feature amount is calculated as data for each feature point extracted from each frame of the video.

When a sample video is input to the video input unit 111, the feature calculation unit 112 calculates motion features from the sample video and stores them in the storage unit 110 as motion feature data 122. At the same time, the feature calculation unit 112 stores the data of the sample video as sample video data 121 in the storage unit 110. When a detection target video is input to the video input unit 111, the feature calculation unit 112 calculates motion features from the detection target video and stores them in the storage unit 110 as motion feature data 132. At the same time, the feature calculation unit 112 stores the data of the detection target video as detection target video data 131 in the storage unit 110.

The time-space batch input unit 113 plays and displays the sample video input to the video input unit 111 on the display device. The time-space batch input unit 113 accepts instruction input from the operator while the sample video is being played and displayed. This instruction input is made to indicate the position of the body part of the person making the characteristic movement when the person in the sample video is performing the detection target movement. The instruction input continues while the movement is being performed, and ends when the movement ends. This allows the range of the section in the sample video where the specific movement is being performed (i.e., temporal information) and the position of the body part of the person making the characteristic movement (i.e., spatial information) to be specified together. Note that the instruction input is made, for example, by dragging the mouse.

Hereinafter, the instruction input from the operator received by the time-space batch input unit 113 will be referred to as the "time-space instruction input", and the information specified by the instruction input will be referred to as the "time-space instruction information". The time-space instruction information includes the start time and end time of the time-space instruction input in the sample video, and the instruction position in each frame from the start time to the end time.

The attention calculation unit 114 calculates the attention level for each feature point based on the designated position for each frame of the sample video in the section where the time-space instruction input is being made. The attention level is used to weight the motion feature amount when the similarity calculation unit 116 calculates the similarity between the sample video and the detection target video. The attention calculation unit 114 stores the calculated attention level in the memory unit 110 as attention level data 123.

The video search unit 115 controls a similarity search process that compares the detection target video data 131 with the sample video data 121 and searches the detection target video data for video similar to the video of the sample video data in the section in which the time-space instruction input is being performed. This similarity search process searches the detection target video data for video in which a person is performing a specified action.

Under the control of the video search unit 115, the similarity calculation unit 116 calculates the similarity between the detection target video and the sample video based on the motion feature amount. In this calculation, the attention level indicated by the attention level data 123 is used as a weight for the motion feature amount. As a result, the motion feature amount of a feature point closer to the position of the time-space instruction input is weighted more heavily to calculate the similarity. The similarity calculation result is notified to the video search unit 115, which extracts a video in which a specific motion is being performed from the detection target video based on the notified calculation result.

The analysis unit 117 performs various analyses based on the search results of the video search unit 115. For example, the analysis unit 117 evaluates the attractiveness and perceived price of a product based on the number of times the action of picking up the product or checking the price of the product is detected.

FIG. 5 is an example of a flowchart showing the overall processing procedure of the motion detection and analysis process performed by the information processing device.
First, in steps S11 and S12, pre-processing for motion detection is performed.

[Step S11] A sample video of a person performing a specific action to be detected is captured by the camera 200, and data of the captured sample video is input to the video input unit 111. The sample video provides a sample image showing the action of the person to be detected, and is used as a search query in a similarity search, which will be described later.

The feature amount calculation unit 112 extracts feature points from each frame of the sample video and calculates a motion feature amount for each feature point. The data of the sample video is stored in the storage unit 110 as sample video data 121. At the same time, the calculated motion feature amount is stored in the storage unit 110 in association with the sample video data 121 as motion feature amount data 122.

[Step S12] The time-space batch input unit 113 plays and displays the sample video based on the sample video data 121. The time-space batch input unit 113 accepts time-space instruction input from the operator while the sample video is being played and displayed. This allows the time range in which a specific action is performed in the sample video and the positions of the body parts of the person making the characteristic movement to be specified all at once.

The attention calculation unit 114 calculates the attention level for each feature point for each frame of the sample video image in a section where the time-space instruction input is being performed, based on the position of the instruction in the time-space instruction input. The calculated attention level is stored in the storage unit 110 as attention level data 123.

Subsequently, in steps S13 and S14, motion detection is performed from the moving image of the detection target.
[Step S13] A detection target moving image that is to be a detection target for a specific action is captured by the camera 200, and data of the captured detection target moving image is input to the video input unit 111. As the detection target moving image, for example, a moving image obtained by capturing images of the inside of a store for one day is used.

The feature calculation unit 112 extracts feature points from each frame of the detection target video and calculates a motion feature for each feature point. Data of the detection target video is stored in the storage unit 110 as detection target video data 131. At the same time, the calculated motion feature is stored in the storage unit 110 in association with the detection target video data 131 as motion feature data 132.

[Step S14] The similarity calculation unit 116 calculates a similarity between the detection target video and the video of the sample video in a section where a time-space instruction input is being performed, based on the motion feature amount. In this calculation, the attention level indicated by the attention level data 123 is used as a weight for the motion feature amount. The video search unit 115 performs a similarity search using the similarity calculation result to detect a video in which a specific motion is being performed from the detection target video.

[Step S15] The analysis unit 117 analyzes the results of the detection of specific actions by the video search unit 115. For example, the analysis unit 117 evaluates the attractiveness or perceived price of the product based on the number of times the action of picking up the product or the action of checking the price of the product is detected.

Each process shown in FIG. 5 will be described in detail below.
<Calculation of motion features from sample video>
6 is a diagram for explaining the motion feature amount. The motion feature amount is a feature amount that represents the motion of a person, and is calculated for each feature point. With the recent progress of machine learning, particularly deep learning, it has become possible to practically estimate the presence area (circumscribed rectangle) of a person appearing in an image and the positions of body parts such as joints from the image.

Therefore, in this embodiment, body parts of a person are used as feature points. For example, a number of predetermined body parts, such as both ends of the eyes, both shoulders, both wrists, the tips of all ten fingers, and the tips of both feet, are used as feature points. Furthermore, a predetermined feature point number is assigned to each feature point. The body parts of a person are detected using known skeletal detection technology.

If the feature point number is i and the number of detectable feature points (number of body parts) is n, then the motion feature amount P for a series of moving images is defined as a function of time t as shown in the following equation (1).
P = {p _i (t)} (where i = 1 to n) ... (1)
That is, the motion feature amount P is defined as a sequence of n functions p _i (t). Fig. 6 shows, as an example, motion feature amounts calculated from a sample video when n=2 (when two feature points are detected). In Fig. 6, the shooting times of the frames from the beginning of the sample video are t ₁ , t ₂ , t ₃ , .... Furthermore, the motion feature amount data 122 is created, for example, as a table in which feature point numbers correspond to times (frame identification information) as shown in Fig. 6.

Each motion feature includes a spatial attribute and a value attribute. That is, the above function p _i (t) includes a spatial attribute X(p _i (t)) and a value attribute A(p _i (t)). Of these, the spatial attribute indicates the two-dimensional coordinates of a feature point (body part) in a frame. Furthermore, if the distance z from the camera 200 to a person can also be detected, the spatial attribute may indicate three-dimensional coordinates. The value attribute will be described later.

When multiple people appear in an image, the motion feature is calculated for each person. For example, people detected from a video are given labels (e.g., numbers) in the order of appearance, and the motion feature P is calculated in association with the labels.

7 is an example of a flowchart showing a process for calculating motion feature amounts from a sample video image, which corresponds to the process in step S11 in FIG.
[Step S21] A sample video showing a person performing a specific action to be detected is input from the camera 200 to the video input unit 111 and passed to the feature calculation unit 112. The feature calculation unit 112 stores the data of the sample video in the storage unit 110 as sample video data 121.

[Step S22] The feature amount calculation unit 112 selects one frame from the beginning of the sample video.
[Step S23] The feature amount calculation unit 112 extracts feature points from the selected frames. As described above, the positions of body parts on the person are extracted as feature points. Each of the extracted feature points is assigned a feature point number that is uniformly determined in advance for each body part.

[Step S24] The feature calculation unit 112 calculates a motion feature for each extracted feature point. The motion feature includes a spatial attribute and a value attribute. The feature calculation unit 112 registers the calculated motion feature in the motion feature data 122 in the storage unit 110.

[Step S25] The feature calculation unit 112 determines whether processing has been completed for all frames in the sample video. If there are unprocessed frames, processing proceeds to step S22, where one unprocessed frame is selected from the top. On the other hand, if all frames have been processed, processing ends.

[First Example of Motion Feature Amount]
FIG. 8 is a diagram showing a first example of a motion feature amount. As described above, the spatial attribute included in the motion feature amount indicates the coordinates of a feature point. Similarly to the spatial attribute, the value attribute included in the motion feature amount may also indicate the coordinates of a feature point. In this case, if the coordinates of the i-th feature point in the frame at time t are (u _i (t), v _i (t)), the function p _i (t) of the motion feature amount for each feature point is expressed by the following formula (2-1), and the spatial attribute and value attribute are expressed by formulas (2-2) and (2-3), respectively. Note that FIG. 8 illustrates an example in which n=2 (two feature points are detected).
p _i (t)=(u _i (t), v _i (t)) (2-1)
X(p _i (t)) = (u _i (t), v _i (t)) ... (2-2)
A(p _i (t)) = (u _i (t), v _i (t)) ... (2-3)
[Second Example of Motion Feature Amount]
The motion feature amount may include, as a value attribute, information indicating a temporal change of the feature point. For example, the feature amount calculation unit 112 calculates the velocity vector (u _i '(t), v _i '(t)) of the feature point by referring to a predetermined number of frames before and after the frame for which the feature amount is to be calculated. In this case, the function p _i (t) of the motion feature amount for each feature point is expressed by the following formula (3-1), and the spatial attribute and value attribute are expressed by formulas (3-2) and (3-3), respectively.
p _i (t)=(u _i (t), v _i (t), u _i '(t), v _i '(t)) (3-1)
X(p _i (t)) = (u _i (t), v _i (t)) ... (3-2)
A(p _i (t)) = (u _i (t), v _i (t), u _i '(t), v _i '(t)) ... (3-3)
Note that the value attribute A(p _i (t)) may include only u _i '(t) and v _i '(t), and may not include u _i (t) and v _i (t).

[Third Example of Motion Feature Amount]
9 is a diagram showing a third example of the motion feature. Information using deep learning technology may be calculated as a value attribute of the motion feature. For example, image frames each including a predetermined number of types (e.g., 100 types) of motions are prepared as training data. Each training data is assigned a motion label that identifies the motion depicted.

As shown in the upper part of Figure 9, a neural network is used for learning, which includes an arrangement of several convolutional layers 141-1, ..., 141-n that perform convolution for each feature point individually, and a fully connected layer 142 that is common to all feature points. In this neural network, a fully connected layer 142 that outputs action labels is placed after the final convolutional layers 143-1, ..., 143-n for each feature point. Then, using this neural network and the above training data, learning is performed in advance so that the action label of the target frame is output when a predetermined number of frames (e.g., five frames) before and after the target frame in the video are input.

After such learning is completed, the fully connected layer 142 is removed as shown in the lower part of FIG. 9. In this state, the output of the final convolutional layers 143-1, ..., 143-n is expected to appropriately express the motion of a person for each corresponding feature point. In other words, the final convolutional layers 143-1, ..., 143-n output appropriate features for discriminating the motion of a person for the corresponding feature points. Therefore, the feature calculation unit 112 inputs a predetermined number of frames around the frame at time t to the post-learning arrays 141-1, ..., 141-n. As a result, value attributes A(p ₁ (t)), ..., A(p _n (t)) for each feature point are calculated from the arrays 141-1, ..., 141-n.

[Fourth Example of Motion Feature Amount]
By simultaneously photographing a person from different angles using multiple cameras, the three-dimensional coordinates of feature points (body parts) can be obtained using the principle of stereoscopic vision. In this case, the function p _i (t) of the motion feature amount for each feature point is expressed as three-dimensional information as shown in the following formula (4-1). Furthermore, when the method described in the above "First Example of Motion Feature Amount" is used, the spatial attribute and value attribute are expressed by formulas (4-2) and (4-3), respectively.
p _i (t) = (u _i (t), v _i (t), w _i (t)) ... (4-1)
X(p _i (t)) = (u _i (t), v _i (t), w _i (t)) ... (4-2)
A(p _i (t)) = (u _i (t), v _i (t), w _i (t)) ... (4-3)
The value attribute may be calculated using the method of the "Second Example of Motion Feature Amount" or the "Third Example of Motion Feature Amount" described above.

<Time-space instruction input and attention level calculation>
The time-space collective input unit 113 plays and displays the sample video and accepts time-space instruction input from the operator. This time-space instruction input operation is an input operation for detecting a specific movement from the sample video. The time-space instruction input is performed when a person in the sample video is performing a detection target movement, by pointing out the position of the body part of the person who makes a characteristic movement in the movement. The instruction input continues while the movement is being performed, and ends when the movement ends. This allows the time range (start time and end time) in which the specific movement is performed in the sample video and the position of the body part of the person who makes the characteristic movement to be specified all at once.

The time-space instruction can be input by a simple operation using a mouse. For example, as shown in FIG. 10, the time-space instruction can be input by dragging the mouse.
Fig. 10 is a diagram showing an example of a time-space instruction input. Fig. 10 illustrates an example of detecting an action of picking up a product. In such an action, it is estimated that the movement of the wrist and palm is characteristic. Therefore, when a person on the reproduced image is picking up a product, the operator performs a time-space instruction input so that the mouse cursor (pointer) 151 points to a position as close as possible to the wrist or palm of the person.

In the example of Fig. 10, the sample video includes 200 frames from frames _F1 to _F200 . In addition, the action of picking up a product is assumed to be performed in 150 frames from frames _F31 to _F180 . In this case, while viewing the sample video, the operator presses the click button while pointing the mouse cursor 151 to the vicinity of the person's wrist or palm in frame _F31 to start a drag operation. Thereafter, the operator continues to press the click button while pointing the mouse cursor 151 to the vicinity of the person's wrist or palm until the above action is completed, thereby continuing the drag operation. Then, the operator stops pressing the click button when frame _F180 is displayed, and ends the drag operation.

By such an operation, the time-space collective input unit 113 acquires the start time t _start and the end time t _end of the mouse drag operation. The start time t _start is information indicating the frame F ₃₁ where a specific operation is started, and the end time t _end is information indicating the final frame F ₁₈₀ where this operation is performed. Note that, as an example in FIG. 10, time t ₁ is acquired as the start time t _start , and time t ₁₅₀ is acquired as the end time t _end . Furthermore, the time-space collective input unit 113 acquires the position indicated by the mouse cursor 151 in each frame from the start time t _start to the end time t _end .

If the information obtained by the spatiotemporal pointing operation in this way is R, then R is expressed as a mapping from the time interval [t _start , t _end ] to a non-negative function defined on the image frame. If this information is expressed as R(x,y;t), then R(x,y;t) represents the attention level (degree of contribution to the action) for the position (x,y) on the image frame at time t, as will be described in detail later. Hereinafter, R(x,y;t) will be referred to as the "attention level."

The method of inputting a time-space instruction is not limited to the method using the mouse drag operation as described above. For example, a method in which a mouse click operation is performed once each at the start time t _start and the end time t _end may be used. For example, in the case of FIG. 10, the click button is pressed once in frame F ₃₁ while pointing the vicinity of the wrist or palm of the person with the mouse cursor 151. From that time, the mouse cursor 151 points to the vicinity of the wrist or palm of the person without pressing the click button. Then, at the end of the movement, the click button is pressed again. As a result, the position pointed to by the mouse cursor 151 in each frame from the first button press to the last button press is recorded.

Even when the button is pressed twice like this, the vicinity of a body part characteristic of the movement to be detected can be recorded by the simple operation of continuing to point to a position on the screen with the mouse cursor 151, just as in the case of using a mouse drag operation. Furthermore, together with this spatial information, the time information, i.e., the start and end times of the movement, can be recorded all at once with a simple operation.

Fig. 11 is a diagram for explaining the attention degree calculation process. In Fig. 11, the designated position (two-dimensional coordinates) of the mouse cursor 151 at time t is ( _x0 (t), _y0 (t)), and a time-space instruction input is made when each frame from time _t1 to time _t150 of the sample video is displayed. In this case, attention degrees R( _x , _y ;t1), R(x, _y ; _t2 ),...,R(x,y; _t150 ) are calculated for each frame at time t1, time t2,...,time _t150 , and are registered in the attention degree data 123.

[First Example of Attention Level]
As a first example, the attention level R(x, y; t) can be calculated by the following formula (5).
R (x, y; t) = exp (-((x-x ₀ (t)) ² + (y-y ₀ (t)) ² )/σ ² ) ... (5)
According to formula (5), the closer a feature point is to the position pointed to by the mouse cursor 151, the larger the attention value becomes. As will be described in detail later, the attention value is used as a weight when calculating the similarity between feature points in similarity search. The greater the attention value, that is, the closer a feature point is to the position pointed to by the time-space instruction input, the greater the weight is assigned to that feature point, and the similarity for that feature point greatly contributes to determining the presence or absence of a specific action.

In addition, in equation (5), σ is a positive constant for adjusting the spatial extent of the attention level. The value of σ can be used to provide an instruction to the information processing device 100 to distinguish whether the motion to be detected is a (characteristic) motion involving a wide range of the body (e.g., the whole body) or a motion involving a narrow range (e.g., the fingertips). For example, if the motion to be detected is a motion involving a narrow range of the body, a small value can be set for σ. This makes it possible to limit the range in which the motion feature value of the feature point contributes significantly to the similarity calculation, centered on the indicated position.

[Second Example of Attention Level]
When inputting a time-space instruction, the operator may use the numeric keypad of a keyboard or the like in parallel with the mouse dragging operation described above to vary the spatial extent of the attention level as the frames progress. In this case, the attention level R(x, y; t) is calculated by the following formula (6).
R (x, y; t) = exp (-((x-x ₀ (t)) ² + (y-y ₀ (t)) ² )/σ ² (t)) ... (6)
In formula (6), ^σ2 (t) is a positive coefficient for adjusting the spatial extent of the attention level, which is set over time. By making σ variable over time as in formula (6), it is possible to adaptively set whether a wide range of body movement or a narrow range of body movement is characteristic within the flow of body movement. As a result, it is possible to improve the accuracy of similarity search.

As another method of using the coefficient σ in calculating the attention level, for example, a method of using separate coefficients σ for the x coordinate and the y coordinate can also be applied.
12 is a flowchart showing an example of a calculation process of an attention level according to a time-space instruction input. The process in FIG. 12 corresponds to the process in step S12 in FIG.

[Step S31] The time-space batch input unit 113 starts playing and displaying the sample video based on the sample video data 121.
[Step S32] The time-space collective input unit 113 determines whether a time-space instruction input has been started for the currently displayed frame. If a time-space instruction input has been started, the process proceeds to step S33. On the other hand, if a time-space instruction input has not been started, the process of step S32 is executed for the next frame to be displayed.

[Step S33] The time-space batch input unit 113 records in the storage unit 110 the start time t _start of the time-space instruction input.
[Step S34] The time-space batch input block 113 records in the storage block 110 the position (coordinates) designated by the mouse cursor in the current frame.

[Step S35] The time-space batch input unit 113 determines whether the time-space instruction input has ended for the next displayed frame. If the time-space instruction input has ended, the process proceeds to step S36. On the other hand, if the time-space instruction input has not ended, the process of step S34 is executed for the current frame (the frame for which the end is to be determined).

[Step S36] The time-space batch input unit 113 records the time of the previous frame in the storage unit 110 as the end time t _end of the time-space instruction input.
[Step S37] The attention level calculation unit 114 selects the leading frame from among unprocessed frames included in the sample video from the start time t _start to the end time t _end (hereinafter referred to as the "input section video").

[Step S38] The attention calculation unit 114 calculates the attention level R for each pixel of the selected frame based on the mouse cursor position recorded in step S34. The calculated attention level R is recorded in the attention level data 123.

[Step S39] The attention calculation unit 114 determines whether processing has been completed for all frames in the input section video image. If there are unprocessed frames, processing proceeds to step S37, where the next unprocessed frame is selected. On the other hand, if all frames have been processed, processing ends.

The calculation of the motion feature amount in step S11 of FIG. 5 and the process of step S12 may be executed in parallel. Furthermore, the process of step S12 may be executed before the calculation of the motion feature amount in step S11. In this case, the motion feature amount needs to be calculated only for each frame of the sample video that is included in the input section video determined by the spatiotemporal instruction input in step S12.

The attention level R may be calculated and recorded for each feature point included in a frame, rather than for all pixels in the frame. The attention level R may also be calculated during a similarity search, which will be described later.

<Calculation of motion features from detection target video>
Fig. 13 is an example of a flowchart showing a calculation process of motion features from a detection target video image. This Fig. 13 corresponds to the process of step S13 in Fig. 5. As shown in Fig. 13, the calculation of motion features from a detection target video image is performed in the same process as the calculation from a sample video image.

[Step S41] A detection target video is input from the camera 200 to the video input unit 111 and passed to the feature calculation unit 112. The detection target video is, for example, a video shot over the course of a day. The feature calculation unit 112 stores the data of the detection target video in the storage unit 110 as detection target video data 131.

[Step S42] The feature calculation unit 112 selects one frame from the beginning of the detection target moving image.
[Step S43] The feature amount calculation unit 112 extracts feature points (body parts) from the selected frames. Each extracted feature point is assigned a feature point number that is uniformly determined in advance for each body part. The same feature point number is assigned to the same body part between the sample video and the detection target video.

[Step S44] The feature calculation unit 112 calculates a motion feature for each extracted feature point. The motion feature includes a spatial attribute and a value attribute. The feature calculation unit 112 registers the calculated motion feature in the motion feature data 132 in the storage unit 110.

[Step S45] The feature calculation unit 112 determines whether processing has been completed for all frames in the detection target video. If there are unprocessed frames, processing proceeds to step S42, where one unprocessed frame is selected from the top. On the other hand, if all frames have been processed, processing ends.

<Similarities-based Action Detection>
The similarity calculation unit 116 calculates a similarity between the detection target video and a video of a section of the sample video where a time-space instruction input is performed (input section video) based on the motion feature amount. The video search unit 115 detects an input section video in which a specific motion is performed from the detection target video by a similarity search using the calculation result of the similarity.

14 is a diagram for explaining the similarity search process between the sample video and the detection target video. As described above, the input section video 161 is a video of a section of the sample video where a time-space instruction input is performed (video from a frame at a start time t _start to a frame at an end time t _end ). In the similarity search process, a partial video having the same length as the input section video 161 is extracted from the detection target video 162, and the extracted partial video is compared with the input section video 161 to determine whether the partial video is similar to the input section video 161.

The partial video images are extracted in sequence from the beginning of the detection target video image 162, sliding one frame at a time towards the end. Figure 14 illustrates an example in which m partial video images are extracted from the detection target video image 162. In this case, partial video images 163-1, 163-2, ..., 163-m are extracted from the beginning of the detection target video image 162. Then, a similarity is calculated between each of the partial video images 163-1, 163-2, ..., 163-m and the input section video image 161 based on the motion feature amount. If the similarity is equal to or greater than a predetermined threshold, it is determined that the partial video image contains a person performing a specific motion.

In addition, dissimilarity (dissimilarity) may be calculated instead of similarity between video images. If the similarity is Sim and the dissimilarity is D, then the dissimilarity D can be converted to similarity Sim by inverting the sign of the dissimilarity D, for example, by setting Sim = -D.

Figure 15 is a diagram for explaining the calculation of similarity between the input section video and the partial video. The similarity calculation unit 116 calculates the similarity between frames (inter-frame similarity) between the input section video extracted from the sample video and the partial video extracted from the detection target video, starting from the top. The similarity calculation unit 116 then calculates the similarity between the input section video and the partial video by adding up all the calculated inter-frame similarities.

In the example of FIG. 15, the input section video includes frames 171-1, 171-2, 171-3, ... from the beginning, and the partial video includes frames 172-1, 172-2, 172-3, ... from the beginning. In this case, the inter-frame similarity is calculated between frames 171-1 and 172-1, between frames 171-2 and 172-2, and between frames 171-3 and 172-3. In this way, the inter-frame similarity is calculated for all frame combinations.

Between frames of the input section video and partial video, motion feature amounts of the same feature points (i.e., the same body parts) are compared. The smaller the difference between the motion feature amounts, the higher the similarity is determined to be. Between frames, the similarities calculated for each feature point in this way are added together to calculate the inter-frame similarity. However, when the similarities calculated for each feature point are added together, a weight is assigned to the similarity depending on the position of the feature point, based on the attention level based on the spatiotemporal instruction input described above.

For example, in FIG. 15, it is assumed that in frame 171-1 of the input section video, designated position 173-1 is designated by a time-space input instruction. In this case, the closer to designated position 173-1 among the feature points on frame 171-1, the greater the weighting is given to the feature points when calculating the similarity between frames 171-1 and 172-1. Also, in FIG. 15, it is assumed that in frame 171-2 of the input section video, designated position 173-2 is designated by a time-space input instruction. In this case, the closer to designated position 173-2 among the feature points on frame 172-1, the greater the weighting is given to the feature points when calculating the similarity between frames 171-2 and 172-2.

As described above, the attention level R is set to a larger value for pixels closer to the position indicated by the spatiotemporal input instruction. When comparing the motion feature amounts of a certain feature point between frames 171-1 and 172-1, the attention level R corresponding to the position of the feature point on frame 171-1 is used as a weight. Therefore, the closer this feature point is to the position indicated by the spatiotemporal input instruction, the larger the weight is multiplied by which the similarity based on the motion feature point is multiplied.

By performing such calculations, the more characteristic the feature points of a body part are in the movement to be detected, the greater their contribution to the similarity calculation. Therefore, the similarity between videos is an index that is sensitive to the similarities and differences in the movements of the body parts that should be noted in the movement to be detected. As a result, movement detection is performed with an emphasis on the movement of that body part, improving the accuracy of movement detection.

The structure of the human body is complex, and when the body is photographed, the way the body appears in the captured image changes in various ways. Even when a person performs a specific action that is the target of detection, there is diversity in the way the body appears as a whole. For example, when a person picks up a product, the movements of the body parts near the hands will be similar, but the movements of the feet may vary greatly depending on the person and the situation. For this reason, when trying to detect the action of picking up a product from an image, information on feature points near the hands is highly useful for detecting the action, but information on feature points near the feet is not very useful for detecting the action.

As described above, in this embodiment, the closer the motion feature value of a feature point is to the position indicated by the time-space instruction input, the higher its contribution to the similarity calculation for motion detection. For example, as shown by the dotted arrow in FIG. 15, among the feature points in frame 171-1, the feature point corresponding to the wrist, for example, is close to indicated position 173-1, but the feature point corresponding to the ankle is far from indicated position 173-1. Therefore, a greater weight (attention level) is assigned to the motion feature value of the feature point corresponding to the wrist than to the motion feature value of the feature point corresponding to the ankle. As a result, during similarity calculation, the contribution of the motion feature value of the feature point corresponding to the wrist is higher than that of the feature point corresponding to the ankle. As a result, it becomes possible to detect a specific motion with high accuracy even when there is a variety in body movements.

One possible method for detecting movements from images is to use images, which are given the position and type of the movement to be detected, as training data for learning, and then use the learning results to detect movements from input images. With this method, many different postures of people are given in the training data, so that specific movements can be detected with high accuracy even when there is a variety in body movements. However, there is a problem in that a large amount of training data must be prepared in advance to achieve high accuracy. In particular, the position and type of the movement to be detected must be specified manually for many images (training data), which means that creating training data is an enormous effort.

Another possible method of detecting actions is to use a person's posture information as a search query to search for images that include a specific action from among a large number of images. However, as described above, there are many different postures that a person can take for a specific action, and these postures change over time. Therefore, it is not easy to specify posture information that represents the action to be detected, and it is difficult to prevent a deterioration in detection accuracy due to the diversity of postures.

In response to such problems, the present embodiment reduces the amount of work required to create the information that must be prepared in advance. That is, a person performing a specific action is photographed, and while playing back and displaying the captured video, a simple operation using the mouse, such as dragging the mouse (spatiotemporal instruction input), is used to specify the area in which the action to be focused is occurring, and the information processing device 100 can calculate the attention level for similarity calculation.

Here, in time-space instruction input, the section in the captured video where a specific movement is taking place can be easily identified by specifying the start and end times using the mouse buttons. This eliminates the need to analyze unnecessary frames in the video, improving processing efficiency. At the same time, the area in which the movement to be focused on is taking place can be easily specified by specifying the mouse cursor, so the operator does not need to have high skills in image analysis, etc. In other words, by simply performing such simple preliminary operations, specific movements can be detected with high accuracy even when there is diversity in body movements. Therefore, highly accurate motion detection can be performed efficiently with little effort.

[First Example of Similarity Calculation]
Here, the motion feature amount of the i-th feature point included in the input section video of the sample video is defined as P = { _pi (t)}. Also, the motion feature amount of the i-th feature point included in the partial video of the detection target video is defined as Q = { _qi (t)}. The dissimilarity D(P,Q;R) is calculated with respect to the attention level R(x,y;t) according to the spatiotemporal instruction input by the following formula (7).
D (P, Q; R) = Σ _t Σ _i R (X (p _i (t)); t) | A (p _i (t)) - A (q _i (t)) | ² ... (7)
In equation (7), R(X(p _i (t)); t) indicates the attention level corresponding to the position of the i-th feature point in the frame at time t. Σ indicates taking the sum of the subscripted variable. The range of the variable time t is the time interval [t _start , t _end ] specified by the spatiotemporal command input. With equation (7), the closer a feature point is to the position specified by the spatiotemporal command input in the input section video, the greater the attention level is multiplied as a weighting for the value attribute A of the motion feature amount.

[Second Example of Similarity Calculation]
In the above formula (7), the scale (magnitude) of the value attribute may affect the accuracy of the similarity determination. In order to correctly determine similar actions as similar even if the scale of the value attribute differs depending on the body size, the position on the image, etc., it is possible to use, for example, cosine similarity as shown in the following formula (8-1).
Sim (P, Q; R) = Cos (P _R , Q _R ) = Cos (P _R · Q _R / | P _R | | Q _R |) ... (8-1)
P _R = {R(X(p _i (t)); t) A(p _i (t))} ...(8-2)
Q _R = {R(X(p _i (t)); t) A(q _i (t))} ...(8-3)
Here, i takes values from 1 to n, and t takes values from the start time t _start to the end time t _end . Also, {...} represents a vector whose elements (values on individual coordinate axes) are the values obtained when the variables i and t are changed within this range. "P _R ·Q _R " indicates the inner product of two vectors, and "|P _R ||Q _R |" indicates the product of the square norm of two vectors (the square root of the sum of the squares of the elements).

[Third Example of Similarity Calculation]
In the first and second examples above, the value attribute of the motion feature is used as is. Here, if the value attribute has position information (if the value differs depending on the body position), the following considerations must be taken into account.

(1) For actions that are strongly influenced by the position on the image (such as the action of removing clothes that are in a fixed position), position information (fixed offset) is provided in the value attribute.
(2) For movements that depend on the relative positional relationship with the target object on the image (for example, when the location of clothes varies from day to day), the position information of the value attribute is replaced with the displacement from the target object (object offset).

(3) For actions that are determined independently of the position on the image (such as falling to the floor), the position information of the value attribute is replaced with the displacement from the center of the body (body offset).
Similarity or dissimilarity taking into account object offset or body offset can be calculated, for example, by replacing A(p _i (t)) in the first and second examples with A(p _i (t))-obj(t) or A(p _i (t))-A(p _μ (t)). The same is true for the motion feature quantity Q. Here, obj(t) indicates a vector of the same length as the value attribute, and the position of the target object is set for elements having position information, and 0 is set for other elements. Furthermore, A(p _μ (t)) indicates the average value of the value attribute at all feature points.

16 is a flowchart showing an example of a motion detection process based on similarity, which corresponds to the process in step S14 in FIG.
[Step S51] The video search unit 115 extracts, from the beginning of the detection target video, a partial video having the same length as the input section video (the section of the sample video from the start time t _start to the end time t _end ).

[Step S52] The similarity calculation unit 116 selects frames from the beginning of each of the input section video image and the partial video image. This identifies the video images to be compared in steps S53 to S55.

[Step S53] The similarity calculation unit 116 selects feature points from the selected frames of the input section video.
[Step S54] The similarity calculation unit 116 acquires a motion feature amount corresponding to the feature point selected from the selected frame of the input section video in step S53, a motion feature amount corresponding to the feature point with the same feature point number present in the selected frame of the partial video, and an attention level corresponding to the coordinates of the former feature point. The similarity calculation unit 116 calculates an inter-frame similarity for the selected feature point based on the acquired motion feature amounts and attention levels.

[Step S55] The similarity calculation unit 116 determines whether processing of all feature points included in the selected frame has been completed. If there are unprocessed feature points, processing proceeds to step S53, where one unprocessed feature point is selected. On the other hand, if all feature points have been processed, processing proceeds to step S56.

[Step S56] The similarity calculation unit 116 determines whether processing of all frames included in the input section video image (or partial video image) has been completed. If unprocessed frames exist, processing proceeds to step S52, where the first frame of the unprocessed frames is selected. On the other hand, if all frames have been processed, processing proceeds to step S57.

[Step S57] The similarity calculation unit 116 adds up all the inter-frame similarities calculated in step S54, and outputs the similarity between the input section video sequence and the partial video sequence.
[Step S58] The video search unit 115 compares the output similarity with a predetermined determination threshold to determine whether the partial video image contains a specific action. If the similarity is equal to or greater than the determination threshold, it is determined that the partial video image contains a specific action.

[Step S59] The video search unit 115 determines whether processing has been completed for all partial video images that can be extracted from the detection target image. If there are unprocessed partial video images, processing proceeds to step S51, where the first frame of the partial video image is advanced by one and the next partial video image is extracted from the detection target image. On the other hand, if all partial video images have been processed, processing ends.

As described above, in the process of FIG. 16, the difference (dissimilarity) between frames may be calculated instead of the similarity between frames. In this case, in step S58, the added difference between frames is converted into a similarity, and the converted similarity is compared with a threshold value.

<Analysis processing>
As an example, it is assumed that the action of picking up a product and the action of checking the price of the product are detected from detection target images taken in one day in steps S13 and S14 of Fig. 5. It is also assumed that the detection of the action of picking up a product is performed for each product.

In this case, the analysis unit 117 evaluates the attractiveness of the product by, for example, comparing the number of times the product is picked up in a day with the average number of times other products are picked up in a day. The analysis unit 117 also evaluates the affordability of the product by comparing the number of times the price of the product is checked in a day with the number of times the product is sold in the same day.

Figure 17 is a diagram showing an example of an analysis table. The analysis table 181 shown in Figure 17 is table information in which data for analyzing a product is temporarily registered. In this analysis table 181, the number of detections, the reference number of times, the ratio, and the analysis results are registered in association with each of the actions of "picking up the product" and "checking the price."

Regarding the action of picking up a product, the number of times the product was picked up in a day is registered in the detection count item. The reference count indicates the average number of times the product was picked up in a day, including other products. The ratio indicates the ratio of the detection count to the reference count. For example, if the ratio is 1.4 or more, the product is evaluated as having high appeal, if the ratio is 0.8 or more and less than 1.4, the product is evaluated as having medium appeal, and if the ratio is less than 0.8, the product is evaluated as having low appeal, and the evaluation results are registered in the analysis results item.

Furthermore, with regard to the operation of checking prices, the number of times the price of the relevant product was checked during the day is registered in the detection results field. The number of times the product was sold on the same day is registered in the reference count field. The ratio indicates the ratio of the detection count to the reference count. For example, if the ratio is 4 or more, the relevant product is evaluated as overpriced, if the ratio is 2 or more but less than 4, the price of the relevant product is evaluated as appropriate, and if the ratio is less than 2, the relevant product is evaluated as underpriced, and the evaluation results are registered in the analysis results field.

Third embodiment
Next, a third embodiment will be described in which a part of the processing of the information processing device 100 according to the second embodiment is modified.

In the second embodiment described above, the search results may over-match the movements of the person captured in the sample video used as the query image, which may cause erroneous detection of the action and reduce the detection accuracy. Therefore, in the third embodiment, the information processing device plays and displays each of the partial video images found by the similarity search, and has the operator confirm whether the specific action is actually captured in each partial video image. The information processing device then uses the confirmation results and each partial video image to perform learning to distinguish whether the video image contains the specific action or not. Finally, the information processing device uses the learning results to re-determine whether each partial video image contains the specific action. This improves the detection accuracy of the action.

Figure 18 shows an example of the configuration of processing functions provided in an information processing device according to the third embodiment. In Figure 18, the same components as those in Figure 4 are denoted with the same reference numerals, and their description will be omitted. The information processing device 100a shown in Figure 18 includes a correction instruction input unit 118 and a discrimination learning unit 119 in addition to the memory unit 110, video input unit 111, feature amount calculation unit 112, time-space batch input unit 113, attention calculation unit 114, video search unit 115, similarity calculation unit 116, and analysis unit 117 shown in Figure 4.

The correction instruction input unit 118 plays and displays each of the partial video images that have been determined to contain a specific action by the video search unit 115 through a similarity search based on similarity, and accepts input indicating whether each partial video image actually contains a specific action. The discrimination learning unit 119 uses each of the partial video images and the content of the input accepted by the correction instruction input unit 118 as training data to perform learning to discriminate whether a video image contains a specific action.

Figure 19 is an example of a flowchart showing the overall processing procedure of the motion detection and analysis process in the third embodiment. In Figure 19, the same processes as in Figure 5 are indicated by the same step numbers. In the third embodiment, the processes of steps S61 and S62 are executed between steps S14 and S15 in Figure 5.

[Step S61] The correction instruction input unit 118 plays and displays each of the partial video images in which the specific action was detected in step S14, and receives input from the operator indicating whether the specific action is actually included in each partial video image. The discrimination learning unit 119 uses these partial video images and the content of the input received by the correction instruction input unit 118 (i.e., information indicating whether the partial video image contains the specific action) as training data to perform learning to discriminate whether the video image contains the specific action.

[Step S62] Using the results of the learning performed in step S61, the video search unit 115 re-determines whether the specific action is included in each partial video image in which the specific action was detected in step S14.

Thereafter, in step S15, various analyses are carried out based on the results of the determination in step S62.
20 is a flowchart showing an example of a procedure for learning and re-determination. Note that steps S71 to S74 correspond to step S61 in FIG. 19, and steps S75 to S77 correspond to step S62 in FIG.

[Step S71] The correction instruction input unit 118 selects one of the partial video images in which a specific action was detected in step S14 of FIG. 19, and plays back and displays the selected partial video image.
[Step S72] The operator checks the reproduced and displayed partial video image, and performs an operation to input whether or not a specific action is captured in the partial video image. The correction instruction input unit 118 accepts the input information.

For example, if a specific action is shown in the displayed partial video image, the operator performs an input operation to indicate this. For example, the operator performs an operation to check a check section provided on the display screen. This notifies the correction instruction input unit 118 that the specific action is shown, and processing proceeds to the next step S73. On the other hand, if the specific action is not shown in the partial video image, the operator performs an operation to play and display the next partial video image. This notifies the correction instruction input unit 118 that the specific action is not shown, and processing proceeds to the next step S73.

[Step S73] The correction instruction input unit 118 determines whether processing has been completed for all partial video images in which a specific action was detected in step S14. If there are unprocessed partial video images, processing proceeds to step S71, where one of the unprocessed partial video images is selected. On the other hand, if all applicable partial video images have been processed, processing proceeds to step S74.

[Step S74] The discrimination learning unit 119 uses the partial video images in which the specific action was detected in step S14 and the input result in step S72 as training data to perform learning to discriminate whether or not the specific action is captured in the video image. In this learning, of these partial video images, the motion feature sets of each partial video image in which it has been input that the specific action is captured are used as positive examples, and the motion feature sets of each of the other partial video images are used as negative examples.

For example, a given function y = f(x; θ) from feature space to real space [0, 1] is prepared. The discrimination learning unit 119 adjusts the unknown parameter θ so that it outputs y = 1 for a group of motion features that belong to positive examples, and outputs y = 0 for a group of motion features that belong to negative examples. In this way, when a value attribute of a motion feature is input, it is learned to output a discrimination result of whether it is a positive example or a negative example.

The value attribute input during learning is represented as {A(q _i (t))}, for example. Here, i takes the values 1 to n, and t takes the value from the start time t _start to the end time t _end . Also, {...} represents a vector whose elements (values on individual coordinate axes) are the values when the variable i.t is changed within this range.

The learning can be performed using a machine learning technique such as a support vector machine (SVM). For example, x·θ is used as the function f(x;θ), where "·" indicates the inner product of vectors.

It is also desirable to assign weights to the value attributes according to their degree of attention. In this case, the input for learning is represented as {R(X( _pi (t));t)A( _qi (t))}.
[Step S75] The video search unit 115 selects one of the partial video images in which the specific action was detected in step S14.

[Step S76] The video search unit 115 inputs the value attribute of the motion feature for the selected partial video into the function y = f(x; θ) that is the result of learning, to determine whether the partial video contains a specific motion. At this time, instead of inputting the value attribute as is, a value attribute weighted according to the attention level may be input. Furthermore, the value obtained as a result of learning is used as θ. Furthermore, whether the partial video contains a specific motion is determined by whether the output value of y is equal to or greater than a predetermined threshold.

[Step S77] The video search unit 115 determines whether processing has been completed for all partial video images in which a specific action was detected in step S14. If there are unprocessed partial video images, processing proceeds to step S75, where one of the unprocessed partial video images is selected. On the other hand, if all applicable partial video images have been processed, processing ends. Note that the processing of steps S75 to S77 may be modified so that the processing of step S76 is performed on all partial video images, including images in which a specific action was not detected.

As described above, learning is performed using the results of the judgment of the presence or absence of a specific action input by the operator, and the action is re-judged using the learning results, thereby reducing the probability of erroneous detection of the action and improving detection accuracy. The operator can also visually check the accuracy of the judgment results using the learning results, and if the accuracy is insufficient, can execute learning again.

Fourth embodiment
In the second and third embodiments, the process for the sample video and the process for detecting a partial video including a specific action from the detection target video are executed in the same device. However, these processes may be executed in separate devices.

Fig. 21 is a diagram showing an example of the configuration of an information processing system according to the fourth embodiment. The information processing system shown in Fig. 21 includes a server device 100b and an information processing device 100c. Note that in Fig. 21, the same components as those in Fig. 4 are denoted by the same reference numerals, and their description will be omitted.

The server device 100b has the processing functions related to the sample video image, broadly classified as the processing functions of the information processing device 100 according to the second embodiment. The server device 100b has a storage unit 110a, a communication unit 191, a feature amount calculation unit 112a, a time-space batch input unit 113, and an attention calculation unit 114.

In this server device 100b, a sample video captured by the camera 200 is transmitted from the information processing device 100c and received by the communication unit 192. The communication unit 191 passes the received sample video to the feature amount calculation unit 112a. The feature amount calculation unit 112a then calculates motion feature amounts for each feature point from the sample video, similar to the feature amount calculation unit 112 in FIG. 4. As a result, the feature amount calculation unit 112a stores the sample video data 121 and the motion feature amount data 122 in the storage unit 110a.

The time-space batch input unit 113 also plays and displays the sample video on the display device and accepts time-space instruction input. The attention level calculation unit 114 calculates the attention level for each feature point based on the indicated position for each frame of the sample video in the section where the time-space instruction input is being made. As a result, attention level data 123 is stored in the storage unit 110a.

When the above process is completed, the communication unit 191 transmits the action feature amount data 122 and the attention level data 123 stored in the storage unit 110a to the information processing device 100c.
On the other hand, the information processing device 100c has a processing function of detecting a partial moving image including a specific action from a detection target moving image, broadly classified as one of the processing functions of the information processing device 100 according to the second embodiment. The information processing device 100c has a storage unit 110, a video input unit 111, a communication unit 192, a feature amount calculation unit 112b, a video search unit 115, a similarity calculation unit 116, and an analysis unit 117.

In this information processing device 100c, when a sample video is captured by the camera 200, the sample video is input to the video input unit 111. The communication unit 192 then stores the data of the sample video in the memory unit 110 as sample video data 121, and transmits the sample video to the server device 100b. After that, when the action feature data 122 and attention level data 123 corresponding to the sample video are transmitted from the server device 100b, the communication unit 192 receives them and stores them in the memory unit 110.

On the other hand, when the detection target moving image is captured by the camera 200, the detection target moving image is passed from the video input unit 111 to the feature amount calculation unit 112b. The feature amount calculation unit 112b calculates the motion feature amount for each feature point from the detection target moving image, similar to the feature amount calculation unit 112 in FIG. 4. As a result, the feature amount calculation unit 112b stores the detection target moving image data 131 and the motion feature amount data 132 in the storage unit 110.

Thereafter, the video search section 115, the similarity calculation section 116 and the analysis section 117 execute the same processes as those of the information processing device 100 in FIG.
The information processing device 100c may further include the correction instruction input unit 118 and the discrimination learning unit 119 shown in FIG.

The processing functions of the devices shown in the above embodiments (for example, a device that realizes the functions of the action recognition system 1 and the information processing device 100) can be realized by a computer. In this case, a program describing the processing contents of the functions that each device should have is provided, and the above processing functions are realized on the computer by executing the program on a computer. The program describing the processing contents can be recorded on a computer-readable recording medium. Examples of computer-readable recording media include magnetic storage devices, optical disks, and semiconductor memories. Examples of magnetic storage devices include hard disk drives (HDDs) and magnetic tapes. Examples of optical disks include CDs (Compact Discs), DVDs (Digital Versatile Discs), and Blu-ray Discs (BD, registered trademark).

When distributing a program, for example, portable recording media such as DVDs and CDs on which the program is recorded are sold. The program can also be stored in a storage device of a server computer, and the program can be transferred from the server computer to other computers via a network.

A computer that executes a program stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its own storage device. The computer then reads the program from its own storage device and executes processing according to the program. Note that the computer can also read a program directly from a portable recording medium and execute processing according to that program. The computer can also execute processing according to the received program each time a program is transferred from a server computer connected via a network.

REFERENCE SIGNS LIST 1 Action recognition system 2 Input acceptance unit 3 Determination unit 10a, 10b, 20a, 20b Frames 11a, 11b Pointing positions 12a, 12b, 22a, 22b Feature points

Claims (9)

an input receiving unit that plays and displays a first moving image captured of a person performing a specific action, and that continuously receives , during the playing and display of the first moving image, an instruction input that indicates a position related to the specific action for each frame included in the first moving image ;
a determination unit that calculates a similarity between the first video and the second video based on a comparison result of feature amounts between a second feature point group including one or more feature points included in the second video and a first feature point group including two or more feature points included in the first video, and determines whether or not the specific action is included in the second video based on the calculated similarity, wherein, when calculating the similarity, the determination unit assigns a greater weight to the comparison result of feature amounts for feature points in the first feature point group that are closer to a position pointed to by the instruction input;
A motion recognition system having the above configuration. the input receiving unit plays and displays a third moving image including the first moving image, and extracts, from the third moving image, a section from a position where the instruction input is started to a position where the instruction input is ended during the playing and display, as the first moving image.
The action recognition system according to claim 1 . the input receiving unit records a position pointed to by the instruction input for each frame included in the first moving image;
the determination unit performs a comparison of feature amounts between the second video and the first video for each frame, and when calculating the similarity, determines the weight based on a designated position corresponding to a frame for which feature amounts are to be compared, among designated positions recorded for each frame included in the first video;
The action recognition system according to claim 1 or 2. the input reception unit receives an input for adjusting a setting value over time while receiving the instruction input;
when calculating the similarity, the determination unit changes the weight value according to a distance from the instruction position by the instruction input to each of two or more feature points included in the first feature point group for each frame according to the input setting value.
The action recognition system according to claim 3. The determination unit is
comparing the first video with a plurality of the second video to determine whether or not the specific action is included in each of the plurality of the second video;
playing and displaying a fourth video determined to include the specific action among the plurality of second videos, and receiving a determination input indicating whether or not the specific action is included for each of the fourth videos;
using the fourth video and the content of the judgment input as training data, learning is performed to discriminate whether the specific action is included in the video;
determining a video including the specific action from among the fourth video using a result of the learning;
The action recognition system according to claim 1 . each of the two or more feature points included in the first feature point group is a body part of a person detected in the first video image;
The action recognition system according to claim 1 . The instruction input is performed by a mouse drag operation.
The action recognition system according to claim 1 . The computer
playing and displaying a first moving image captured of a person performing a specific action , and continuously receiving, during the playing and display of the first moving image, an instruction input for indicating a position related to the specific action for each frame included in the first moving image ;
calculating a similarity between the first video and the second video based on a comparison result of feature amounts between a second feature point group including one or more feature points included in the second video and a first feature point group including two or more feature points included in the first video, and determining whether or not the second video includes the specific action based on the calculated similarity;
Execute the process,
In the calculation of the similarity, a feature point in the first feature point group that is closer to the pointing position by the pointing input is weighted more heavily in a comparison result of the feature amounts.
Action recognition method. On the computer,
playing and displaying a first moving image captured of a person performing a specific action , and continuously receiving, during the playing and display of the first moving image, an instruction input for indicating a position related to the specific action for each frame included in the first moving image ;
calculating a similarity between the first video and the second video based on a comparison result of feature amounts between a second feature point group including one or more feature points included in the second video and a first feature point group including two or more feature points included in the first video, and determining whether or not the second video includes the specific action based on the calculated similarity;
Execute the process,
In the calculation of the similarity, a feature point in the first feature point group that is closer to the pointing position by the pointing input is weighted more heavily in a comparison result of the feature amounts.
Motion recognition program.

Images