JP7619546B2 - Motion identification program, information processing device, and motion identification method - Google Patents

Description

The present invention relates to an action identification program, an information processing device, and an action identification method, and more specifically, to an action identification program, an information processing device, and an action identification method that use video data of an object captured to extract characteristics of the action of the object captured in the video data, and recognize the action from the extracted characteristics.

For example, in the field of nursing care, nursing care costs are calculated based on the tasks performed by the caregiver for the person receiving care. Specifically, such calculations of nursing care costs may be made, for example, based on qualitative human judgment.

However, for example, if the criteria for judgment differ for each caregiver, the calculation of nursing care costs as described above may not always be done accurately. For this reason, in recent years, various studies have been conducted to quantitatively evaluate the content of the work performed by caregivers (caregiver actions).

JP 2014-021944 A

Yuki Niitsuma, Syunpei Torii, Yuichi Yaguchi, Ryuichi Oka: “Time-segmentation and position-free recognition of air-drawn gestures and characters in videos”, Multimedia Tools and Applications, An International Journal, ISSN 1380-7501, Volume 75, Number 19, pp.11615--11639. https://mediadrive.jp/products/et/ "Spotting Method for Classification of Real World Data": Ryuichi Oka, The Computer Journal, Vol.41, No.8, pp.559-565 (1998).

One possible method for quantitatively expressing the above-mentioned caregiver's movements is to use an acceleration sensor that measures the caregiver's movements in three-dimensional space.

However, when performing measurements using acceleration sensors, the caregiver must wear multiple acceleration sensors at many points on their arms, legs, etc., which places a heavy burden on the caregiver. In addition, when multiple acceleration sensors are used, it is necessary to integrate the data measured by each acceleration sensor, making it difficult to easily identify the recognition of the caregiver's movements.

In response to this, a method for quantitatively evaluating the actions of a caregiver can be considered, for example, by capturing an image of the caregiver with an imaging device (camera) and recognizing the actions by using the captured video data. In this method, for example, a hidden Markov model (HMM), which is a method for recognizing time-series patterns, is used to perform category recognition based on vector features extracted from the video data.

However, because HMM is a method that is primarily used in the field of speech recognition, it is unable to accurately recognize the complex actions of caregivers.

As a method for recognizing the movements of an object by using video data, for example, continuous DP (see Patent Document 1 and Non-Patent Document 1), which is a method for recognizing a movement pattern generated by a single stroke, has already been proposed. However, in the case of an object that cannot be represented by a single stroke, such as a caregiver, in which multiple points move simultaneously in a complex manner, it is not possible to accurately recognize the movements.

In addition, there have been attempts to recognize the movements of caregivers using deep learning, such as R-CNN (Recurrent Convolutional Neural Net). However, R-CNN was developed with the aim of targeting time series vectors, and is therefore unable to accurately recognize movements.

The object of the present invention is to provide a motion identification program, an information processing device, and a motion identification method that enable the motions of a caregiver to be quantitatively expressed.

The motion identification program of the present invention for achieving the above object is characterized in that it causes a computer to execute the following processes: identify first image data and second image data captured after the first image data from video data; calculate a first optical flow indicating displacement for each of a plurality of pixels in the first image data and the second image data; sort the first optical flows for each of the plurality of pixels into a plurality of groups based on the orientations corresponding to the first optical flows for each of the plurality of pixels; identify motion characteristics of an object shown in the first image data and the second image data for each of the plurality of groups based on the lengths corresponding to the first optical flows for each of the plurality of pixels included in each group; and output information indicating the identified motion characteristics.

In addition, in one aspect, the motion identification program of the present invention for achieving the above object is characterized in that it identifies third image data captured after the second image data from the video data, calculates a second optical flow indicating a displacement for each of the plurality of pixels in the second image data and the third image data, and assigns the first optical flow for each of the plurality of pixels and the second optical flow for each of the plurality of pixels to the plurality of groups based on the orientations corresponding to the first optical flow for each of the plurality of pixels and the second optical flow for each of the plurality of pixels, and identifies the motion characteristics of the object for each of the plurality of groups based on the lengths corresponding to the first optical flow for each of the plurality of pixels and the second optical flow for each of the plurality of pixels included in each group.

In one aspect, the motion identification program of the present invention for achieving the above object is characterized in that it quantizes the orientations corresponding to the first optical flow for each of the plurality of pixels and the second optical flow for each of the plurality of pixels, and assigns the first optical flow for each of the plurality of pixels and the second optical flow for each of the plurality of pixels to the plurality of groups such that optical flows having the same quantized orientation are assigned to the same group.

In one aspect, the motion identification program of the present invention for achieving the above object is characterized in that, for each of the plurality of groups, a sum of lengths corresponding to the first optical flow for each of the plurality of pixels contained in each group and the second optical flow for each of the plurality of pixels is calculated, and the motion characteristics of the object are identified based on the sums corresponding to each of the plurality of groups.

In one aspect, the motion identification program of the present invention for achieving the above object is characterized in that, for each of the plurality of groups, a value is obtained by dividing the sum corresponding to each group by the elapsed time from the image capture time of the second image data to the image capture time of the third image data, and the value for each of the plurality of groups is identified as the motion characteristic of the object.

In one aspect, the motion identification program of the present invention for achieving the above object is characterized in that it calculates, for each of the plurality of groups, a first sum of lengths corresponding to the first optical flow for each of the plurality of pixels contained in each group and a second sum of lengths corresponding to the second optical flow for each of the plurality of pixels contained in each group, identifies, for each of the plurality of groups, the number of sums among the first sums and the second sums corresponding to each group that is equal to or greater than a predetermined threshold, and identifies the number for each of the plurality of groups as a motion characteristic of the object.

In one aspect, the motion identification program of the present invention for achieving the above object is characterized in that it calculates, for each of the plurality of groups, a first sum of lengths corresponding to the first optical flow for each of the plurality of pixels included in each group and a second sum of lengths corresponding to the second optical flow for each of the plurality of pixels included in each group, calculates, for each combination of a first group and a second group included in the plurality of groups, an absolute value of the difference between the first sum corresponding to the first group and the second sum corresponding to the second group, and identifies the absolute value for each combination of the first group and the second group as a motion characteristic of the object.

The motion identification program of the present invention for achieving the above object is characterized in that it causes a computer to execute the following process: identify first image data and second image data captured after the first image data from video data; calculate a first optical flow indicating displacement for each of a plurality of pixels in the first image data and the second image data; sort each of the first optical flows for each of the plurality of pixels into a plurality of groups based on the orientation corresponding to each of the first optical flows for each of the plurality of pixels; determine whether a first category for the motion of the object captured in the first image data and the second image data matches a second category for the motion of the object captured in the third image data and the fourth image data from a correlation value between a length corresponding to each of the first optical flows for each of the plurality of pixels for each of the plurality of groups and a length corresponding to each of the second optical flows indicating the displacement for each pixel between the third image data and the fourth image data; and output information indicating whether the first category matches the second category.

In one aspect, the motion identification program of the present invention for achieving the above object is characterized in that it calculates a sum of the correlation values for each of the multiple groups, determines whether the sum is equal to or greater than a predetermined threshold, and if it is determined that the sum is equal to or greater than the predetermined threshold, determines that the first category and the second category match.

In one aspect, the motion identification program of the present invention for achieving the above object is characterized in that it performs a predetermined blurring process on the first optical flows for each of the plurality of pixels, and distributes each of the first optical flows for each of the plurality of pixels into a plurality of groups based on the orientations corresponding to the first optical flows for each of the plurality of pixels for which the predetermined blurring process has been performed.

In addition, the information processing device of the present invention for achieving the above object is characterized by having an image identification unit that identifies first image data and second image data captured after the first image data from video data, a flow calculation unit that calculates a first optical flow indicating a displacement for each of a plurality of pixels in the first image data and the second image data, a flow allocation unit that allocates each of the first optical flows for each of the plurality of pixels into a plurality of groups based on an orientation corresponding to each of the first optical flows for each of the plurality of pixels, a motion identification unit that identifies motion characteristics of an object captured in the first image data and the second image data based on a length corresponding to each of the first optical flows for each of the plurality of pixels included in each group for each of the plurality of groups, and an information output unit that outputs information indicating the identified motion characteristics.

In addition, the information processing device of the present invention for achieving the above object includes an image identification unit that identifies first image data and second image data captured after the first image data from video data; a flow calculation unit that calculates a first optical flow indicating displacement for each of a plurality of pixels in the first image data and the second image data; a flow allocation unit that allocates each of the first optical flows for each of the plurality of pixels into a plurality of groups based on the orientation corresponding to each of the first optical flows for each of the plurality of pixels; a motion determination unit that determines whether or not a first category for the motion of the object captured in the first image data and the second image data matches a second category for the motion of the object captured in the third image data and the fourth image data based on a correlation value between a length corresponding to each of the first optical flows for each of the plurality of pixels for each of the plurality of groups and a length corresponding to each of the second optical flows indicating the displacement for each pixel between the third image data and the fourth image data; and an information output unit that outputs information indicating whether or not the first category matches the second category.

The motion identification method of the present invention for achieving the above object is characterized in that it causes a computer to execute the following processes: identify first image data and second image data captured after the first image data from video data; calculate a first optical flow indicating displacement for each of a plurality of pixels in the first image data and the second image data; sort the first optical flows for each of the plurality of pixels into a plurality of groups based on the orientations corresponding to the first optical flows for each of the plurality of pixels; identify motion characteristics of an object shown in the first image data and the second image data for each of the plurality of groups based on the lengths corresponding to the first optical flows for each of the plurality of pixels included in each group; and output information indicating the identified motion characteristics.

In addition, the motion identification method of the present invention for achieving the above object is characterized in that it causes a computer to execute the following process: identify first image data and second image data captured after the first image data from video data; calculate a first optical flow indicating displacement for each of a plurality of pixels in the first image data and the second image data; sort each of the first optical flows for each of the plurality of pixels into a plurality of groups based on the orientation corresponding to each of the first optical flows for each of the plurality of pixels; determine whether or not a first category for the motion of the object captured in the first image data and the second image data matches a second category for the motion of the object captured in the third image data and the fourth image data based on a correlation value between a length corresponding to each of the first optical flows for each of the plurality of pixels for each of the plurality of groups and a length corresponding to each of the second optical flows indicating the displacement for each pixel between the third image data and the fourth image data; and output information indicating whether or not the first category matches the second category.

The motion identification program, information processing device, and motion identification method of the present invention make it possible to quantitatively express the motions of a caregiver.

FIG. 1 is a diagram illustrating the configuration of an information processing device 1. FIG. 2 is a diagram illustrating the action identification process according to the first embodiment. FIG. 3 is a flowchart illustrating details of the action identification process according to the first embodiment. FIG. 4 is a flowchart illustrating details of the action identification process in the first embodiment. FIG. 5 is a flowchart illustrating details of the action identification process according to the first embodiment. FIG. 6 is a diagram illustrating the optical flow and the directional pattern. FIG. 7 is a diagram illustrating details of the action identification process in the first embodiment. FIG. 8 is a diagram illustrating details of the action identification process according to the first embodiment. FIG. 9 is a diagram illustrating details of the action identification process according to the first embodiment. FIG. 10 is a diagram illustrating the motion identification process according to the second embodiment. FIG. 11 is a flowchart illustrating details of the action identification process according to the second embodiment. FIG. 12 is a flowchart illustrating details of the action identification process according to the second embodiment.

Below, an embodiment of the present invention will be described with reference to the drawings. However, such an embodiment does not limit the technical scope of the present invention.

[Configuration of information processing device]
First, a description will be given of the hardware configuration of the information processing device 1. FIG.

The information processing device 1 has the hardware configuration of a general-purpose computer device, and for example, as shown in FIG. 1, has a processor, a CPU 101, a memory 102, a network interface 103, and a storage medium 104. Each part is connected to each other via a bus 105.

The storage medium 104 has, for example, a program storage area (not shown) that stores a program (not shown) for performing a process (hereinafter also referred to as a motion identification process) to extract the characteristics of the motion of a caregiver (hereinafter also referred to as a target object) shown in video data (image data). The storage medium 104 also has, for example, a storage area 110 that stores information used when performing the motion identification process. Note that the storage medium 104 may be, for example, a HDD (Hard Disk Drive).

The CPU 101 executes a program loaded from the storage medium 104 (storage area 110) to the memory 102 to perform the action-specific processing.

The network interface 103 also communicates with, for example, the operation terminal 5.

[Movement Identification Processing in the First Embodiment]
Next, the motion identification process according to the first embodiment will be described with reference to FIG.

As shown in FIG. 2, the CPU 101 of the information processing device 1 functions as an image identification unit 11, a flow calculation unit 12, a flow allocation unit 13, an action identification unit 14, and an information output unit 15 by working in cooperation with a program.

The image identification unit 11, for example, acquires video data stored in the memory area 110. The video data is, for example, video data captured in advance of care work by a caregiver (care work that requires quantitative evaluation). Then, the image identification unit 11, for example, identifies first image data included in the video data and second image data captured after the first image data.

The flow calculation unit 12 calculates, for example, a first optical flow indicating a displacement for each pixel in the first image data and the second image data identified by the image identification unit 11.

The flow allocation unit 13 allocates each of the first optical flows to a plurality of groups, for example, based on the direction in which the first optical flows calculated by the flow calculation unit 12 are facing.

Specifically, the flow allocation unit 13, for example, quantizes the orientation corresponding to each of the first optical flows calculated by the flow calculation unit 12. Then, the flow allocation unit 13 allocates each of the first optical flows to a plurality of groups such that optical flows having the same quantized orientation are allocated to the same group.

The movement identification unit 14, for example, identifies (extracts) the movement characteristics of the caregiver shown in the first image data and the second image data for each of the multiple groups assigned by the flow allocation unit 13, based on the respective lengths of the first optical flows contained in each group.

The information output unit 15 outputs information indicating the movement characteristics (movement characteristics of the caregiver) identified by the movement identification unit 14.

In other words, the information processing device 1 in this embodiment groups (quantizes) multiple optical flows so that optical flows with similar orientations are included in the same group. Then, the information processing device 1 determines the caregiver's action target for each optical flow included in each group.

This enables the information processing device 1 in this embodiment to extract motion features that maintain information about the direction of the caregiver's motion. Therefore, by using the motion features extracted from each optical flow, the information processing device 1 can quantitatively represent the motion of the caregiver.

[Details of the Action Identification Process in the First Embodiment]
Next, the details of the action identification process in the first embodiment will be described below. Figures 3 to 9 are diagrams for explaining the details of the action identification process in the first embodiment.

[Optical Flow]
First, the optical flow will be described. Fig. 6 is a diagram for explaining the optical flow and the directional pattern.

Optical flow is a flow that indicates the displacement (movement distance and movement direction) of each pixel between image data contained in video data, and can be expressed as two-dimensional vector data, for example, as shown on the left side of Figure 6.

Specifically, optical flow is used, for example, when recognizing the movement of an object contained in video data (image data).

[Character Recognition Using Directional Patterns]
Next, character recognition (OCR: Optical Character Reader) using directional patterns (Non-Patent Document 2) will be described.

In character recognition using directional patterns, the Sobel method or the like is applied to the image data (light and dark image data) of the character to be recognized to extract edges at each coordinate point contained in the image data (step 1). Here, the edge (two-dimensional vector) extracted from the image data is expressed, for example, as in the following formula (1). In the following formula (1), x is the X coordinate of each coordinate point contained in the image data, y is the Y coordinate of each coordinate point contained in the image data, r is the magnitude of the vector, and θ is the orientation of the vector (angle from the X axis).

Next, in character recognition using directional patterns, for example, a split range that includes θ corresponding to the edge extracted from the image data is identified from among the eight ranges (hereinafter also referred to as split ranges) shown in Figure 7, and θ is quantized to a value corresponding to the identified split range (step 2).

Specifically, if the θ corresponding to an edge extracted from image data is included in the division range corresponding to "θ = 1", the θ corresponding to the edge is converted to 1.

Note that, although the following description will be given assuming that there are eight divided ranges, a number of divided ranges other than eight (e.g., 16 divided ranges) may also be provided.

Next, in character recognition using directional patterns, a two-dimensional scalar field is formed for each divided range by placing r corresponding to the edge corresponding to each divided range on the xy plane (step 3). That is, for example, if there are eight divided ranges, eight two-dimensional scalar fields are formed.

After that, in character recognition using directional patterns, blurring is performed for each two-dimensional scalar field (step 4). In this case, for example, blurring is performed by a convolution operation using a Gaussian function, or blurring is performed by averaging multiple values corresponding to the coordinates surrounding each coordinate in the two-dimensional scalar field.

Here, while the edges of characters contained in image data can be expressed as in the above formula (1), the optical flow determined from video data (multiple image data) can be expressed, for example, as in the following formula (2). In the following formula (2), t indicates the time when each image data was captured.

Therefore, it can be determined that character recognition using directional patterns such as those described above can also be applied to recognizing the movements of an object using optical flow. Therefore, in the movement identification process in this embodiment, movement recognition is performed using directional patterns, as described below.

[Flowchart of the action identification process in the first embodiment]
3 to 5 are flow charts for explaining the details of the motion identification process in the first embodiment. Note that the following will explain a case where optical flows corresponding to two sections (time periods) are calculated from three image data included in video data.

The image identification unit 11, for example, acquires video data stored in the memory area 110. Then, as shown in FIG. 3, the image identification unit 11 identifies, from the video data, for example, first image data included in the video data, second image data captured after the first image data, and third image data captured after the second image data (S11).

Specifically, the image specification unit 11 acquires, for example, first image data captured at t=0, second image data captured at t=1, and third image data captured at t=2.

The first image data and the second image data may be consecutive image data in the video data stored in the memory area 110, or may be discontinuous image data. Similarly, the second image data and the third image data may be consecutive image data in the video data stored in the memory area 110, or may be discontinuous image data.

The image identification unit 11 may also sequentially identify image data from the video data and continuously perform the processes from S12 onward on each of the identified image data.

Then, the flow calculation unit 12 calculates a first optical flow indicating the displacement for each pixel in the first image data and the second image data identified in the processing of S11 (S12).

Specifically, the flow calculation unit 12 calculates a first optical flow indicating the displacement for each of a plurality of pixels in the first image data and the second image data, as shown on the right side of FIG. 6.

The flow calculation unit 12 also calculates a second optical flow indicating displacement for each pixel in the second image data and the third image data identified in the processing of S11 (S13).

In this case, the flow calculation unit 12 calculates a first optical flow corresponding to the section between the first image data and the second image data, and a second optical flow corresponding to the section between the second image data and the third image data.

Specifically, the flow calculation unit 12 expresses each optical flow by using, for example, the above formula (2).

Note that, in the following, in the processing of S12 and S13, a case where optical flows corresponding to two sections are calculated will be described, but optical flows corresponding to three or more sections may also be calculated.

Next, the flow allocation unit 13 quantizes the orientations corresponding to the first optical flow and the second optical flow calculated in the processes of S12 and S13 (S14).

Specifically, the flow allocation unit 13 identifies a division range, for example, from among the eight division ranges shown in FIG. 7, that contains θ corresponding to each optical flow, and converts θ to a value corresponding to the identified division range.

Then, the flow allocation unit 13 allocates each of the first optical flow and the second optical flow to multiple groups so that optical flows with the same quantized value in the process of S14 are allocated to the same group (S15).

Specifically, for each division range, the flow allocation unit 13 identifies optical flows whose quantized values are included in each division range. Then, the flow allocation unit 13 groups the optical flows so that optical flows corresponding to the same division range are included in the same group.

Then, as shown in Fig. 4, the motion identification unit 14 calculates, for each of the multiple groups, the sum of the lengths corresponding to the first optical flows included in each group (hereinafter, also referred to as the first sum) (S21). Specifically, the motion identification unit 14 calculates, for each of the multiple groups, the sum of the values corresponding to the directional patterns of the first optical flows included in each of the groups (the first sum).

The motion identification unit 14 also calculates, for each of the multiple groups, the sum of lengths corresponding to the second optical flows included in each group (hereinafter also referred to as the second sum) (S22). Specifically, the motion identification unit 14 calculates, for each of the multiple groups, the sum of values corresponding to the directional patterns of the second optical flows included in each of the groups (the second sum). The processes of S21 and S22 are described in detail below.

[Details of the processes in S21 and S22]
The motion identification unit 14 first calculates a directional pattern corresponding to each of the first optical flow and the second optical flow by using the following formula (3). In the following formula (3), F(x, y, θ':t) indicates a directional pattern, θ' indicates each of the divided ranges, and T indicates a time difference between the imaging times of each image data used to calculate the optical flow. Note that δ(θ'-θ) is a delta function that takes the value 1 when θ' and θ match, and takes the value 0 when θ' and θ do not match.

That is, in formula (3), the value corresponding to the division range that does not include the orientation of each optical flow is 0. On the other hand, in formula (3), the value corresponding to the division range that includes the orientation of each optical flow is a value other than 0.

Furthermore, the motion identification unit 14 calculates the first sum and the second sum corresponding to each of the multiple groups by using the following formula (4). In the following formula (4), K(θ', t) indicates the sum (first sum and second sum) of values corresponding to the directional patterns of the multiple optical flows included in each group.

If optical flows corresponding to three or more sections are calculated in the processes of S12 and S13, the motion identification unit 14 calculates the sums corresponding to the three or more sections in the processes of S21 and S22.

Next, the motion identification unit 14 calculates, for example, for each of the multiple groups, a total sum corresponding to the multiple optical flows calculated from the image data captured from the start time to the end time. The start time indicates the start time of the time period in which the motion characteristics of the caregiver are identified, and the end time indicates the start time of the time period in which the motion characteristics of the caregiver are identified. The time between the start time and the end time is, for example, about several seconds to several minutes. In this case, the motion identification unit 14 may divide the calculated total value by the time from the start time to the end time.

Specifically, for example, when the start time corresponds to the imaging time of the second image data and the end time corresponds to the imaging time of the third image data, the motion identification unit 14 calculates the sum of the first sum and the second sum corresponding to each group for each of the multiple groups (S23). Then, for example, the motion identification unit 14 divides the calculated sum by the elapsed time from the imaging time of the second image data to the imaging time of the third image data for each of the multiple groups.

More specifically, the motion identification unit 14 calculates the value by using, for example, the following formula (5): In the following formula (5), t _start indicates a start time, and t _end indicates an end time.

Then, the motion identification unit 14 identifies each of the total values for the multiple groups calculated in the process of S23 as a first motion characteristic of the target object (S24). That is, the first motion characteristic is, for example, an index indicating the average amount of motion by the caregiver.

Then, the motion identification unit 14, for example, for each of the multiple groups, identifies the number of sums that are equal to or greater than a predetermined threshold among the sums corresponding to the multiple optical flows calculated from the image data captured from the start time to the end time. Note that in this case, the motion identification unit 14 may divide the number of identified sums by the time from the start time to the end time.

Specifically, for example, when the start time corresponds to the imaging time of the second image data and the end time corresponds to the imaging time of the third image data, the motion identification unit 14 identifies, for each of the multiple groups, the number of first sums and second sums corresponding to each group that are equal to or greater than a predetermined threshold value, as shown in FIG. 5 (S31). Then, for example, the motion identification unit 14 divides, for each of the multiple groups, the number of identified sums by the elapsed time from the imaging time of the second image data to the imaging time of the third image data.

More specifically, the motion identification unit 14 calculates the number of sums equal to or greater than a predetermined threshold by, for example, using the following formula (6). In the following formula (6), h represents a threshold, and 1[X] represents a function that takes a value of 1 when X is equal to or greater than 0, and takes a value of 0 when X is less than 0.

Then, the motion identification unit 14 identifies each of the total numbers for each of the multiple groups identified in the process of S31 as a second motion characteristic of the target object (S32). That is, the second motion characteristic is, for example, an index indicating the number of times the caregiver performed a motion with a motion amount greater than a threshold value.

Then, for example, for each combination of two groups (hereinafter also referred to as the first group and the second group) included in the multiple groups, the motion identification unit 14 calculates the sum corresponding to the first group among the sums corresponding to the multiple optical flows calculated from the image data captured from the start time to the end time (hereinafter also referred to as the specific image data). Also, for example, for each combination of the first group and the second group, the motion identification unit 14 calculates the sum corresponding to the second group among the sums corresponding to the multiple optical flows calculated from other image data captured a predetermined time after the specific image data. Then, for example, for each combination of the first group and the second group, the motion identification unit 14 calculates the sum of the absolute value of the difference between the sum corresponding to the first group and the sum corresponding to the second group. Note that in this case, the motion identification unit 14 may divide the calculated sum by the time from the start time to the end time.

Specifically, for example, when the start time corresponds to the imaging time of the second image data and the end time corresponds to the imaging time of the third image data, the motion identification unit 14 calculates the absolute value (total of absolute values) of the difference between the first sum corresponding to the first group and the second sum corresponding to the second group for each combination of the first group and the second group included in the multiple groups (S33). Then, for example, the motion identification unit 14 divides the calculated absolute value (total of absolute values) for each combination of the first group and the second group by the elapsed time from the imaging time of the second image data to the imaging time of the third image data.

More specifically, the motion identification unit 14 calculates the sum of the absolute values of the differences between the sums corresponding to the first group and the sums corresponding to the second group, for example, by using the following formula (7). In the following formula (7), θ' indicates the division range corresponding to the first group, Δθ' indicates the division range corresponding to the second group, and τ indicates a predetermined time.

Then, the motion identification unit 14 identifies each of the absolute values for each combination of two groups included in the multiple groups calculated in the process of S33 as a third motion characteristic of the target object (S34). That is, the third motion characteristic is, for example, an index indicating the amount of angular displacement of the motion by the caregiver.

Then, the information output unit 15 outputs, for example, at least one of the first action characteristic, the second action characteristic, and the third action characteristic to the operation terminal 5 (S35).

Specifically, the information output unit 15 outputs the first action features corresponding to each group (each group corresponding to each of the eight divided ranges) in a state where they are distributed in a regular octagonal graph, for example, as shown in Figures 8 and 9.

This enables the information processing device 1 to output each index (first movement characteristic, second movement characteristic, and third movement characteristic) that quantitatively represents the movement characteristics of the caregiver.

In this regard, the first movement characteristic, the second movement characteristic, and the third movement characteristic are indices calculated by integrating values distributed in the scalar field for each group, and therefore do not have information on coordinates in the scalar field. Therefore, the information processing device 1 is able to output the movement pattern of the caregiver in a state that is easy to visualize.

In the above example, the case of extracting the movement characteristics of a caregiver has been described, but the information processing device 1 may also extract movement characteristics in the field of sports, for example.

[Movement Identification Processing in the Second Embodiment]
Next, the motion identification process according to the second embodiment will be described with reference to Fig. 10. The motion identification process according to the second embodiment will be described with reference to Figs.

As shown in FIG. 10, the CPU 101 of the information processing device 1 functions as an image identification unit 21, a flow calculation unit 22, a blur processing unit 23, a flow allocation unit 24, an action determination unit 25, and an information output unit 26 by working in cooperation with a program.

The image identification unit 21, for example, acquires video data stored in the memory area 110. Then, the image identification unit 21, for example, identifies first image data included in the video data and second image data captured after the first image data.

The flow calculation unit 22 calculates, for example, a first optical flow that indicates the displacement for each pixel between the first image data and the second image data identified by the image identification unit 21.

The blurring processing unit 23 performs, for example, a predetermined blurring process on the first optical flow calculated by the flow calculation unit 22.

Specifically, the blurring processing unit 23 performs blurring processing on the first optical flow using, for example, a Gaussian function.

The flow allocation unit 24 allocates each of the first optical flows into a plurality of groups, for example, based on the direction in which the first optical flows that have been blurred by the blur processing unit 23 are facing.

Specifically, the flow allocation unit 24, for example, quantizes the orientation corresponding to each of the first optical flows that have been blurred by the blur processing unit 23. Then, the flow allocation unit 24 allocates each of the first optical flows to a plurality of groups such that optical flows having the same quantized orientation are allocated to the same group.

The motion determination unit 25 calculates, for example, for each of the multiple groups sorted by the flow sorting unit 24, a correlation value between the length corresponding to each of the first optical flows included in each group and the length corresponding to each of the second optical flows indicating the displacement of each pixel between the third image data and the fourth image data prepared in advance. Specifically, the motion determination unit 25 calculates, for example, for each of the multiple groups, a correlation value between the directional pattern corresponding to each of the first optical flows included in each group and the directional pattern corresponding to each of the second optical flows prepared in advance. Then, the motion determination unit 25 determines, for example, from the correlation value for each of the multiple groups sorted by the flow sorting unit 24, whether or not a category (hereinafter also referred to as the first category) of the motion of the object shown in the first image data and the second image data matches a category (hereinafter also referred to as the second category) of the motion of the object shown in the third image data and the fourth image data.

The information output unit 26 outputs, for example, the determination result by the operation determination unit 25 (information indicating whether the first category and the second category match).

That is, the information processing device 1 in this embodiment calculates, for example, a correlation value between the directional pattern of the newly calculated first optical flow and the directional pattern of the second optical flow, the category of the movement of which is known in advance. Then, if the calculated correlation value satisfies a predetermined condition, the information processing device 1 determines that the first category corresponding to the first optical flow matches the second category (the category known in advance) corresponding to the second optical flow.

This enables the information processing device 1 to, for example, recognize the category of the caregiver's actions shown in each image data.

[Details of the Action Identification Process in the Second Embodiment]
Next, the details of the action identification process in the second embodiment will be described below. Figures 11 and 12 are flow charts illustrating the details of the action identification process in the second embodiment.

The image identification unit 21, for example, acquires video data stored in the memory area 110. Then, as shown in FIG. 11, the image identification unit 11 identifies, for example, first image data included in the video data and second image data captured after the first image data from the video data (S41).

Then, the flow calculation unit 22 calculates a first optical flow that indicates the displacement for each pixel between the first image data and the second image data identified in the processing of S41 (S42).

Next, the flow allocation unit 24 quantizes the orientations corresponding to each of the first optical flows calculated in the process of S42 (S43).

Then, the blurring processing unit 23 performs a predetermined blurring process on the first optical flow that has been quantized in the process of S43 (S44). A specific example of the process of S44 will be described below.

[Specific example of the process of S44]
The blurring processing unit 23 performs a process of blurring the coordinates (x, y) in the two-dimensional scalar field of the directional pattern expressed by the above formula (3) to a nearby value. This allows the blurring processing unit 23 to spatially expand the value of θ in formula (3), and allows the feature of θ to exist even at nearby points. This is different from the blurring process that spatially averages the vectors between each optical flow and other optical flows located at the coordinates surrounding each optical flow. Therefore, the information processing device 1 can improve the recognition accuracy of the category of the caregiver's actions.

Specifically, for example, when the Gaussian function is expressed as in the following formula (8), the directional pattern obtained by performing blurring processing using the Gaussian function is expressed as in the following formula (9). In the following formula (8), σ represents a parameter. Also, in the following formula (9), G(x, y, θ:t) represents the directional pattern obtained by performing blurring processing using the Gaussian function, i represents the X coordinate of each coordinate point included in the image data, and j represents the Y coordinate of each coordinate point included in the image data.

Note that when the pixel size of (x, y) is different from the pixel size of (i, j), for example when the pixel size of (x, y) is 512 x 512 and the pixel size of (i, j) is 32 x 32, the directional pattern obtained by blurring using a Gaussian function can be expressed as the following formula (10).

Next, the flow allocation unit 24 allocates the first optical flows that have been blurred in the process of S44 to multiple groups so that optical flows with the same orientation quantized in the process of S43 are allocated to the same group (S51).

Then, the motion determination unit 25 calculates, for example, for each of the multiple groups sorted in the process of S51, a correlation value between the length of the first optical flow that has been blurred in the process of S44 and the length of the second optical flow that indicates the displacement for each pixel between the third image data and the fourth image data, which are reference image data prepared in advance (S52).

Specifically, the motion determination unit 25 calculates a correlation value between the directional pattern of the first optical flow and the directional pattern of the second optical flow for each of the multiple groups assigned in the process of S51.

Next, the operation determination unit 25 calculates the sum of the correlation values for each of the multiple groups assigned in the process of S51 (S53).

Furthermore, the operation determination unit 25 determines whether the first category and the second category match, for example, depending on whether the sum calculated in the processing of S53 is equal to or greater than a predetermined threshold (S54).

After that, the information output unit 26 outputs information indicating the determination result of whether or not the first category and the second category match (S55). Specific examples of the processes from S52 to S54 are described below.

[Specific example (1) of the process from S52 to S54]
First, a first specific example of the processes from S52 to S54 will be described.

The first specific example is an example in which a directional pattern of a second optical flow, the category of which is known, is used to identify a category corresponding to a directional pattern of a first optical flow, the category of which is unknown. Hereinafter, the directional pattern of a second optical flow, the category of which is known, is also referred to as a reference directional pattern, and the directional pattern of a first optical flow, the category of which is unknown, is also referred to as an unknown directional pattern.

In the process of S52, the operation determination unit 25 calculates a cosine correlation value between the reference directivity pattern and the unknown directivity pattern for each of the multiple groups. Specifically, the operation determination unit 25 calculates the cosine correlation value for each of the multiple groups by using, for example, the following formula (11). In the following formula (11), ρ(G(θ:t), Z(θ:t _fixed )) indicates the cosine correlation value, G(i, j, θ:t) indicates the unknown directivity pattern, and Z(i, j, θ:t _fixed ) indicates the reference directivity pattern.

Then, in the process of S53, the operation determination unit 25 calculates the sum of the cosine correlation values for each of the multiple groups by using, for example, the following formula (12): In the following formula (12), ρ(G(t), Z(t _fixed )) represents the sum of the cosine correlation values.

Furthermore, in the process of S54, if, for example, the sum of the cosine correlation values calculated in the process of S53 is equal to or greater than a predetermined threshold value, the movement determination unit 25 determines that the category of the movement corresponding to the reference directionality pattern and the category of the movement corresponding to the unknown directionality pattern match.

[Specific example (2) of the process from S52 to S54]
Next, a second specific example of the process from S52 to S54 will be described.

The first specific example identifies a category corresponding to an unknown directional pattern of a first optical flow calculated from two image data (first image data and second image data) by using a reference directional pattern of a second optical flow calculated from two image data (third image data and fourth image data). In contrast, the second specific example is an example of identifying a category of an action corresponding to a series of unknown directional patterns by determining whether or not a reference directional pattern of a second optical flow corresponding to multiple intervals (time periods) calculated from three or more image data (hereinafter also referred to as a series of reference directional patterns) is included in an unknown directional pattern of a first optical flow corresponding to multiple intervals calculated from three or more image data (hereinafter also referred to as a series of unknown directional patterns).

In the process of S52, the operation determination unit 25 calculates the cosine correlation value between the sequence of the reference directivity pattern and the sequence of the unknown directivity pattern, for example, by using the above formula (11).

Then, in the process of S53, the operation determination unit 25 calculates the sum of the cosine correlation values for each of the multiple groups, for example, by using the above formula (12).

Then, in the process of S54, the action determination unit 25 uses continuous DP (see Patent Document 1, Non-Patent Document 1, and Non-Patent Document 3) to determine whether or not the category for the action corresponding to the sequence of the reference directional pattern is included in the category corresponding to the sequence of the unknown directional pattern.

Specifically, the operation determination unit 25 calculates Q(t) by using, for example, the following equation (13): In the following equation (13), τ corresponds to t _fixed in the above equation (12).

Then, if there is a time when Q(t) is equal to or greater than a predetermined threshold and is a local maximum, the motion determination unit 25 determines that the sequence of reference directional patterns is included in the sequence of unknown directional patterns, and recognizes the category of motion corresponding to the sequence of reference directional patterns as the category of motion corresponding to the sequence of unknown directional patterns.

That is, in the second specific example, by using continuous DP, matching is performed using local correlation values between a sequence of reference directional patterns and a sequence of unknown directional patterns, and a category of the action corresponding to the sequence of unknown directional patterns is identified.

1: Information processing device 5: Operation terminal 11: Image identification unit 12: Flow calculation unit 13: Flow allocation unit 14: Action identification unit 15: Information output unit 21: Image identification unit 22: Flow calculation unit 23: Blur processing unit 24: Flow allocation unit 25: Action identification unit 26: Information output unit 101: CPU
102: Memory 103: Network interface 104: Storage medium 105: Bus

Claims (7)

Identifying first image data and second image data captured after the first image data from the video data;
Identifying third image data captured after the second image data from the video data;
Calculating a first optical flow indicative of a displacement for each of a plurality of pixels in the first image data and the second image data;
calculating a second optical flow indicative of a displacement for each of the plurality of pixels in the second image data and the third image data;
sorting the first optical flows for the plurality of pixels and the second optical flows for the plurality of pixels into a plurality of groups based on orientations corresponding to the first optical flows for the plurality of pixels and the second optical flows for the plurality of pixels ;
calculating, for each of the plurality of groups, a first sum of lengths corresponding to the first optical flows for each of the plurality of pixels included in each group, and a second sum of lengths corresponding to the second optical flows for each of the plurality of pixels included in each group;
identifying, for each of the plurality of groups, the number of sums that are equal to or greater than a predetermined threshold among the first sums and the second sums corresponding to each group;
Identifying the number of each of the plurality of groups as a motion characteristic of the object captured in the first image data and the second image data;
outputting information indicating the identified operational characteristics;
An action specifying program that causes a computer to execute a process. In claim 1 ,
In the allocation process,
quantizing orientations corresponding to the first optical flow for each of the plurality of pixels and the second optical flow for each of the plurality of pixels;
assigning the first optical flow for each of the plurality of pixels and the second optical flow for each of the plurality of pixels to the plurality of groups such that optical flows having the same quantized orientation are assigned to the same group;
2. An action specifying program comprising: In claim 2 ,
In the identifying process,
calculating a sum of the first sum and the second sum corresponding to each of the plurality of groups;
identifying a motion characteristic of the object based on the sum values corresponding to each of the plurality of groups;
2. An action specifying program comprising: In claim 3 ,
In the identifying process,
obtaining a value calculated by dividing the sum corresponding to each of the plurality of groups by an elapsed time from an image capture time of the second image data to an image capture time of the third image data;
identifying the values for each of the plurality of groups as motion characteristics of the object;
2. An action specifying program comprising: In claim 2 ,
In the identifying process,
Calculating, for each of the plurality of groups, a first sum of lengths corresponding to the first optical flow for each of the plurality of pixels included in each group, and a second sum of lengths corresponding to the second optical flow for each of the plurality of pixels included in each group;
calculating an absolute value of a difference between the first sum corresponding to the first group and the second sum corresponding to the second group for each combination of a first group and a second group included in the plurality of groups;
identifying the absolute value for each combination of the first group and the second group as a motion characteristic of the object;
2. An action specifying program comprising: an image specification unit that specifies, from video data, first image data and second image data captured after the first image data , and specifies, from the video data, third image data captured after the second image data ;
a flow calculation unit that calculates a first optical flow indicating a displacement for each of a plurality of pixels in the first image data and the second image data, and calculates a second optical flow indicating a displacement for each of the plurality of pixels in the second image data and the third image data ;
a flow allocation unit that allocates the first optical flow for each of the plurality of pixels and the second optical flow for each of the plurality of pixels to a plurality of groups based on orientations corresponding to the first optical flow for each of the plurality of pixels and the second optical flow for each of the plurality of pixels ;
a motion identification unit that calculates, for each of the plurality of groups, a first sum of lengths corresponding to the first optical flows for each of the plurality of pixels included in each group and a second sum of lengths corresponding to the second optical flows for each of the plurality of pixels included in each group, identifies, for each of the plurality of groups, the number of the first sums and the second sums corresponding to each of the plurality of groups that is equal to or greater than a predetermined threshold, and identifies the number for each of the plurality of groups as a motion characteristic of an object captured in the first image data and the second image data;
and an information output unit that outputs information indicating the identified operation characteristic.
23. An information processing apparatus comprising: Identifying first image data and second image data captured after the first image data from the video data;
Identifying third image data captured after the second image data from the video data;
Calculating a first optical flow indicative of a displacement for each of a plurality of pixels in the first image data and the second image data;
calculating a second optical flow indicative of a displacement for each of the plurality of pixels in the second image data and the third image data;
sorting the first optical flows for the plurality of pixels and the second optical flows for the plurality of pixels into a plurality of groups based on orientations corresponding to the first optical flows for the plurality of pixels and the second optical flows for the plurality of pixels ;
calculating, for each of the plurality of groups, a first sum of lengths corresponding to the first optical flows for each of the plurality of pixels included in each group, and a second sum of lengths corresponding to the second optical flows for each of the plurality of pixels included in each group;
identifying, for each of the plurality of groups, the number of sums that are equal to or greater than a predetermined threshold among the first sums and the second sums corresponding to each group;
Identifying the number of each of the plurality of groups as a motion characteristic of the object captured in the first image data and the second image data;
outputting information indicating the identified operational characteristics;
A motion identification method comprising the steps of: causing a computer to execute a process.

Images