JP7624875B2 - Task recognition device and task recognition method - Google Patents

Description

The present invention relates to a task recognition device and a task recognition method, and in particular to a task recognition device and a task recognition method suitable for task monitoring by associating the relationship between workers or robots working in a manufacturing site, etc., and surrounding objects to recognize tasks.

In manufacturing sites, improvement activities are being carried out to ensure the safety of human work and to improve the accuracy of work content in order to maintain product quality. One of these improvement activities is the development of technology related to motion recognition in human work, which is being applied to the industrial field. For example, there are cases where it is necessary to detect whether actions are being taken in accordance with standard operating procedures at the manufacturing site, or to calculate the amount of time required for a specific task. The manufacturing sites that this technology targets are likely to be workplaces where a variety of machines and tools are installed, where machine movements differ depending on the process, and where different tools are used for each task.

Meanwhile, in computer vision technology related to task recognition, the rapid advances in computer performance and the ease of collecting large amounts of general data have led to the widespread use of motion recognition technology using machine learning. With this motion recognition technology, video image data is input, and an information processing device recognizes the motions made by people in the video. An example of a public database related to this type of motion recognition technology is Non-Patent Document 1, which assigns labels such as "Playing Piano," "Surfing," and "Table Tennis Shot" to each piece of image data.

Also, object algorithms for acquiring object regions from images are described in, for example, Non-Patent Documents 2 and 3.

In relation to this, a method for linking objects present in an image using an information processing device is disclosed in, for example, Patent Document 1. Patent Document 1 proposes a method for an information processing device for counting people, which finds the connection between a mirror image and a real image reflected in a mirror, and realizes accurate person counting even in images in which a mirror image is reflected. To find the connection between a mirror image and a real image, two or more moving objects corresponding to the same real moving object among the multiple moving objects are identified based on the similarity of the speed according to the positional changes of the multiple moving objects in the image, and it is determined whether the change in the amount of movement over time is similar, and the connection between the two objects (real image and mirror image) is estimated.

Techniques for behavior recognition using sensors and cameras are described in, for example, Non-Patent Document 4.

In addition, Non-Patent Document 5 describes a technique for behavior recognition using a graph convolution network (GCN) as one aspect of machine learning. GCN is a type of so-called convolutional neural network, and is a method for generating a graph consisting of nodes with feature and class information of an object and edges connecting the nodes, and inferring a target node class or the class of the entire graph using the feature of adjacent nodes. In GCN, there is a method for predefining the connections between nodes or automatically determining them according to changes in the video. Non-Patent Document 5 adopts two methods. The first method is a method for detecting the position of an object in a video captured by a camera, calculating the overlap rate from the degree of overlap between the objects, and then estimating the relationship between the two objects according to the overlap rate. The second method is a method for regarding similar objects as the same object when they are captured in frame images acquired in a time series, and determining the relationship between the objects between frames in the time direction. With such a method, a graph can be generated by predefining or estimating the relationship between objects, making it possible to recognize behavior taking the objects into consideration.

JP 2020-87358 A

UNIVERSITY OF CENTRAL FLORIDA, "UCF101 - Action Recognition Data Set", [online], [Retrieved April 16, 2021], Internet <URL: https://www.crcv.ucf.edu/research/data-sets/ucf101/> Ren, Shaoqing, et al., “Faster r-cnn: Towards real-time object detection with region proposal networks.”, IEEE transactions on pattern analysis and machine intelligence 39.6, 2016, p. 1137-1149 Yolo(Redmon, Joseph, et al. "You only look once: Unified, real-time object detection.", Proceedings of the IEEE conference on computer vision and pattern recognition, 2016 Vrigkas, M., Nikou, C. and Kakadiaris, I. A. “A Review of Human Activity Recognition Methods”, Frontiers in Robotics and AI, 2015 Wang, Xiaolong, and Abhinav Gupta, “Videos as space-time region graphs”, Proceedings of the European conference on computer vision (ECCV), 2018.

When applying computer vision technology to motion recognition technology for workers in manufacturing sites, it is necessary to recognize the unique motions that differ for each manufacturing site. In such cases, a common method is to focus only on the target worker and capture his or her movements, but this method makes it difficult to distinguish between workers who are performing multiple different tasks in the same posture. Furthermore, in manufacturing sites where machines are present, when performing a certain task, not only the worker but also the surrounding machines may move at the same time, or tools may be handled at the same time. In other words, there are cases where the object (person or object) being recognized as the target of task recognition and the surrounding objects change at the same time.

When applied to manufacturing sites, there are cases where the appearance of a machine changes but there is no significant movement. As described above, Non-Patent Document 5 uses the overlap rate or similarity between objects to estimate the relevance between objects, but no matter which method is adopted, it is possible that the relationship between objects in a manufacturing site cannot be properly handled. For example, when a machine door opens during processing work and a new workpiece is brought in and inserted into the machine, even if they are operating simultaneously, they may not overlap in the image space depending on the installation position of the camera. Therefore, it is necessary to estimate the relevance using a method that does not depend on the overlap or distance between objects.

Patent Document 1, which proposes a method for finding the connection between real images and mirror images, assumes that the target objects are the same. When performing a certain task, multiple objects (machines, workpieces, workers) present at the manufacturing site may move simultaneously, but the amount of movement over time between these machines, workpieces, workers, etc. differs for each object. For this reason, it is difficult to estimate the relevance of objects at the manufacturing site from the similarity of the change in movement amount over time between machines, workpieces, and workers.

In addition, in Patent Document 1 and Non-Patent Document 5, overlap rate, similarity, etc. are used as indices for determining the connection between objects, and a threshold value is set for the judgment. Therefore, this method can only judge that there is a correlation between the objects when the indices have dramatic values. Even if the overlap rate or similarity of the indices does not exceed a certain threshold, such a method cannot judge that there is a connection between the objects, and a problem arises in that an algorithm is required to estimate the correlation from the positional relationship of the objects, etc.

The object of the present invention is to provide a task recognition device that can generically estimate the relationships between objects using a simple algorithm when performing task recognition using connections between objects in tasks such as manufacturing sites.

The task recognition device of the present invention is preferably configured to analyze image data and perform task recognition of objects related to the task by analyzing the image data, and includes an image acquisition unit that acquires image data captured by a capture device, an analysis area detection unit that analyzes the image data obtained from the image acquisition unit and detects an area related to the object, a feature extraction unit that extracts features of the image of the area related to the object detected by the analysis area detection unit and calculates a change in the feature of the image within a predetermined period of time, and a relevance estimation unit that calculates the object relevance between a first object and a second object based on a first change in the feature of the image detected as an area related to a first object within a predetermined period of time and a second change in the feature of the image detected as an area related to a second object within the same predetermined period of time as the first change.

The present invention provides a task recognition device that can generically estimate the relationships between objects using a simple algorithm when performing task recognition using connections between objects in tasks such as manufacturing sites.

FIG. 1 is a schematic configuration diagram of an activity recognition system. 1 is a functional configuration diagram of a task recognition device according to a first embodiment. FIG. FIG. 2 is a diagram illustrating a hardware/software configuration of the task recognition device. 4 is a flowchart showing a series of processes from image acquisition to learning processing in the task recognition device according to the first embodiment. 13 is a flowchart showing details of a process for extracting an amount of change in a feature amount (in the case of a change in motion); 13 is a flowchart showing details of a feature amount change extraction process (in the case of texture change); 13 is a flowchart showing details of a relevance estimation process. 13 is a flowchart showing details of a graph generation process. 1 is a flowchart showing a series of processes from image acquisition to inference processing by the task recognition device. FIG. 11 is a diagram for explaining a specific image of calculating an object relevance from image data. 11A and 11B are diagrams illustrating an example in which object relevance is applied to graph generation. FIG. 13 is a diagram for explaining how a node work label is assigned to a node as an inference result. FIG. 11 is a functional configuration diagram of a task recognition device according to a second embodiment. 10 is a flowchart showing a series of processes from image acquisition to learning processing in a task recognition device according to a second embodiment. FIG. 13 is a diagram showing an example of an object relevance editing screen. 13A to 13C are diagrams illustrating a specific image of calculating an object relevance from image data according to the third embodiment.

Each embodiment of the present invention will be described below with reference to Figures 1 to 15.

[Embodiment 1]
Hereinafter, a first embodiment of the present invention will be described with reference to FIGS. 1 to 11. FIG.

In this embodiment, an example of a task recognition system is described that detects a worker and surrounding objects from the way a user works, expresses the state in a graph structure, and recognizes work actions by machine learning using a GCN technique based on the graph structure. Using a graph structure as a data structure for recognizing work actions is just one example, and a similar network structure or algorithm may be used for the application of this embodiment.

First, we will explain the configuration of the task recognition system using Figures 1 to 3.

As shown in FIG. 1, the task recognition system 1 is configured such that an image capture device 6, a sensor 3, and a task recognition device 100 are connected via a communication means 4.

The communication means 4 may be wired or wireless, and may be a LAN (Local Area Network) or a WAN (Wide Area Network) such as the Internet. It may also be a communication means that complies with serial standards such as USB (Universal Serial Bus) or RS-232C.

The image acquisition device 6 (photographing device) is a device that acquires image data showing the worker 2 and surrounding objects 5, and is, for example, a camera (digital camera (RGB camera), infrared camera, thermography camera, time-of-flight (TOF) camera, stereo camera, etc.) that acquires (takes) image data of videos and still images. Although the image acquisition device 6 is shown as one unit in FIG. 1, multiple cameras may be used and multiple images from different cameras may be used when there are multiple subjects to be photographed.

Sensor 3 is provided in the work environment where worker 2 performs work, detects the state of worker 2 and the work environment, and outputs physical information. Sensor 3 is, for example, a motion detection sensor, a human sensor, a temperature sensor, a humidity sensor, an acceleration sensor, a speed sensor, an acoustic sensor (microphone), an ultrasonic sensor, a vibration sensor, a millimeter wave radar, a laser radar (LIDAR: Laser Imaging Detection and Ranging), or an infrared depth sensor.

The task recognition device 100 is a device that performs processing to recognize the task actions of a worker 2 or the task actions of a surrounding object 5 based on image data acquired by an image acquisition device 6.

Next, the functional configuration of the task recognition device will be described with reference to FIG.
As shown in FIG. 2 , the activity recognition device 100 has the following functional units: an image acquisition unit 101, an analysis area detection unit 102, a feature extraction unit 103, a relevance estimation unit 104, a graph generation unit 105, an activity learning unit 106, an activity reasoning unit 107, and a memory unit 110.

The image acquisition unit 101 is a functional unit that acquires image data 200 acquired from the image acquisition device 6. The image data 200 is, for example, frame data constituting still image data or video data sent from the image acquisition device 6.

The analysis area detection unit 102 is a functional unit that detects an object area from the image obtained by the image acquisition unit 101. The object area can be detected by a method in which the user manually draws a rectangle on the image, or by a method in which the area is automatically acquired using an object detection algorithm such as those shown in Non-Patent Documents 2 and 3. Either a manual or automatic method may be used.

The feature extraction unit 103 is a functional unit that extracts features and their changes in an image.

In the feature extraction function, the feature extraction unit 103 extracts features such as color features, motion features, and CNN (Convolutional Neural Network) features obtained from an image. Note that in this embodiment, features are used as scalar values. In the feature extraction function, the feature extraction unit 103 secures data in a time series such as previous and next frames, and extracts the amount of change in the feature using the difference in the features of the previous and next frames. Features can be extracted not only from a specified area, but also as an amount of change such that the area can be expressed by one point, such as position information.

The relevance estimation unit 104 is a functional unit that calculates an object relevance that estimates the strength of the connection between objects using the change in the feature amount output by the feature extraction unit 103. The relevance estimation unit 104 normalizes (adjusts the scale between 0 and 1) the change amount obtained from each object using the correction parameter 201. Then, the normalized change in the feature amount is calculated as the strength of the relationship between the objects, and the result calculated as, for example, their product is output as the object relevance. The correction parameter 201 is a parameter for normalizing the change in the feature amount, and has the purpose of filling in the difference in the size of the rectangle between objects and the average change amount. In the above explanation, an example was shown in which the strength of the relationship between objects is calculated as the product of each change in the feature amount, but a calculation method such as the sum of each change in the feature amount or a weighted linear sum may also be used.

The graph generation unit 105 is a functional unit that generates a graph used in the GCN. The graph generation unit 105 generates graph nodes for the number of areas acquired by the analysis area detection unit 102, assigns features obtained by the feature extraction process of the feature extraction unit 103 to each node, and assigns a label of the node identification label 202. The label of the node identification label 202 is the name of an object in the manufacturing site, such as "robot", "worker", or "machine". It is assumed that there is a connection between each node, and edges are provided between those nodes, and weights are assigned to the graph edges using the object relevance obtained from the relevance estimation unit 104.

The task learning unit 106 is a functional unit that performs machine learning using GCN based on the node task label 203, which is the correct label for the task, and the graph generated by the graph generation unit 105, to generate an inference model 210. In the machine learning process of the task learning unit 106, learning is performed using a GCN algorithm based on the generated graph, and the inference model 210 is output as a result. The node task label 203 is a correct label that recognizes the task, for example, "welding" or "screw tightening" when the node is a "robot", or "control panel operation" or "floor cleaning" when the node is a "worker".

The task inference unit 107 is a functional unit that performs inference on the image data 200 for which task recognition is to be performed based on the inference model 210 obtained as a result of learning by the task learning unit 106, and obtains the inference result 220.

The memory unit 110 is a functional unit that stores data. The memory unit 110 stores image data 200, correction parameters 201, node identification labels 202, node work labels 203, inference models 210, and inference results 220.

Next, the hardware and software configuration of the task recognition device will be described with reference to FIG. 3. The task recognition device 100 can be realized by a general information processing device (computer), and as shown in FIG. 3, includes a processor 11, a main memory device 12, a communication device 13, an input device 14, an output device 15, and an auxiliary memory device 16.

The processor 11 is a semiconductor device that executes programs on the main memory device 12 and controls each part of the task recognition device 100, and is, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), an AI (Artificial Intelligence) chip, etc.

The main memory device 12 is a device that temporarily stores programs and data, and is, for example, a RAM (Random Access Memory) such as an SRAM (Static Random Access Memory) or a DRAM (Dynamic Random Access Memory).

The communication device 13 is an interface device that communicates with other devices via the communication means 4, and is, for example, a NIC (Network Interface Card), a wireless communication module, a USB (Universal Serial Interface) module, a serial communication module, etc. The communication device 13 can also function as an input device that receives information from other devices that are communicatively connected. The communication device 13 can also function as an output device that transmits information to other devices that are communicatively connected. The task recognition device 100 communicates with the image acquisition device 6 and the sensor 3 via the communication means 4 using the communication device 13.

The input device 14 is a user interface device that accepts information from the user, such as a keyboard, mouse, card reader, touch panel, etc.

The output device 15 is a user interface device that outputs various types of information (display output, audio output, print output, etc.), and is, for example, a display device (LCD (Liquid Crystal Display), graphics card, etc.) that visualizes various types of information, an audio output device (speaker), a printer, etc.

The auxiliary storage device 16 is a hard disk drive, a solid state drive (SSD), etc. The programs and data stored in the auxiliary storage device 16 are loaded into the main storage device 12 as needed, and the processor 11 references the loaded data and executes the programs.

In this embodiment, an image acquisition program 161, an analysis area detection program 162, a feature extraction program 163, a relevance estimation program 164, a graph generation program 165, a work learning program 165, and a work inference program 166 are installed in the auxiliary storage device 16.

The image acquisition program 161, the analysis area detection program 162, the feature extraction program 163, the relevance estimation program 164, the graph generation program 165, the work learning program 165, and the work inference program 166 are programs that respectively realize the functions of the image acquisition unit 101, the analysis area detection unit 102, the feature extraction unit 103, the relevance estimation unit 104, the graph generation unit 105, the work learning unit 106, and the work inference unit 107.

Note that, although the various functions of the task recognition device 100 have been described as being implemented by the processor 11 reading and executing a program stored in the main memory device 12, they may also be implemented by the hardware (FPGA, ASIC, AI chip, etc.) that constitutes the task recognition device 100.

Next, the processing of the task recognition device will be explained using Figures 4 to 8.

First, the image acquisition unit 101 of the task recognition device 100 acquires the image data 200 transmitted from the image acquisition device 6 (S100).

Next, the analysis area detection unit 102 detects an object area from the image data 200 obtained by the image acquisition unit 101.

Next, the feature extraction unit 103 extracts features between a specified number of frames obtained from the image (S102).

Next, the feature extraction unit 103 secures data in a time series, such as previous and next frames, and extracts the amount of change in the feature using the difference between the features of the previous and next frames (S103). The process of extracting the amount of change in the feature will be described in detail later.

Next, the relevance estimation unit 104 calculates an object relevance that estimates the strength of the connection between the objects using the amount of change in the feature extracted by the process S103 of extracting the amount of change in the feature of the feature extraction unit 103 (S104). The process of calculating the object relevance between the objects will be described in detail later.

Next, the graph generation unit 105 generates a graph to be used in the GCN based on the object association between the objects in S104 (S105).

Next, the work learning unit 106 performs machine learning using GCN based on the graph generated in S105 to generate an inference model 210 (S106).

Next, the process of extracting the amount of change in features will be described in detail using Figures 5A and 5B.

This process corresponds to S103 in FIG. 4. In this embodiment, we will explain in detail the cases of movement change and texture change as examples of features, but the feature of image data may be any data that contributes to task recognition.

First, we will use Figure 5A to explain the process of extracting changes in motion from a moving object in an image as changes in features.

First, a group of predetermined object regions is acquired (S200). Here, in order to acquire the regions from which movement is to be acquired, the regions do not necessarily need to be focused on the object itself, but can be predetermined regions associated with the object in image space.

Next, the amount of movement is calculated from within the region using images from previous and next frames by the optical flow method (S201). Optical flow is a method for analyzing the pattern of how the movement of an object appears between adjacent frames, which is caused by the movement of the object or camera.

Next, the sum of the amount of movement within the area obtained for each object is calculated (S202).

Next, the sum of the calculated amounts of movement is output as the amount of change in the feature (S203).

Next, we will use Figure 5B to explain the process of extracting changes in image texture as changes in feature quantities.

A change in the texture of an image is when there is a change in the RGB values or gray values of the image.

First, a set of object regions is obtained (S210).

Next, a trained model capable of identifying an object or changes in an object is acquired as a feature extraction model (S211). For example, models for extracting local features of an image include a template feature model, a holistic feature model, a co-occurrence feature model, and a part feature model.

Feature quantities for each object are obtained by applying the image data of each object region to the feature extraction model (S212).

Then, previous and next feature amounts are obtained from image data such as previous and next frames, and the distance between the feature amounts of the previous and next frames is calculated (S212). For example, when the feature amount consists of n elements, the distance between the feature amounts is the n-dimensional Euclidean distance.

Next, the distance value calculated in S212 is output as the change in the feature amount (S214).

Next, a process for estimating the relevance between objects will be described with reference to FIG.
This process corresponds to S104 in FIG.

First, the amount of change in the feature amount of each object region extracted in S103 of FIG. 4 is obtained (S300).

Next, the amount of change in the feature amount is normalized using the correction parameters 201 (S301). This is because the feature amount of each region can change significantly, so it is meaningful to match the scale, and correction using the correction parameters 201 is not necessarily required.

Next, the object relevance is calculated based on the amount of change in the features of the two regions normalized in S301 (S302). This object relevance is an index showing the degree of object relationship.

For example, when the feature amount of the area of object A is C _A and the feature amount of the area of object B is C _B , the product of these two is taken and the object relevance is calculated by the following (Equation 1).
Object relevance (object A, object B)=C _A ×C _B ... (Equation 1)

Furthermore, for example, the sum of the two is calculated, and the object relevance is calculated by the following (Equation 2).
Object relevance (object A, object B)=C _A +C _B ... (equation 2)

Furthermore, for example, a weighted linear sum of the two is taken, and the object relevance is calculated by the following (Equation 3).
Object relevance (object A, object B)=K _A C _A +K _B C _B ... (Equation 3)

Here, K _A and K _B are weighting coefficients determined depending on how much importance is placed on the amount of change in the feature amounts of object A and object B, respectively.
Moreover, calculations other than these may be performed.

Next, the object relevance obtained in S313 is normalized using the correction parameter 201 (S303). This is to make learning easier in the subsequent learning phase, and correction using the correction parameter 201 is not necessarily required.

Next, the object relevance calculated in S303 is output (S304).

Next, the graph generation process will be described in detail with reference to FIG.
The feature amount obtained by the feature amount extraction process in S102 of FIG. 4 is acquired for each object region (S400).

Next, the node specific labels assigned to each object region are obtained from the node specific labels 202 (S401).

Next, the object relevance between each object obtained by the relevance estimation process in S104 of FIG. 4 is obtained (S402).

Next, the features acquired in S411 and the node-specific label acquired in S412 are assigned to each node (S403).

Next, the object relevance is assigned as the weight of the edge between the nodes (S414). This weight is, for example, the edge weight used in the convolutional layer of the GCN.

In this way, by using the generated graph to learn, for example, using a GCN, it is possible to construct an inference model 210 that can estimate the activity of an object that corresponds to an edge that has been assigned a node-specific label.

Next, a series of processes from image acquisition to inference processing by the task recognition device will be described with reference to FIG.
The processes of S100 to S105 are similar to those shown in Fig. 4. Here, the image data 200 input in S100 is image data 200 for making inferences regarding object task recognition.

After the graph generation in S105, the task inference unit 107 performs task recognition inference based on the inference model 210 obtained in S106 in FIG. 4, and outputs the inference result (S110). The inference result is a graph in which a node task label 203 is assigned to each node. A specific example of the inference result will be described in detail later.

Next, a specific image of calculating an object relevance from image data will be described with reference to FIG.
Here, an example of calculating the object relevance of object A and object B for time series data such as video data will be described. Here, clock 1, clock 2, ... are the range of the time axis of the time series data. When object A and object B exist on an image, image features are obtained from image data obtained in the same frame. When focusing on clock 1 (C01) between a predetermined number of frames, a distance value is calculated using Euclidean distance or the like from the features obtained in feature extraction F1A in the first frame and feature extraction F2A in the last frame, and a change amount d1A in the feature amount of object A is obtained. On the other hand, feature extraction is performed in the same manner for object B, and a change amount d1B in the feature amount is obtained from the distance value of object B. From the change amount A11D and the change amount A21D, an object relevance P1 is obtained using the product thereof (in the case of (Equation 1)) or the like. P1 is the object relevance of clock 1 (C01). A similar process is performed for clock 2 (C02) to calculate object relevance P2. A similar process is performed for each clock every time video data is obtained. It is not necessary to use different frames in each clock, and some of the frames may be the same.

Next, an example of applying the object relevance to graph generation will be described with reference to FIG.
Assume that for object A, a change amount F11A in the feature amount obtained within a predetermined clock is acquired, and for object B, a change amount F11B in the feature amount obtained within the same clock as for object A is acquired. At this time, the change amount of each feature amount is normalized by the correction parameter 201, and for object A, a change amount F11AN in the feature amount after normalization is acquired, and for object B, a change amount F11BN in the feature amount after normalization is acquired.

Next, object relevance P11 is calculated from the product of each change amount (in the case of (Equation 1)), and is applied as weight w11 to the edge connecting object A and object B shown in graph G01 from the change amount at a specified clock. In this way, similar processing is performed between each object, and weights are applied to the edges connecting the objects. Figure 10 shows an example in which the average object relevance of object A and object B in the time range of interest is 0.1, and this is applied as the weight of the edge connecting the robot node and the processing machine node.

Next, the task recognition result of the inference result will be described with reference to FIG.
FIG. 11 is a diagram for explaining how a node work label is assigned to a node as an inference result.

In this embodiment, task recognition is performed by the task inference process of S110 in FIG. 8 based on the inference model 210 generated by the task learning process of S106 in FIG. 4, and node task labels 203 are obtained for the objects represented as each node.

In the example of Figure 11, as shown in Figure 11(a), the node operation labels obtained at time T1 for node 1 "robot", node 2 "machine", and node 3 "workpiece" are node 1 "welding", node 2 "stop", and node 3 "welded". Figure 11(b) is a timing chart targeting node 1 "robot". This timing chart shows that at time T1, the robot is performing welding work.

As described above, according to this embodiment, object relevance is calculated based on the amount of change in image features, and GCN learning and inference are performed using object relevance as edge weights based on a graph in which objects are nodes and related objects are edges. As a result, when performing task recognition using connections between objects in task recognition at manufacturing sites, etc., task recognition can be performed that estimates the relevance between objects in a general purpose manner using a simple algorithm.

[Embodiment 2]
The second embodiment will be described below with reference to FIGS.

In the first embodiment, an example of a task recognition device that calculates the object relevance of each object from the amount of change in the feature amount of the object and applies it as the edge weight of the GCN graph to perform task recognition was described.

In this embodiment, a task recognition device performs task recognition similar to that of embodiment 1, and further, an example is described in which the user can edit the object association between objects corresponding to nodes.

The following will focus on the differences from embodiment 1.

First, the functional configuration of the task recognition device according to the second embodiment will be described with reference to FIG.
The task recognition apparatus 100 of this embodiment includes a relevance editing unit 108 in addition to the components of the task recognition apparatus 100 of the first embodiment shown in FIG.

The relevance editing unit 108 is a functional unit that edits the relevance value of objects recognized as nodes.

The storage unit 110 also stores association data 204. The association data 204 is data that holds the value of the association degree of an object recognized as a node.

Next, a series of processes from image acquisition to learning processing of the task recognition device according to embodiment 2 will be described with reference to FIG. 13.

In the series of processes from image acquisition to learning processing in the task recognition device of this embodiment, in addition to the processes shown in FIG. 4 of the first embodiment, a relevance editing process S120 is added between S104 and S105.

The relevance editing process S120 is a process for editing the relevance values of objects recognized as nodes in the graph. This process resets the object relevance defined by the user, and the object relevance results obtained in the relevance estimation process S104 and the object relevance defined by the user are output as relevance data 204 and passed to the graph generation process S105.

Next, a user interface provided by the task recognition device will be described with reference to FIG.
The task recognition device 100 of this embodiment can display and edit the object relevance assigned to the edges of the graph using the object relevance editing screen 400 shown in FIG. 14. The task recognition device 100 reads the result of the relevance estimation process S04 or a setting file that allows a specific object to be selected from candidates. Next, the user selects two objects to link the objects. In the example of FIG. 14, object A is selected in the box 411 of object 1, and object B is selected in the box 412 of object 2. The degree of relevance between the objects selected in each box can be entered in the box 42, and the user can set an arbitrary degree of relevance between the objects. In addition, the type of feature can be selected using the feature box 43, and the amount of change in the feature of the object can be set in the box 441 of object 1 and the box 442 of object 2, respectively, and the user has the function of setting an arbitrary object relevance. In addition, the graph display 500 visualizes the degree of association, and it can be seen that nodes 511 and 512 are connected, that features 53 and the like are displayed for each object, and that objects A and B are connected. Box 52 allows the numerical value of the object association degree between objects A and B to be edited.

As described above, the task recognition device of this embodiment allows the user to arbitrarily edit the amount of change in the feature quantities of each object and the object relevance.

[Embodiment 3]
Hereinafter, the third embodiment of the present invention will be described with reference to FIG.
In the first embodiment, a task recognition device has been described that obtains the amount of change in the feature amount of an object region in an image, calculates the amount of change in the feature amount in a certain object region and the amount of change in the feature amount in another object region to show the relationship between the objects, calculates the object relevance between the two objects, and performs machine learning using this as the weight of the edges of a graph in a GCN.

In this embodiment, as in embodiment 1, the amount of change in the feature amount of an object region in an image is found, and then the object relevance between one object and another object is calculated. However, this embodiment also focuses on the transition of image data over time to calculate the object relevance.

The following will focus on the differences from embodiment 1.

The task recognition device 100 of this embodiment has a configuration similar to the functional configuration shown in FIG. 2 of the first embodiment, and calculates the object relevance between the two objects by calculating the amount of change in the feature amount in the area of the object and the amount of change in the feature amount in the area of the other object, which is the relevance estimation process S104 of FIG. 4, but the method is different.

The relevance estimation process S104 in the third embodiment will be described below with reference to FIG.
The relevance estimation unit 104 of the task recognition apparatus 100 acquires feature amount 1 (FA1) obtained by the optical flow or the like of the object A (60A) and feature amount 2 (FA2) obtained by 3DCNN or the like.

3DCNN is a method for recognizing actions in videos that takes spatiotemporal information into account by combining spatial information (2D) and temporal information (1D) and performing 3D convolution.

The relevance estimation unit 104 also acquires feature 1 (FB1) and feature 2 (FB2) for object B (60B) in the same manner as for object A (60A).

Next, distance values (d11, d12, d21, d22) are calculated between features obtained within the same clock between objects.

Then, the difference in distance values between the preceding and succeeding clocks is calculated, and the different feature amount 1 and feature amount 2 are combined to calculate the object relevance. For example, the distance d11 between feature amount 1 of object A and feature amount 1 of object B obtained at clock 1 is calculated. The distance d12 is calculated similarly at clock 2. The change between clock 2 and clock 1 is obtained from the absolute value of the difference between distance d11 and distance d12, and the degree of change between the clocks (D1R) is obtained. For example, the reciprocal of the value obtained by adding 1 to the absolute value Δ1 of the difference between distance d11 and distance d12 is taken to obtain the degree of change between the clocks. That is, the degree of change between the clocks (D1R) is calculated by the following (Equation 4).
D1R=1/(Δ1+1)...(Formula 4)

Similarly, for feature amount 2 of object A and feature amount 2 of object B, the change between clocks 2 and 1 is obtained from the absolute value of the difference between distance d21 and distance d22, and the degree of change between the clocks (D2R) is obtained. For example, the reciprocal of the value obtained by adding 1 to the absolute value Δ2 of the difference between distance d21 and distance d22 is taken to obtain the degree of change between the clocks. That is, the degree of change between the clocks (D2R) is calculated by the following (Equation 5).
D2R=1/(Δ2+1)...(Formula 5)
Then, the arithmetic mean value is calculated from the degrees of change D1R and D2R, and this is set as the similarity S.

In generating the graph, when the edge weight of the graph of clock 1 is W1 and the edge weight of the graph of clock 2 is W2, the final edge weight W2' of clock 2 is calculated by the following (Equation 6).
W2'=W1×S+W2 (Equation 6)

In other words, when the degree of change between clocks is small, the weight of the edge in the graph for clock 2 is not emphasized, and when the degree of change between clocks is large, the weight of the edge in the graph for clock 2 is emphasized.

As described above, in this embodiment, by calculating the difference between objects and the difference between clocks, it is possible to calculate object relevance taking into account time series data and generate a GCN graph.

1 ... task recognition system, 2 ... worker, 3 ... sensor, 4 ... communication means, 5 ... surrounding object, 6 ... image acquisition device,
100...work recognition device,
11: processor, 12: main storage device, 13: communication device, 14: input device, 15: output device, 16: auxiliary storage device,
101: image acquisition unit, 102: analysis area detection unit, 103: feature extraction unit, 104: relevance estimation unit, 105: graph generation unit, 106: work learning unit, 107: work reasoning unit, 108: relevance editing unit, 110: memory unit,
200: image data, 201: correction parameters, 202: node specific labels, 203: node working labels, 204: association data, 210: inference model, 220: inference results

Claims (10)

A task recognition device that analyzes image data and performs task recognition of an object related to the task by analyzing the image data,
an image acquisition unit that acquires image data captured by the imaging device;
an analysis area detection unit that analyzes the image data obtained from the image acquisition unit and detects an area related to an object;
a feature extraction unit that extracts a feature amount of an image of a region related to the object detected by the analysis region detection unit and calculates a change amount of the feature amount of the image within a predetermined period of time;
and a relevance estimation unit that calculates an object relevance between a first object and a second object based on a first change in a feature amount for the image detected as an area related to a first object within a predetermined period of time and a second change in a feature amount for the image detected as an area related to a second object within the same predetermined period of time as the first change in the feature amount. the first change amount of the feature quantity and the second change amount of the feature quantity are scalar quantities,
2. The task recognition device according to claim 1, wherein the object relevance between the first object and the second object is calculated by any one of a product, a sum, or a weighted linear sum of a first change amount in the feature amount and a second change amount in the feature amount. The task recognition device according to claim 1 further comprises a task learning unit that performs machine learning for task recognition based on the object relevance calculated by the relevance estimation unit and the feature amount of the image of the area related to the object extracted by the feature extraction unit, and outputs an inference model. The task recognition system according to claim 3 further comprises a task inference unit that performs task recognition inference based on image data captured by a photographing device and the inference model output by the task learning unit. The data includes data expressing a graph in which regions related to the object are nodes and related objects are connected by edges;
4. The task recognition device according to claim 3, wherein an inference model is generated that generates a node task label for the node by using the object relevance as a weight between edges and a Graph Convolution Network (GCN). The data includes data expressing a graph in which regions related to the object are nodes and related objects are connected by edges;
2. The task recognition apparatus according to claim 1, further comprising a display/editing unit for displaying and editing the feature amount of the area related to the object and the object relevance. the relevance estimation unit calculates a first object relevance between the first object and the second object based on a first amount of change in a feature amount related to the image detected as a region related to a first object within a first period and a second amount of change in a feature amount related to the image detected as a region related to a second object within the first period;
calculating a second object relevance between the first object and the second object based on a first change amount of a feature amount related to the image detected as a region related to a first object within a second period and a second change amount of a feature amount related to the image detected as a region related to a second object within the second period;
calculating a degree of change in the object relevance based on an absolute value of a difference between the first object relevance and the second object relevance, and calculating a similarity based on a degree of change in the object relevance using a first amount of change within the first period;
correcting a weight of a first edge corresponding to a first period in the graph by a similarity based on a degree of change in the object relevance;
6. The task recognition device according to claim 5, further comprising: a sum of a weight of a first edge corresponding to the corrected first period and a weight of a second edge in the second period, to calculate the corrected weight of the second edge in the second period. A task recognition method using a task recognition device that analyzes image data and recognizes tasks of a worker and a task device by analyzing the image data, comprising:
an image acquiring step in which an image acquiring unit acquires image data captured by the photographing device;
an analysis region detection step in which an analysis region detection unit analyzes the image data obtained by the image acquisition step and detects a region related to the object;
a feature extraction step of extracting a feature amount related to an image of the area related to the object detected by the analysis area detection step, and calculating a change amount of the feature amount related to the image within a predetermined period of time;
a relevance estimation step of calculating an object relevance between the first object and the second object based on a first change amount of a feature amount of the image detected as an area related to a first object within a predetermined period and a second change amount of a feature amount of the image detected as an area related to a second object within the same predetermined period as the first change amount;
a task learning step of performing machine learning for task recognition based on the object relevance calculated in the relevance estimation step and feature amounts related to the image of the area related to the object extracted in the feature extraction step, and outputting an inference model;
A task recognition method comprising a task inference step of performing inference for task recognition based on image data captured by a photographing device and the inference model output by the task learning step. the first change amount of the feature quantity and the second change amount of the feature quantity are scalar quantities,
9. The task recognition method according to claim 8, wherein the object relevance between the first object and the second object is calculated by any one of a product, a sum, or a weighted linear sum of a first change amount in the feature amount and a second change amount in the feature amount. the task recognition device has data representing a graph in which regions related to the object are nodes and related objects are connected by edges;
9. The task recognition method according to claim 8, further comprising the steps of: generating an inference model that generates a node task label for the node by using the object relevance as a weight between edges and a graph convolution network (GCN).

Images