JP7654983B2 - WORK ESTIMATION DEVICE, CONTROL METHOD FOR WORK ESTIMATION DEVICE, INFORMATION PROCESSING PROGRAM, AND RECORDING MEDIUM - Google Patents

Description

The present invention relates to a work estimation device that estimates the work being performed by a worker in a workplace.

Conventionally, various attempts have been made to use imaging data that captures the state of work processes in production sites such as factories to improve the work processes. For example, Patent Document 1 listed below discloses a safety management system that uses the results of analyzing image data to determine the behavior of workers. Research is also being conducted into HOI detection, which detects the actions of a human being with respect to an object captured in a captured image as an interaction between the human and the object (HOI).

JP 2018-173957 A

However, there is a problem in that prediction of work content based on HOI detection alone is not sufficiently accurate.

One aspect of the present invention aims to realize a task estimation device that can estimate with sufficient accuracy the task content that a worker is performing on a task object from a captured image.

In order to solve the above problems, a task estimation device according to one aspect of the present invention is a task estimation device that estimates the content of a task being performed by a worker in a workplace, and includes: a prediction unit that uses a task prediction model, which is a trained model that receives an image of the workplace or feature information generated from the workplace image as an input and outputs the content of the task being performed by the worker captured in the workplace image on the work object captured in the workplace image, from the workplace image or the feature information; a determination unit that determines whether the prediction by the prediction unit is correct or not, using at least one of the task content predicted without using the task prediction model and the determination result of whether the worker is present in the workplace; and an estimation unit that, when the determination unit determines that the prediction by the prediction unit is correct, estimates the task content predicted by the prediction unit as the task content being performed by the worker in the workplace.

According to the above configuration, the work estimation device determines whether the work content predicted using the work prediction model is correct using at least one of the work content predicted without using the work prediction model and the determination result of the presence or absence of the worker in the workplace. Then, when it is determined that the prediction using the work prediction model is correct, the work estimation device estimates that the work content predicted using the work prediction model is the work content being performed by the worker in the workplace.

In other words, the work estimation device verifies the accuracy of the work content predicted using the work prediction model using at least one of the work content predicted without using the work prediction model and the determination result of whether or not the worker is present in the workplace.

Therefore, the work estimation device has the effect of being able to estimate with high accuracy the content of the work being performed by the worker in the workplace from the workplace image.

The work estimation device according to one aspect of the present invention may further include a work detection unit that predicts the content of the work being performed by the worker in the work area from detection information of a sensor that detects at least one state of the tool used in the work, the work object, and the work area, and the determination unit may determine that the prediction by the prediction unit is correct when the content of the work predicted by the prediction unit matches the content of the work predicted by the work detection unit.

According to the above configuration, the work estimation device determines whether the work content predicted from the detection information of the sensor matches the work content predicted using the work prediction model. Then, when the work estimation device determines that the two match, it estimates that the work content predicted using the work prediction model is the work content being performed by the worker in the workplace.

In other words, the task estimation device verifies the accuracy of the task content predicted using the task prediction model based on the task content predicted from the sensor detection information.

Therefore, the work estimation device has the effect of being able to estimate with high accuracy the content of the work being performed by the worker in the workplace from the workplace image.

The work estimation device according to one aspect of the present invention may further include a presence/absence prediction unit that predicts whether the worker is present in the workplace from the overall image, the workplace image, or the feature information generated from the overall image or the workplace image, using a presence/absence model that is a trained model that receives as input (A) an overall image of a plurality of workplaces including the workplace, or (B) the workplace image, or (C) feature information generated from the overall image or the workplace image, and outputs whether the worker is present in the workplace, and the determination unit may determine that the prediction by the prediction unit is correct when the presence/absence prediction unit predicts that the worker is present in the workplace.

According to the above configuration, the work estimation device predicts whether or not the worker is present in the workplace using the presence/absence model. Then, when the work estimation device predicts that the worker is present in the workplace, it estimates that the content of the work predicted using the work prediction model is the content of the work being performed by the worker in the workplace.

In other words, the work estimation device verifies the accuracy of the work content predicted using the work prediction model based on the presence or absence of the worker in the workplace predicted using the presence or absence model.

Therefore, the work estimation device has the effect of being able to estimate with high accuracy the content of the work being performed by the worker in the workplace from the workplace image.

The work estimation device according to one aspect of the present invention may further include a work detection unit that predicts the content of the work being performed by the worker in the work area from detection information from a sensor that detects at least one state of the tool used in the work, the work object, and the work area, and a presence/absence prediction unit that predicts whether the worker is present in the work area from the overall image, the work area image, or the feature information generated from the overall image or the work area image using a presence/absence model that is a trained model that receives as input (A) an overall image of a plurality of work areas including the work area, or (B) the work area image, or (C) feature information generated from the overall image or the work area image and outputs whether the worker is present in the work area. The determination unit may determine that the prediction by the prediction unit is correct when the presence/absence prediction unit predicts that the worker is present in the work area and the content of the work predicted by the prediction unit matches the content of the work predicted by the work detection unit.

According to the above configuration, the work estimation device predicts the content of the work predicted from the detection information of the sensor, and predicts the presence or absence of the worker in the workplace. Then, when the work estimation device predicts that the worker is present in the workplace and the content predicted from the detection information of the sensor matches the content predicted using the work prediction model, it determines that the content predicted using the work prediction model is correct.

In other words, the work estimation device verifies the accuracy of the work content predicted using the work prediction model based on the work content predicted from the detection information of the sensor and the presence or absence of the worker in the workplace predicted using the presence or absence model.

Therefore, the work estimation device has the effect of being able to estimate with high accuracy the content of the work being performed by the worker in the workplace from the workplace image.

In a work estimation device according to one aspect of the present invention, the work prediction model may be constructed by machine learning using a learning dataset in which the content of the work being performed by the worker in the work area, predicted from detection information from a sensor that detects the tool used in the work, the work object, and at least one state of the work area, is labeled to the work area image or feature information generated from the work area image.

According to the above configuration, the work estimation device constructs the work prediction model by machine learning using a learning dataset that is labeled with "the content of the work being performed by the worker in the workplace" predicted from the detection information of the sensor.

It is known that supervised learning requires a large amount of training data (training dataset), and the labeling process required to generate such training data and the labels (metadata) attached to the training data are also called "annotations."

For example, even if the teacher data learned when constructing the work prediction model does not require the setting of a bounding box for detecting the worker, the following five annotations are required for the work area image. That is, it is necessary to set the X and Y coordinates of the lower left and upper right points of the four points that indicate the range of the bounding box for detecting the work object for the work area image, that is, four annotations are required. Furthermore, it is necessary to assign information (label) to the work area image that indicates "the content of the work that the worker captured in the work area image is performing on the work object captured in the work area image," that is, one annotation is required.

And if we were to have a human determine "the work that the worker captured in the workplace image is performing on the work object captured in the workplace image" for each of the large number of workplace images, the cost required would be extremely high.

Therefore, when generating the teacher data to be learned when constructing the work prediction model, the "work content" predicted from the detection information of the sensor, rather than the "work content" determined by a human from the workplace image, is assigned as a label indicating the "work content." In other words, in the teacher data to be learned when constructing the work prediction model, the "work content" predicted from the detection information of the sensor, rather than the "work content" determined by a human from the workplace image, is assigned as a label indicating the "work content."

Therefore, the cost required to construct the task prediction model (particularly the annotation cost required to generate the learning dataset) can be significantly reduced compared to the case where "content determined by a human from the workplace image" is used as a label indicating the "task content". For example, one of the above-mentioned five annotations required for the workplace image is changed from "content determined by a human" to "content predicted by a device such as the task estimation device from the detection information of the sensor", thereby reducing annotation costs by 20%.

Therefore, with the above configuration, the task estimation device has the effect of being able to estimate with high accuracy the content of the task being performed by the worker in the workplace from the workplace image using the task prediction model with significantly reduced annotation costs.

In order to solve the above problem, a control method according to one aspect of the present invention is a control method for a work estimation device that estimates the content of work being performed by a worker in a workplace, and includes a prediction step of predicting the content of work being performed by the worker on the work object captured in the workplace image from the workplace image or the feature information using a work prediction model, which is a trained model that receives an image of the workplace or feature information generated from the workplace image and outputs the content of work being performed by the worker captured in the workplace image on the work object captured in the workplace image, from the workplace image or the feature information; a determination step of determining whether the prediction in the prediction step is correct or not, using at least one of the content of work predicted without using the work prediction model and the determination result of whether the worker is present in the workplace; and an estimation step of estimating the content of the work predicted in the prediction step as the content of the work being performed by the worker in the workplace, if it is determined in the determination step that the prediction in the prediction step is correct.

According to the above configuration, the control method determines whether the work content predicted using the work prediction model is correct using at least one of the work content predicted without using the work prediction model and the determination result of the presence or absence of the worker in the workplace. Then, when it is determined that the prediction using the work prediction model is correct, the control method estimates that the work content predicted using the work prediction model is the work content being performed by the worker in the workplace.

In other words, the control method verifies the accuracy of the work content predicted using the work prediction model using at least one of the work content predicted without using the work prediction model and the determination result of whether or not the worker is present in the workplace.

The control method thus has the effect of being able to estimate with high accuracy the content of the work being performed by the worker in the workplace from the workplace image.

According to one aspect of the present invention, it is possible to estimate with sufficient accuracy from the captured image the work that a worker is performing on a work object.

1 is a block diagram showing a configuration of a main part of an information processing device according to a first embodiment of the present invention; FIG. 2 is a diagram showing an overall outline of a control system including the information processing device of FIG. 1 . 10 is a flow diagram illustrating an overall overview of a process executed by the information processing device in FIG. 1 when verifying a first prediction using only a task prediction of a second prediction. 10 is a flow diagram illustrating an overall outline of a process executed by the information processing device in FIG. 1 when the first prediction is verified using only the presence/absence prediction of the second prediction. 10 is a flow diagram illustrating an overall outline of a process executed by the information processing device of FIG. 1 when a first prediction is verified using both presence/absence prediction and work prediction. FIG. 10 is a diagram illustrating an example of annotations required for generating teacher data.

[Embodiment 1]
Hereinafter, an embodiment according to one aspect of the present invention (hereinafter also referred to as "the present embodiment") will be described with reference to the drawings. Note that the same or corresponding parts in the drawings are given the same reference numerals and their description will not be repeated. In the present embodiment, an information processing device 10 will be described as a typical example of "an operation estimation device that estimates the content of an operation Op being performed by a worker Hu at a work site Ar."

In the following description, "n", "m", "p", "q", "x", and "y" each represent an integer greater than or equal to 1, with "p" and "q" being different integers and "x" and "y" being different integers.

§1. Application Example In order to facilitate understanding of the information processing device 10 (task estimation device) according to one aspect of the present invention, first, an environment in which the information processing device 10 is applied will be described. For example, the information processing device 10 uses the analysis results of a work site image Ic obtained by capturing an image of a work site Ar to generate information (support information) that contributes to the efficiency of a work Op performed by a worker Hu on a work object Ob. First, a worker Hu, a work object Ob, a work Op, a work site Ar, and a factory Fa including one or more work sites Ar will be described.

(About the factory and work process)
In a factory Fa, which is a production site, for example, various products are produced through one or more work processes Pr(1), Pr(2), Pr(3), ..., Pr(n). Each of the multiple work processes Pr(1), Pr(2), Pr(3), ..., Pr(n) carried out in the factory Fa is, for example, a "painting" process, a "main work assembly" process, a "installation of the main work into the main body" process, and an "inspection" process.

When it is necessary to distinguish between multiple factories Fa, they will be distinguished by adding suffixes such as "(A)", "(B)", "(C)", ..., "(Z)" to the reference numerals. For example, they will be distinguished by notation such as "Factory Fa(A)", "Factory Fa(B)", "Factory Fa(X)", ..., "Factory Fa(Z)". When there is no particular need to distinguish between each of the multiple factories Fa, they will simply be referred to as "Factory Fa".

In addition, when it is necessary to distinguish between multiple work processes Pr, they are distinguished by adding suffixes such as "(1)", "(2)", "(3)", ..., "(n)" to the reference numeral. For example, they are distinguished by notation such as "work process Pr(1)", "work process Pr(2)", "work process Pr(3)", ..., "work process Pr(n)". When there is no particular need to distinguish between multiple work processes Pr, they are simply referred to as "work process Pr".

(About the work)
In a work process Pr, a worker Hu performs one or more operations Op(1), Op(2), Op(3), ... Op(n). For example, in a work process Pr(x), a worker Hu performs operations Op(x1), Op(x2), and Op(x3). For example, when a work process Pr(m) is a "painting" process, the operation Op(x1) may be "holding" the work object Ob, the operation Op(x2) may be "painting" the work object Ob, and the operation Op(x3) may be "transporting" the work object Ob.

When it is necessary to distinguish between multiple work Ops, they will be distinguished by adding suffixes such as "(1)", "(2)", "(3)", ..., "(n)" to the reference numerals, and when there is no particular need to distinguish between them, they will simply be referred to as "work Ops".

(About the workplace)
The factory Fa includes one or more work sites Ar(1), Ar(2), Ar(3), ..., Ar(n). The work sites Ar are also called "cells", that is, the factory Fa includes one or more cells. Each of the one or more work sites Ar(1), Ar(2), Ar(3), ..., Ar(n) is associated with one or more operation processes Pr(1), Pr(2), Pr(3), ..., Pr(n). That is, the work site Ar(m) is an area where the worker Hu performs the operation Op(m) included in the operation process Pr(m).

When it is necessary to distinguish between multiple workplaces Ar, they will be distinguished by adding suffixes such as "(1)", "(2)", "(3)", ..., "(n)" to the reference numerals, and when there is no particular need to distinguish between them, they will simply be referred to as "workplace Ar".

(About the work object)
In the work area Ar, one or more work objects Ob that are the targets of the work Op performed by the worker Hu are placed. For example, in the work area Ar(x), work objects Ob(x1), Ob(x2), Ob(x3), ..., Ob(xn) are placed.

The work object Ob is an entity other than the worker Hu, for example, a so-called "workpiece," or, for example, a tool used by the worker Hu when performing the work Op.

When it is necessary to distinguish between multiple work objects Ob, the reference numerals are followed by suffixes such as "(1)", "(2)", "(3)", ..., "(n)" to distinguish between them. When no particular distinction is required, they are simply referred to as "work objects Ob".

(About the workers)
In the factory Fa, there are one or more workers Hu who perform the work Op included in the work process Pr, for example, workers Hu(1), Hu(2), Hu(3), ..., Hu(n). The workers Hu may be identified, for example, by a worker ID attached to the top of the hat worn by the worker Hu. Specifically, the workers Hu(1) and Hu(2) in the factory Fa may be identified by the worker ID(1) attached to the hat worn by the worker Hu(1) and the worker ID(2) attached to the hat worn by the worker Hu(2). In addition, a certain worker Hu(1) may be regarded as a work object Ob for another worker Hu(2).

When it is necessary to distinguish between multiple workers Hu, they will be distinguished by adding suffixes such as "(1)", "(2)", "(3)", ..., "(n)" to the reference numerals, and when there is no particular need to distinguish between them, they will simply be referred to as "worker Hu".

(About equipment and operation)
The factory Fa includes one or more devices 40(1), 40(2), 40(3), ... 40(n). The devices 40 include an input device 41 such as a sensor, and an output device 42 such as an actuator.

The output device 42 is a device whose output is controlled by the PLC 20. The worker Hu may use the output of the output device 42 when performing the work Op. Specifically, the worker Hu may use the operation Ac executed by the output device 42 to perform the work Op. That is, the worker Hu may use the output device 42 as a tool when performing the work Op. In other words, the output device 42 may be a tool used by the worker Hu when performing the work Op. For example, when the worker Hu performs the work Op(x) included in the work process Pr(x), the output device 42(x) may be used. Specifically, the worker Hu may perform the work Op(x) by using the operation Ac(x) executed by the output device 42(x).

The input device 41 is a sensor (detection device) that detects at least one state of the tools (e.g., the output device 42 and the tools 90 other than the output device 42 described below) used in the operation Op, the work object Ob, and the work area Ar. For example, the input device 41 detects the power consumption of the tools, the gripping of the tools, work object Ob, etc. by the worker Hu, the pressing of a button provided on the tools, work area Ar, work object Ob, etc. by the worker Hu, the opening and closing of the opening and closing mechanism provided on the tools, work area Ar, work object Ob, etc.

Specifically, the input device 41 may be, for example, a current detection sensor, and may detect the current flowing through the "tool 90(x): electric screwdriver" used by the worker Hu when performing the "task Op(x): screw tightening task". The input device 41 may also be, for example, a grip sensor, and may detect that the "tool 90(x): electric screwdriver" is gripped by the worker Hu. Furthermore, the input device 41 may be, for example, a start/end button, and may detect pressing by the worker Hu. The input device 41 may also be, for example, a photoelectric sensor, and may detect the opening and closing of an opening/closing mechanism provided on a tool, a work area Ar, a work object Ob, etc.

The input device 41 periodically outputs information indicating the detected state (hereinafter also referred to as "detection information Id") to the PLC 20 via the control network 50. In particular, the input device 41 periodically outputs to the PLC 20 the detection information Id including the "time at which at least one state of the tool used in the work Op, the work object Ob, and the work area Ar was detected (detection time Td)."

When it is necessary to distinguish between the multiple devices 40, the devices 40 are distinguished by adding suffixes such as "(1)", "(2)", "(3)", ..., "(n)" to the reference numerals, and when there is no particular need to distinguish between them, they are simply referred to as "devices 40". Similarly, when it is necessary to distinguish between the multiple input devices 41, the devices 40 are distinguished by adding suffixes such as "(1)", "(2)", "(3)", ..., "(n)" to the reference numerals, and when there is no particular need to distinguish between them, they are simply referred to as "input devices 41". Furthermore, when it is necessary to distinguish between the multiple output devices 42, the devices 42 are distinguished by adding suffixes such as "(1)", "(2)", "(3)", ..., "(n)" to the reference numerals, and when there is no particular need to distinguish between them, they are simply referred to as "output devices 42".

Furthermore, when it is necessary to distinguish between multiple operations Ac, they are distinguished by adding suffixes such as "(1)", "(2)", "(3)", ..., "(n)" to the reference numerals, and when there is no particular need to distinguish between them, they are simply referred to as "operations Ac".

In addition, the detection information Id indicating "at least one state of the tool (e.g., output device 42(x) or tool 90(x)) used in the operation Op(x), the work object Ob(x), and the work area Ar(x)" may be specifically referred to as "detection information Id(x)."

(About tools)
In addition to the output device 42, the factory Fa may include a tool 90 (not shown) other than the output device 42. When performing an operation Op, the worker Hu may use the tool 90. Specifically, the worker Hu may perform the operation Op by using an action Ac performed by the tool 90. For example, when the worker Hu performs an operation Op(x) included in a work process Pr(x), the tool 90(x) may be used. Specifically, the worker Hu may perform the operation Op(x) by using an action Ac(x) performed by the tool 90(x). As an example, the worker Hu may perform "operation Op(x): screw tightening operation" by using "action Ac(x): rotation operation" performed by "tool 90(x): electric screwdriver".

As described above, the state of the tool 90 (e.g., the state of being used by the worker Hu) is detected by the input device 41, and the input device 41 periodically outputs the detected "tool 90 state" to the PLC 20 via the control network 50. That is, when the input device 41 detects the "tool 90 state", it periodically outputs detection information Id including the detected "tool 90 state" and the time when the "tool 90 state" was detected (detection time Td) to the PLC 20.

(About the workshop images)
The work place image Ic is a captured image of the work place Ar, and each of the multiple work place images Ic is a captured image of each of the multiple work places Ar. For example, the work place image Ic is a captured image of a worker Hu performing a task Op in the work place Ar from a substantially frontal view, and the work place image Ic is a captured image generated for each work place Ar.

The multiple work area images Ic may be distinguished from each other by the work area Ar captured in each of the multiple work area images Ic. For example, the multiple work area images Ic may be distinguished from each other by associating each of the multiple work area images Ic with the work area Ar captured in each of the work area images Ic.

That is, the work area image Ic captured in the work area Ar(1) may be referred to as work area image Ic(1), and the work area image Ic captured in the work area Ar(2) may be referred to as work area image Ic(2). Similarly, the work area image Ic captured in the work area Ar(n) may be referred to as work area image Ic(n).

When there is no need to distinguish between multiple work area images Ic based on the work area Ar captured in each of them, they are simply referred to as "work area images Ic."

(About the whole image)
While the work area image Ic is an image of each work area Ar, the overall image Ia is an image in which multiple work areas Ar are captured, for example, an image in which the entire factory Fa is captured as viewed from the ceiling of the factory Fa.

First, the following describes an overview of the control system 1 including the information processing device 10 using FIG. 2.

(Overview of the system including the information processing device)
2 is a diagram showing an overall overview of the control system 1 and the like including the information processing device 10. In the environment described above, the information processing device 10 estimates with high accuracy, from a workplace image Ic obtained by capturing an image of a workplace Ar, the content of an operation Op being performed by a worker Hu captured in the workplace image Ic.

The cell camera 30 is installed in the workplace Ar (i.e., the cell) and generates a workplace image Ic. For example, a cell camera 30 is installed for each workplace Ar, and each of the multiple cell cameras 30 is installed in a position for each workplace Ar where it can capture an image of a worker Hu performing an operation Op in the workplace Ar from approximately the front. The cell camera 30 outputs the generated workplace image Ic to the information processing device 10, and in particular outputs the workplace image Ic to the information processing device 10 together with the image capture time Tp, which is the time when the workplace Ar was captured.

When distinguishing between the cell cameras 30 installed in each work area Ar in the factory Fa, each of the multiple cell cameras 30 may be distinguished by associating each of the multiple cell cameras 30 with the work area Ar in which the cell camera 30 is installed. For example, the cell camera 30 installed in work area Ar(1) may be described as cell camera 30(1), and the cell camera 30 installed in work area Ar(2) may be described as cell camera 30(2). Similarly, the cell camera 30 installed in work area Ar(n) may be described as cell camera 30(n).

The ceiling camera 80 is installed, for example, on the ceiling of the factory Fa and generates an overall image Ia looking down diagonally from above on the multiple work areas Ar, for example, generating an overall image Ia capturing the entire factory Fa using a single ceiling camera 80. However, it is not essential that a single ceiling camera 80 is installed on the ceiling of the factory Fa, and each of multiple ceiling cameras 80 installed on the ceiling of the factory Fa may generate an overall image Ia capturing multiple work areas Ar in the factory Fa. In other words, it is sufficient for the ceiling camera 80 to be able to generate an overall image Ia capturing multiple work areas Ar.

In the following explanation, an example will be described in which one ceiling camera 80, which is a wide-area imaging camera, is installed on the ceiling of the factory Fa, and the ceiling camera 80 generates an overall image Ia by capturing an image of the entire factory Fa from above. The ceiling camera 80 outputs the generated overall image Ia to the information processing device 10, and in particular outputs the overall image Ia to the information processing device 10 together with the overall time Ta, which is the time when the entire factory Fa was captured (i.e., the time when the overall image Ia was generated).

(Prediction of work content through image analysis of workplace images)
The information processing device 10 is included in the control system 1 and is communicatively connected to the cell camera 30 via, for example, a USB (Universal Serial Bus) cable. The information processing device 10 acquires a work area image Ic of the work area Ar from the cell camera 30, performs image analysis on the acquired work area image Ic, and predicts the content of the work Op performed by the worker Hu captured in the work area image Ic on the work object Ob. The information processing device 10 predicts "the content of the work Op performed by the worker Hu captured in the work area image Ic on the work object Ob" from the work area image Ic using a HOI (Human Object Interaction) detection algorithm. In particular, the information processing device 10 acquires from the cell camera 30 the capture time Tp, which is the time when the work area image Ic was generated, together with the work area image Ic, and executes the following predictions from the acquired work area image Ic and the capture time Tp. That is, the information processing device 10 predicts "the content of the work Op being performed by the worker Hu on the work object Ob in the work place image Ic at the imaging time Tp" The "content of the work Op being performed by the worker Hu on the work object Ob in the work place image Ic" predicted from the work place image Ic, particularly the "content of the work Op being performed by the worker Hu on the work object Ob in the work place Ar at the imaging time Tp" is also referred to as a first prediction Pf.

For example, the information processing device 10 performs image analysis on a workplace image Ic(x) captured of the workplace Ar(x) to predict the content of the work Op(x) being performed by a worker Hu(x) on a work object Ob(x) in the workplace Ar(x). In particular, the information processing device 10 predicts, from the workplace image Ic(x), "the content of the work Op(x) being performed by a worker Hu(x) on a work object Ob(x) in the workplace Ar(x) at the imaging time Tp when the workplace image Ic(x) is captured." The "content of the work Op(x) being performed by a worker Hu(x) on a work object Ob(x) in the workplace Ar(x) at the imaging time Tp when the workplace image Ic(x) is captured," predicted from the workplace image Ic(x), may be referred to as "first prediction Pf(x)."

(Prediction of worker presence or absence through image analysis of the entire image)
The information processing device 10 is included in the control system 1 and is communicatively connected to the ceiling camera 80 via, for example, a USB (Universal Serial Bus) cable. The information processing device 10 acquires an overall image Ia captured from the ceiling camera 80 overlooking the entire factory Fa, and performs image analysis on the acquired overall image Ia to predict whether the worker Hu is in the work area Ar. The information processing device 10 predicts, from the overall image Ia, "whether the worker Hu is in the work area Ar captured in the overall image Ia" using, for example, a movement line detection algorithm (object detection algorithm) such as YOLO. In particular, the information processing device 10 acquires, from the ceiling camera 80, the overall image Ia together with the overall time Ta, which is the time when the overall image Ia was generated, and executes the following prediction from the acquired overall image Ia and the overall time Ta. That is, the information processing device 10 predicts "whether the worker Hu is in the work area Ar (particularly, the work area Ar captured in the work area image Ic) at the overall time Ta".

Therefore, the information processing device 10 can predict "whether or not a worker Hu is present in the workplace Ar (particularly, the workplace Ar captured in the workplace image Ic) at the overall time Ta corresponding to the imaging time Tp."

The "presence or absence of worker Hu in the workplace Ar" predicted from the overall image Ia, and in particular the prediction of "whether or not worker Hu is in the workplace Ar at the overall time Ta corresponding to the imaging time Tp", is also referred to as the presence or absence prediction Pa.

For example, the information processing device 10 predicts "whether worker Hu(x) is in the workplace Ar(x) captured in the workplace image Ic(x) generated by the cell camera 30(x)" by image analysis of the overall image Ia generated by the ceiling camera 80. In particular, the information processing device 10 predicts "whether worker Hu(x) is in the workplace Ar(x) at the overall time Ta corresponding to the 'imaging time Tp at which the workplace image Ic(x) was captured'" from the overall image Ia. The prediction of "whether worker Hu(x) is in the workplace Ar(x) at the overall time Ta corresponding to the 'imaging time Tp at which the workplace image Ic(x) was captured'" predicted from the overall image Ia is sometimes referred to as "presence prediction Pa(x)".

(Prediction of work content using detection information)
Furthermore, the information processing device 10 acquires, from a PLC (Programmable Logic Controller) 20, process information Ip generated from operation results La indicating the contents and results of operations Ac executed by devices 40 whose operations Ac are controlled by the PLC 20. The process information Ip includes, for example, at least one of "the time when the devices 40 executed the operations Ac" and "the time when the execution of the operations Ac by the devices 40 was detected by the input device 41 or the like" as a detection time Td.

The information processing device 10 grasps the state of the output device 42 (e.g., the operation state including the content and result of the operation Ac) from the process information Ip acquired from the PLC 20, and predicts the content of the work Op being performed by the worker Hu using the output device 42 from the grasped state. In particular, the information processing device 10 grasps the "state of the output device 42 at 'detection time Td corresponding to the image capture time Tp'" from the process information Ip including the detection time Td. Then, the information processing device 10 predicts the "content of the work Op being performed by the worker Hu using the output device 42 at 'detection time Td corresponding to the image capture time Tp'" from the grasped "state of the output device 42 at 'detection time Td corresponding to the image capture time Tp'".

However, the PLC 20 may grasp the state of the output device 42 from the output of the equipment 40, and predict the content of the work Op being performed by the worker Hu using the output device 42 from the grasped state. The PLC 20 may then output process information Ip including the predicted content of the work Op to the information processing device 10. In other words, it may be the PLC 20 or the information processing device 10 that predicts the content of the work Op being performed by the worker Hu using the output device 42 from the state of the output device 42. It is sufficient for the information processing device 10 to acquire or generate the "content of the work Op being performed by the worker Hu using the output device 42" predicted from the state of the output device 42.

Here, the process information Ip is information indicating the state of the output device 42, etc., generated from "the output of the equipment 40 controlled by the PLC 20, and the control contents of the output device 42 by the PLC 20", etc. Therefore, the PLC 20 can also be considered as a detection device that detects the state of the output device 42 from "the output of the output device 42 obtained as feedback from the output device 42, and the control contents of the output device 42", etc. In addition, the process information Ip can also be considered as a type of "detection information Id", which is information indicating "the state of at least one of the tools (e.g., the output device 42 and the tool 90) used in the work Op, the work object Ob, and the work area Ar".

Furthermore, the information processing device 10 acquires detection information Id from the PLC 20, which indicates "at least one state of the tools (e.g., the output device 42 and the equipment 90) used in the work Op, the work object Ob, and the work area Ar" detected by the input device 41.

In addition, the information processing device 10 may obtain the detection information Id indicating the detected state of the input device 41 directly from the input device 41 without going through the PLC 20.

The detection information Id includes a detection time Td, which is the time when the input device 41 detects "at least one state of the tool used in the work Op, the work object Ob, and the work area Ar."

The information processing device 10 grasps the state of at least one of the tools used in the work Op, the work object Ob, and the work area Ar based on the detection information Id acquired from the PLC 20, and predicts the content of the work Op being performed by the worker Hu from the grasped state.

However, it may be the PLC 20 that grasps the state of at least one of the tools, the work object Ob, and the work area Ar using the detection information Id acquired from the input device 41, and predicts the content of the work Op being performed by the worker Hu from the grasped state. The PLC 20 may then output process information Ip including the predicted content of the work Op to the information processing device 10. In other words, it may be the PLC 20 or the information processing device 10 that predicts the content of the work Op being performed by the worker Hu from the state of at least one of the tools, the work object Ob, and the work area Ar. It is sufficient for the information processing device 10 to acquire or generate the "content of the work Op being performed by the worker Hu" predicted from the state of at least one of the tools, the work object Ob, and the work area Ar.

That is, the information processing device 10 acquires from the PLC 20 the detection information Id (including the process information Ip) indicating "at least one state of the tool used in the work Op, the work object Ob, and the work area Ar." The information processing device 10 predicts "the content of the work Op being performed by the worker Hu" from the acquired detection information Id, and in particular, predicts "the content of the work Op being performed by the worker Hu at the 'detection time Td corresponding to the image capture time Tp'." However, the information processing device 10 may acquire from the PLC 20 the "content of the work Op being performed by the worker Hu" predicted from the detection information Id by the PLC 20 together with the detection information Id or instead of the detection information Id. For example, the information processing device 10 may acquire from the PLC 20 the "content of the work Op being performed by the worker Hu at the 'detection time Td corresponding to the image capture time Tp'" together with the detection information Id or instead of the detection information Id.

The "contents of the work Op being performed by the worker Hu" predicted from the detection information Id (including the process information Ip), and in particular the "contents of the work Op being performed by the worker Hu at the 'detection time Td corresponding to the imaging time Tp'", is also referred to as the work prediction Po.

For example, the information processing device 10 predicts the content of the work Op(x) being performed by a worker Hu(x) in a workplace Ar(x) from the following detection information Id(x) (including process information Ip(x)). That is, the information processing device 10 predicts the content of the work Op(x) from the detection information Id(x) indicating "the tools provided in the workplace Ar(x), the work object Ob(x) placed in the workplace Ar(x), and at least one state of the workplace Ar(x)." The "tools provided in the workplace Ar(x) (for example, at least one of the output device 42(x) and the tool 90(x))" are the tools used by the worker Hu(x) when performing the work Op(x). In particular, the information processing device 10 predicts "the content of the work Op(x) being performed by the worker Hu(x) on the work object Ob(x) in the work area Ar(x) at the 'detection time Td' corresponding to the imaging time Tp'" from the process information Ip(x). The "content of the work Op(x) being performed by the worker Hu(x) at the 'detection time Td' corresponding to the imaging time Tp'" predicted from the detection information Id(x) (including the process information Ip(x)) may be referred to as the "work prediction Po(x)."

(Verifying predicted work content and outputting estimated results)
The information processing device 10 verifies the "contents of the work Op being performed by the worker Hu on the work object Ob" predicted from the work place image Ic, using at least one of the following two predictions. That is, the information processing device 10 verifies the contents predicted from the work place image Ic, using at least one of the "presence or absence of the worker Hu" predicted from the overall image Ia, and the "contents of the work Op being performed by the worker Hu" predicted from the detection information Id. That is, the information processing device 10 verifies the accuracy of the first prediction Pf, using at least one of the presence or absence prediction Pa and the work prediction Po, and in particular, verifies the accuracy of the first prediction Pf(x), using at least one of the presence or absence prediction Pa(x) and the work prediction Po(x).

The first prediction Pf(x) is a prediction of "the content of the work Op(x) being performed by a worker Hu(x) on a work object Ob(x) in the workplace Ar(x)" predicted by the information processing device 10 from the workplace image Ic(x). The "first prediction Pf(x)" is, for example, "the content of the work Op(x) being performed by a worker Hu(x) on a work object Ob(x) in the workplace Ar(x) at the imaging time Tp when the workplace image Ic(x) is captured" predicted from the workplace image Ic(x).

The presence/absence prediction Pa(x) is a prediction of "whether or not a worker Hu is present in the workplace Ar(x)" predicted by the information processing device 10 from the overall image Ia. The presence/absence prediction Pa(x) is, for example, a prediction of "whether a worker Hu(x) is in the workplace Ar(x) at the overall time Ta corresponding to the 'imaging time Tp at which the workplace image Ic(x) was captured'" predicted from the overall image Ia.

The work prediction Po(x) is a prediction of "the content of the work Op(x) being performed by the worker Hu(x) at the workplace Ar(x)" predicted by the information processing device 10 from the detection information Id(x). The work prediction Po(x) is, for example, "the content of the work Op(x) being performed by the worker Hu(x) at the workplace Ar(x) at 'the detection time Td corresponding to the imaging time Tp'" predicted from the detection information Id(x).

For example, if the presence/absence prediction Pa(x) predicts that "worker Hu is present in the workplace Ar(x)", the information processing device 10 (particularly, the determination unit 150 described later) determines that "the first prediction Pf(x) is correct". More specifically, if the presence/absence prediction Pa(x) predicts that "worker Hu(x) is in the workplace Ar(x) at 'overall time Ta corresponding to the imaging time Tp'", the information processing device 10 determines that "the first prediction Pf(x) is correct". Then, the information processing device 10 (particularly, the estimation unit 160 described later) outputs the first prediction Pf(x) determined to be correct by the determination unit 150 to the outside as an estimation result.

For example, when the work Op(x) in the work prediction Po(x) and the work Op(x) in the first prediction Pf(x) match, the information processing device 10 (particularly, the determination unit 150 described later) determines that the "first prediction Pf(x) is correct." More specifically, the information processing device 10 checks whether the "contents of the work Op(x) at the 'detection time Td corresponding to the imaging time Tp'" in the presence/absence prediction Pa(x) match with the "contents of the work Op(x) at the imaging time Tp" in the first prediction Pf(x). Then, when the information processing device 10 confirms that the two match, it determines that the "first prediction Pf(x) is correct." Then, the information processing device 10 (particularly, the estimation unit 160 described later) outputs the first prediction Pf(x) determined to be correct by the determination unit 150 to the outside as an estimation result.

For example, when the presence/absence prediction Pa(x) predicts that "worker Hu is present in the workplace Ar(x)," the information processing device 10 (particularly the determination unit 150 described later) checks whether the work prediction Po(x) and the first prediction Pf(x) match. More specifically, when the presence/absence prediction Pa(x) predicts that "worker Hu(x) is in the workplace Ar(x) at 'total time Ta corresponding to the imaging time Tp,'" the information processing device 10 checks whether the work prediction Po(x) and the first prediction Pf(x) match. When the information processing device 10 confirms whether the "contents of work Op(x) at the detection time Td corresponding to the imaging time Tp" in the presence/absence prediction Pa(x) match with the "contents of work Op(x) at the imaging time Tp" in the first prediction Pf(x), the information processing device 10 determines that the first prediction Pf(x) is correct. Then, the information processing device 10 (particularly the estimation unit 160 described later) outputs the first prediction Pf(x) determined to be correct by the determination unit 150 to the outside as an estimation result.

(Master-slave control system)
In the control system 1, the devices 40 used to perform the work process Pr in the factory Fa are controlled by the PLC 20 as a line controller. That is, the control system 1 is constructed as a master-slave control system in which the PLC 20 is the master and each of the devices 40 is a slave, and each of the devices 40 is communicatively connected to the PLC 20 via a network (control network 50). The PLC 20 is called a "master" in the sense that it manages data transmission via the control network 50. The terms "master" and "slave" are defined with a focus on the control function of data transmission on the control network 50, and there is no particular limitation on the type of information transmitted and received between the devices.

The PLC 20 is a control device (controller) that controls the entire control system 1, and is connected to each of the multiple devices 40 so as to be able to communicate with them. The PLC 20 acquires information from each of the multiple devices 40 as input devices 41 (measurement devices) as input data. The PLC 20 executes arithmetic processing using the acquired input data according to a user program previously incorporated. The PLC 20 executes the arithmetic processing to determine the control content for the control system 1, and for example, determines the control content for each of the multiple devices 40 as output devices 42 such as actuators. The PLC 20 then outputs control data corresponding to the determined control content to each of the multiple devices 40 (particularly, the output device 42). The PLC 20 repeatedly acquires input data from each of the multiple devices 40 (specifically, the input device 41) and acquires control data for each of the multiple devices 40 (specifically, the output device 42) at a predetermined period (control period). For example, a display unit and an operation unit (not shown) may be connected to the PLC 20. The display unit is composed of a liquid crystal panel capable of displaying images, and the operation unit is typically composed of a touch panel, keyboard, mouse, etc.

The device 40 is a slave in the control system 1, which is a master-slave control system with the PLC 20 as the master. The device 40 is an input device 41 that repeatedly transmits input data to the PLC 20 at a predetermined control period, or an output device 42 that repeatedly receives control data from the PLC 20 at a predetermined control period and operates according to the received control data.

The equipment 40 (particularly, the input device 41) is, for example, a sensor that transmits detection results, etc., as input data to the PLC 20. For example, the input device 41 may be a photoelectric sensor, a barcode reader that transmits the read results, or even an inspection machine (tester) that transmits the inspection results.

The device 40 (specifically, the output device 42) may be a robot or the like that serves as an output device for performing operations such as screw tightening and picking, or may be any type of actuator.

In addition, the device 40 may be a PT (Programmable Terminal) to which multiple input devices 41 are connected.

The control network 50 transmits various data received by the PLC 20 or transmitted by the PLC 20, and typically may use various types of industrial Ethernet (registered trademark), and may also be referred to as a field network. Known industrial Ethernet (registered trademark) include, for example, EtherCAT (registered trademark), Profinet IRT, MECHATROLINK (registered trademark)-III, Powerlink, SERCOS (registered trademark)-III, and CIP Motion, and any of these may be adopted. Furthermore, a field network other than the industrial Ethernet (registered trademark) may also be used. For example, if motion control is not performed, DeviceNet, CompoNet/IP (registered trademark), etc. may be used.

In this embodiment, a control system 1 is described in which data frames are transferred sequentially over a control network 50, thereby transmitting and receiving data between a PLC 20 (master) and a device 40 (slave). That is, data frames are transferred sequentially over the control network 50 at a predetermined control period, thereby repeatedly transmitting and receiving data between the PLC 20 and the device 40 at each control period. Data may be transmitted and received between multiple devices 40, i.e., multiple slaves, by transferring data frames sequentially over the control network 50.

The information processing device 10 is included in the control system 1, but the information processing device 10 does not have to be a slave in the control system 1.

In control system 1, which is a master-slave control system in which device 40 is a slave, PLC 20, which is the master, receives operation results La and detection information Id from device 40, which is a slave, for example, repeatedly at predetermined control intervals.

(Regarding process information)
In the control system 1, which is a master-slave control system in which the device 40 is a slave, the PLC 20, which is the master, receives an operation result La from the device 40, which is a slave, for example, repeatedly at a predetermined control period. The operation result La is information indicating "the content and result of the operation Ac performed by the device 40." That is, the device 40 transmits the operation result La, which indicates the content and result of the operation Ac actually performed when performing the work process Pr, to the PLC 20 at a predetermined period. For example, the device 40(x) transmits the operation result La(x), which indicates the content and result of the operation Ac(x) performed when performing the work process Pr(x), to the PLC 20 at a control period.

The PLC 20 acquires, for example, the measurement result, which is the result of the measurement operation performed by the equipment 40 as the input device 41 (measurement device), as the operation result La of the equipment 40. Furthermore, if the equipment 40 is an inspection machine, the PLC 20 acquires the result of the inspection operation performed by the equipment 40, for example, the inspection result, such as "whether the inspection criteria were met or not met," as the operation result La of the equipment 40. Furthermore, the PLC 20 acquires, for example, the result of the output operation performed by the equipment 40 as the output device 42, as the operation result La of the equipment 40. If the equipment 40 is a robot that performs screw tightening, picking, etc., the PLC 20 acquires the operation result La, such as the number of screw tightenings, the picking result (picking success or picking error), etc., as the operation result La of the equipment 40.

The PLC 20 repeatedly receives operation results La from the equipment 40 at a predetermined cycle, which indicate the content and results of the operations Ac actually performed by the equipment 40 when carrying out the work process Pr, and transmits, i.e., transfers, the received operation results La to the information processing device 10 as process information Ip. The PLC 20 also transmits information generated using the operation results La repeatedly received at a predetermined cycle from the equipment 40 to the information processing device 10 as process information Ip.

Furthermore, the PLC 20 may transmit the operation result La repeatedly received from the device 40 at a predetermined cycle as process information Ip to the outside of the control system 1. For example, the PLC 20 may transmit the operation result La repeatedly received from the device 40 at a predetermined cycle as process information Ip to an in-house LAN (Local Area Network) shown in FIG. 2 connected to a MES (Manufacturing Execution System) or the like.

The information processing device 10 identifies the operation start time Tms, operation completion time Tme, and operation period Da of the operation Ac performed by the equipment 40 when carrying out the work process Pr from the process information Ip (operation result La) acquired from the PLC 20. Then, the information processing device 10 performs various judgments on the operation Ac using the identified operation start time Tms, operation completion time Tme, operation period Da, and a predetermined operation standard Sa corresponding to the operation Ac.

However, the above-mentioned judgment on the operation result La (particularly, operation Ac) included in the process information Ip may be performed by PLC 20, and PLC 20 may transmit the result of the above-mentioned judgment to the information processing device 10 by including it in the process information Ip or instead of the process information Ip.

In the above description, the operation start time Tms is the time when the device 40 used in the work process Pr starts to perform the operation Ac when carrying out the work process Pr, and the operation completion time Tme is the time when the operation Ac is completed. The operation period Da is the period from the operation start time Tms to the operation completion time Tme.

As described above, the process information Ip can also be considered as information indicating "the state of the tools (particularly, the output device 42) used in the work Op." The process information Ip can also be considered as information indicating, for example, "the state of the tools (particularly, the output device 42) used in the work Op at 'the detection time Td corresponding to the imaging time Tp.'" Therefore, the process information Ip can be understood as a type of detection information Id indicating "the state of at least one of the tools (e.g., the output device 42 and the tool 90), the work object Ob, and the work area Ar used in the work Op."

(Regarding detection information)
In the control system 1, the PLC 20 receives detection information Id from the slave device 40 (particularly, the input device 41) at a predetermined control cycle, for example. The detection information Id is information indicating "at least one state of the tool used in the work Op (e.g., the output device 42 and an instrument 90 other than the output device 42 described later), the work object Ob, and the work area Ar" detected by the input device 41. That is, the device 40 (particularly, the input device 41) transmits "at least one state of the tool used in the work Op by the worker Hu, the work object Ob, and the work area Ar" to the PLC 20 at a predetermined cycle. The detection information Id includes information indicating the time (i.e., the detection time Td) at which the input device 41 detected "at least one state of the tool used in the work Op, the work object Ob, and the work area Ar".

The PLC 20 acquires, for example, information indicating the magnitude of the current flowing through the appliance 90(x) (such as the power consumption of the appliance 90(x)) from the equipment 40(y) serving as the input device 41(y) as detection information Id(x) about the appliance 90(x).

The PLC 20 acquires, for example, information indicating whether the tool 90(x) is held by the worker Hu from the equipment 40(y) serving as the input device 41(y), as detection information Id(x) about the tool 90(x).

The PLC 20 acquires, for example, information indicating the opening and closing of the opening and closing mechanism of the work object Ob(x) from equipment 40(y) serving as an input device 41(y), as detection information Id(x) for the work object Ob(x).

The PLC 20 acquires, for example, information indicating whether a button on the work object Ob(x) has been pressed from the equipment 40(y) serving as the input device 41(y), as detection information Id(x) about the work object Ob(x).

The PLC 20 acquires, for example, information indicating the opening and closing of the opening and closing mechanism provided in the workplace Ar(x) from equipment 40(y) as an input device 41(y), as detection information Id(x) for the workplace Ar(x).

The PLC 20 acquires, for example, information indicating whether a button provided in the workplace Ar(x) has been pressed from equipment 40(y) serving as an input device 41(y), as detection information Id(x) for the workplace Ar(x).

The PLC 20 repeatedly receives the detection information Id from the input device 41 at a predetermined period and transmits, i.e., transfers, the received detection information Id to the information processing device 10.

Furthermore, the PLC 20 may transmit the detection information Id repeatedly received from the input device 41 at a predetermined cycle to the outside of the control system 1, similar to the process information Ip. For example, the PLC 20 may transmit the detection information Id repeatedly received from the input device 41 at a predetermined cycle to an in-house LAN (Local Area Network) shown in FIG. 2 connected to a MES (Manufacturing Execution System) or the like.

The information processing device 10 identifies "at least one state of the tools used by the worker Hu in the work Op, the work object Ob, and the work area Ar" from the detection information Id acquired from the PLC 20, and the detection time Td, which is the time when the input device 41 detected that state. Then, the information processing device 10 predicts the content of the work Op being performed by the worker Hu at the detection time Td from the identified detection time Td and "at least one state of the tools used by the worker Hu in the work Op, the work object Ob, and the work area Ar". In other words, the information processing device 10 predicts the content of the work Op being performed by the worker Hu at the detection time Td from the detection information Id acquired from the PLC 20.

However, the process of predicting the content of the work Op being performed by the worker Hu at the detection time Td from the detection information Id may be performed by the PLC 20. In this case, the PLC 20 may transmit the prediction result to the information processing device 10 by including it in the detection information Id or instead of the detection information Id.

(For systems and devices other than master-slave control systems)
2 shows an in-house LAN system, other network systems, etc., in addition to the control system 1 as a master-slave control system. The in-house LAN is connected to a process information DB (Database) also called MES. The process information DB stores information indicating "standard operations that should be performed by the equipment 40 used to perform the work process Pr" as an operation standard Sa.

In the example shown in FIG. 2, an event management device 60 that monitors and manages various events that occur in the factory Fa is connected to the process information DB as the MES via an in-house LAN. However, it is not essential that the event management device 60 is connected to the process information DB via an in-house LAN, and the event management device 60 does not have to be provided.

Furthermore, the process information DB is connected to the PLC 20 via an in-house LAN. Although not shown, the process information DB and the information processing device 10 may be connected. In addition to the MES, the in-house LAN may also be connected to an ERP (Enterprise Resources Planning), WMS (Warehouse Management System), etc. (not shown).

In FIG. 2, the process information DB is connected to a video storage server, etc., via an "other network" different from the control network 50 and the in-house LAN. The information processing device 10 is connected to the video storage server, etc., via the other network, and the work site image Ic, the overall image Ia, the contents of the work Op estimated by the information processing device 10, and support information, etc. transmitted from the information processing device 10 are stored in the video storage server, etc. In addition, an external device 70 realized by a PC (Personal Computer) or the like is connected to the video storage server, etc. The external device 70 displays, for example, the work site image Ic, the overall image Ia, the contents of the work Op estimated by the information processing device 10, and support information, etc., and performs visualization of the process information Ip, etc. That is, the external device 70 displays a list of information necessary for improving the work process Pr, and displays information indicating the work process Pr that is a bottleneck, the date and time of an error that occurred in the work process Pr, etc., in association with the corresponding work site image Ic.

As described above, the cell camera 30 captures an image of the workplace Ar to generate a workplace image Ic, and transmits the generated workplace image Ic to the information processing device 10 via a communication cable, such as a USB (Universal Serial Bus) cable.

The ceiling camera 80 also captures an overhead image of the entire factory Fa to generate an overall image Ia, and transmits the generated overall image Ia to the information processing device 10 via a communication cable, such as a USB cable.

The information processing device 10 may be, for example, a data extraction device realized by a PC or the like, which combines the process information Ip acquired from the PLC 20 and the work area image Ic acquired from the cell camera 30 to enable efficient use of both. The information processing device 10 visualizes the work process Pr by combining the process information Ip, including the "operation result La showing the content and result of the actual operation Ac of the multiple devices 40 in the factory Fa," with the "work area image Ic" and the "content of the work Op estimated from the work area image Ic." The information processing device 10 visualizes the work process Pr by combining the process information Ip acquired from the PLC 20, the work area image Ic acquired from the cell camera 30, and the content of the work Op estimated from the work area image Ic, and also analyzes the work process Pr efficiently and precisely.

For example, the information processing device 10, in addition to visualizing the process information Ip, extracts the work process Pr that is a bottleneck, and combines a workplace image Ic that captures the implementation status of the extracted work process Pr with the process information Ip of the extracted work process Pr. Therefore, for example, it becomes easy for the user to identify the cause of the bottleneck and the cause of the defect for the work process Pr that is a bottleneck, and the user can efficiently carry out improvement work for the work process Pr. In addition, the information processing device 10 can also be used for traceability when a defect occurs.

In addition, the information processing device 10 can improve the accuracy of both the analysis of the workplace image Ic and the analysis of the process information Ip by combining the workplace image Ic and each of the process information Ip of the multiple devices 40. For example, the information processing device 10 uses the process information Ip (operation result La) acquired from the PLC 20 to identify the operation start time Tms at which the device 40 starts the operation Ac and the operation completion time Tme at which the operation Ac is completed. The information processing device 10 performs an analysis on the workplace image Ic captured during the operation period Da, which is the period from the operation start time Tms to the operation completion time Tme, and can associate the operation Ac of the device 40 during the operation period Da with the work Op of the worker Hu with high accuracy.

Furthermore, the information processing device 10 acquires a motion standard Sa indicating "standard motion that should be performed by the equipment 40 used to perform the work process Pr" from the process information DB. Then, the information processing device 10 uses the acquired motion standard Sa to perform a judgment on "actual motion Ac performed by the equipment 40 used to perform the work process Pr". The information processing device 10 may perform an analysis on a work site image Ic capturing the implementation status of the work process Pr corresponding to the motion Ac that is determined to be different from the standard motion indicated by the motion standard Sa.

(Estimation of work content)
The information processing device 10 analyzes the workplace image Ic to predict the content of the work Op performed by the worker Hu on the work object Ob. In particular, the information processing device 10 predicts "the content of the work Op(x) performed by the worker Hu(x) on the work object Ob(x) in the workplace Ar(x) at the image capturing time Tp when the workplace image Ic(x) is captured" from the workplace image Ic(x) capturing the workplace Ar(x). The work Op performed by the worker Hu in the workplace Ar is generally a work Op performed on the work object Ob. Therefore, the work Op can be regarded as an interaction between the worker Hu and the work object Ob for a combination (pair) of the worker Hu and the work object Ob.

Here, conventionally, methods have been researched for detecting a human object interaction (HOI), which is an interaction between a human captured in the workplace image Ic and an object captured in the workplace image Ic, from the workplace image Ic. For example, a HOI detection technique is known that localizes both the human and the object and identifies the interaction between the two.

The information processing device 10 can use an HOI detection algorithm to estimate the content of the work Op that the worker Hu is performing on the work object Ob. That is, the information processing device 10 can predict the content of the work Op that the worker Hu is performing on the work object Ob for a "pair of worker Hu and work object Ob" captured in the work area image Ic, by using an HOI detection algorithm that detects the HOI.

However, it is difficult to say that the accuracy of predictions using the HOI detection algorithm is sufficiently high. In other words, it is difficult to say that the prediction accuracy of the HOI detection algorithm, which predicts from the workplace image Ic the content of the work Op being performed by the worker Hu captured in the workplace image Ic on the work object Ob captured in the workplace image Ic, is sufficiently high. In particular, it is known that when the following phenomena occur in the workplace image Ic, the accuracy of HOI detection for that workplace image Ic generally tends to decrease.

That is, it is known that the accuracy of HOI detection for a work area image Ic in which an obstacle or the like in the foreground hides all or part of at least one of the worker Hu and the work object Ob, i.e., "occlusion," generally tends to decrease.

In addition, it is known that when the size (scale) of at least one of the worker Hu and the work object Ob captured in the work area image Ic is (extremely) small or large, the accuracy of HOI detection for that work area image Ic generally tends to decrease.

It is known that the accuracy of HOI detection for a work area image Ic where "truncation" occurs, in which all or part of at least one of the worker Hu and the work object Ob is not captured because it goes outside the imaging range, tends to generally decrease.

It is known that when at least one of the worker Hu and the work object Ob is wholly or partially transparent, the accuracy of HOI detection for the work area image Ic generally tends to decrease.

Furthermore, it is known that the accuracy of HOI detection for a work area image Ic in which a work object Ob is captured at an angle, size, etc. that is not normally captured tends to generally decrease. For example, it is known that if the angle (rotation), size, etc. of the work object Ob captured in the work area image Ic is of a rare type, the accuracy of HOI detection for that work area image Ic generally tends to decrease.

It is also known that if the lighting at the time of capturing an image is too bright or too dark, the accuracy of HOI detection for that work area image Ic generally tends to decrease.

In the following description, an example will be described in which the information processing device 10 detects the worker Hu and the work object Ob from the work site image Ic using a bounding box for HOI detection (i.e., prediction of the contents of the work Op). However, it is not essential for the information processing device 10 to use a bounding box to detect the worker Hu and the work object Ob. The information processing device 10 may use "Mask R-CNN" to execute a classification task (segmentation) of the worker Hu and the work object Ob for the work site image Ic. In addition, the information processing device 10 may detect the worker Hu and the work object Ob from the work site image Ic using an Attention mechanism used in deep learning, simultaneously with "Mask R-CNN". The information processing device 10 may improve the prediction accuracy of HOI detection by using "Mask R-CNN", an Attention mechanism, etc.

(Verification of HOI detection results)
As described above, it is difficult to say that the accuracy of prediction using the HOI detection algorithm is sufficiently high. Therefore, the information processing device 10 verifies the content (i.e., the first prediction Pf(x)) of the HOI (i.e., the operation Op) predicted using the HOI detection algorithm using at least one of the presence/absence prediction Pa(x) and the operation prediction Po(x).

The information processing device 10 improves the robustness and estimation accuracy of the estimation of the "contents of the work Op" by using the HOI detection algorithm in combination with a "movement line detection algorithm such as YOLO" and "detection information Id from the input device 41, etc." In other words, the information processing device 10 uses at least one of the presence/absence prediction Pa by the movement line detection algorithm and the work prediction Po predicted from the detection information Id as a noise filter for the first prediction Pf (HOI detection result) by the HOI detection algorithm.

The information processing device 10 is a work estimation device that estimates the "contents of the work Op" using at least one of a "traffic flow detection algorithm" and "detection information Id from an input device 41, etc." in addition to the HOI detection algorithm.

(Reduction of annotation costs)
The task prediction model 171 as a trained model used in the HOI detection algorithm is constructed by supervised learning on a set of teacher data Dt1 (learning data set DS1) in which the following labels are attached to the workplace image Ic or its feature information. That is, the task prediction model 171 is constructed by machine learning using a set of teacher data Dt1 in which the "contents of the task Op performed by the worker Hu captured in the workplace image Ic on the work object Ob captured in the workplace image Ic" are attached as labels. The "label" attached to the workplace image Ic or its feature information in the teacher data Dt1 is also referred to as an "annotation" or "metadata".

In this embodiment, the "content of the work Op being performed by the worker Hu captured in the workplace image Ic on the work object Ob captured in the workplace image Ic" that is attached as a label is a work prediction Po predicted from the detection information Id. In other words, the work prediction model 171 is constructed by supervised learning on a set of teacher data Dt1 in which "the 'content of the work Op being performed by the worker Hu' predicted from the detection information Id is attached as a label to the workplace image Ic or its feature information."

It is generally known that generating the large amount of training data Dt1 required to build a trained model is extremely time-consuming, and the so-called annotation cost is extremely high. For example, if one were to have a human determine "the content of the work Op that a worker Hu captured in the workplace image Ic is performing on a work object Ob captured in the workplace image Ic" for each of a large number of workplace images Ic, the cost required would be extremely high.

Therefore, in this embodiment, the "contents of the work Op being performed by the worker Hu" predicted from the detection information Id is used as a "label" to be attached to the workplace image Ic or its feature information. This makes it possible to reduce the cost (annotation cost) required to generate a large amount of training data Dt1, that is, the cost required to construct the work prediction model 171. Therefore, the information processing device 10 can realize highly accurate and robust estimation of the "contents of the work Op" using the work prediction model 171 constructed at low cost.

(Summary of the information processing device according to this embodiment)
The contents described above with reference to Fig. 2 can be summarized as follows: That is, the information processing device 10 is an operation estimation device that estimates the contents of an operation Op being performed by a worker Hu at a workplace Ar, and includes a first prediction unit 120 (prediction unit), a determination unit 150, and an estimation unit 160.

The first prediction unit 120 predicts "the content of the work Op being performed by the worker Hu on the work object Ob" from "the workplace image Ic or the feature information generated from the workplace image Ic" using the work prediction model 171. The work prediction model 171 is a trained model that receives as input "the workplace image Ic captured of the workplace Ar" or "the feature information generated from the workplace image Ic" and outputs the following information. In other words, the work prediction model 171 is a trained model that outputs "the content of the work Op being performed by the worker Hu captured in the workplace image Ic on the work object Ob captured in the workplace image Ic".

The determination unit 150 determines whether the prediction by the first prediction unit 120 is correct or not by using at least one of the "contents of the work Op predicted without using the work prediction model 171" and the "determination result of whether or not the worker Hu is present in the workplace Ar." In other words, the determination unit 150 determines whether the first prediction Pf is correct or not by using at least one of the work prediction Po and the presence/absence prediction Pa.

When the determination unit 150 determines that the prediction by the first prediction unit 120 is correct, the estimation unit 160 estimates that the content of the work Op predicted by the first prediction unit 120 is "the content of the work Op being performed by the worker Hu at the workplace Ar." In other words, when the estimation unit 160 verifies that the first prediction Pf is correct using at least one of the work prediction Po and the presence/absence prediction Pa, it outputs the first prediction Pf as an estimation result for "the content of the work Op being performed by the worker Hu at the workplace Ar."

According to the above configuration, the information processing device 10 determines whether the content of the work Op predicted using the work prediction model 171 (i.e., the first prediction Pf) is correct or not using at least one of the following two predictions. That is, the information processing device 10 determines whether the first prediction Pf is correct or not using at least one of the content of the work Op predicted without using the work prediction model 171 (work prediction Po) and the judgment result of the presence or absence of the worker Hu in the workplace Ar (presence prediction Pa).

Then, when the information processing device 10 determines that the prediction using the work prediction model 171 is correct, it estimates that the content of the work Op predicted using the work prediction model 171 is "the content of the work Op being performed by the worker Hu at the workplace Ar."

In other words, the information processing device 10 verifies the accuracy of the content of the work Op predicted using the work prediction model 171 using at least one of the content of the work Op predicted without using the work prediction model 171 and the determination result of whether or not a worker Hu is present in the workplace Ar.

Therefore, the information processing device 10 has the effect of being able to estimate with high accuracy the "contents of the work Op being performed by the worker Hu in the workplace Ar" from the workplace image Ic.

The information processing device 10 further includes a work prediction unit 142 (work detection unit) that predicts "the contents of the work Op being performed by the worker Hu in the work area Ar" from the detection information Id. The detection information Id is information that indicates "the tool used in the work Op, the work object Ob, and at least one state of the work area Ar" detected by the input device 41 (sensor).

When the content of the work Op predicted by the first prediction unit 120 (i.e., the first prediction Pf) matches the content of the work Op predicted by the work prediction unit 142 (i.e., the work prediction Po), the determination unit 150 determines that the prediction by the first prediction unit 120 is correct.

According to the above configuration, the information processing device 10 determines whether the content of the work Op predicted from the detection information Id of the input device 41 matches the content of the work Op predicted using the work prediction model 171. Then, when the information processing device 10 determines that the two match, it estimates that the content of the work Op predicted using the work prediction model 171 is "the content of the work Op being performed by the worker Hu at the workplace Ar."

In other words, the information processing device 10 verifies whether the content of the work Op predicted using the work prediction model 171 is correct or not based on the content of the work Op predicted from the detection information Id of the input device 41.

Therefore, the information processing device 10 has the effect of being able to estimate with high accuracy the "contents of the work Op being performed by the worker Hu in the workplace Ar" from the workplace image Ic.

The information processing device 10 further includes a presence/absence prediction unit 141, and when the presence/absence prediction unit 141 predicts that "the worker Hu is present in the workplace Ar", the determination unit 150 determines that the prediction of the first prediction unit 120 is correct. The presence/absence prediction unit 141 predicts "whether the worker Hu is present in the workplace Ar" using the presence/absence model 172 from the overall image Ia, or the workplace image Ic, or feature information generated from the overall image Ia or the workplace image Ic. The presence/absence model 172 is a trained model, and its input is (A) the overall image Ia of a plurality of workplaces Ar including the workplace Ar, or (B) the workplace image Ic, or (C) feature information generated from the overall image Ia or the workplace image Ic. The output of the presence/absence model 172 is information indicating "whether the worker Hu is present in the workplace Ar".

According to the above configuration, the information processing device 10 predicts "whether or not a worker Hu is present in the workplace Ar" using the presence/absence model 172. Then, when the information processing device 10 predicts that "a worker Hu is present in the workplace Ar", it estimates that the content of the work Op predicted using the work prediction model 171 is "the content of the work Op being performed by the worker Hu in the workplace Ar".

In other words, the information processing device 10 verifies the accuracy of the content of the work Op predicted using the work prediction model 171 based on the "presence or absence of the worker Hu in the work area Ar" predicted using the presence or absence model 172.

Therefore, the information processing device 10 has the effect of being able to estimate with high accuracy the "contents of the work Op being performed by the worker Hu in the workplace Ar" from the workplace image Ic.

The information processing device 10 may include both the work prediction unit 142 and the presence prediction unit 141. When the presence prediction unit 141 predicts that the worker Hu is present in the workplace Ar and the following condition is satisfied, the determination unit 150 determines that the prediction of the first prediction unit 120 is correct. In other words, when the content of the work Op predicted by the first prediction unit 120 matches the content of the work Op predicted by the work prediction unit 142, the determination unit 150 determines that the prediction of the first prediction unit 120 is correct.

According to the above configuration, the information processing device 10 predicts the content of the work Op from the detection information Id of the input device 41, and predicts the presence or absence of the worker Hu in the workplace Ar. Then, when the information processing device 10 predicts that the worker Hu is present in the workplace Ar and the following condition is satisfied, it determines that the prediction of the first prediction unit 120 is correct. In other words, when the content predicted from the detection information Id of the input device 41 matches the content predicted using the work prediction model 171, the information processing device 10 determines that the content predicted using the work prediction model 171 is correct.

In other words, the information processing device 10 verifies the accuracy of the content of the work Op predicted using the work prediction model 171 based on the content of the work Op predicted from the detection information Id of the input device 41 and the presence or absence of the worker Hu in the workplace Ar predicted using the presence or absence model 172.

Therefore, the information processing device 10 has the effect of being able to estimate with high accuracy the "contents of the work Op being performed by the worker Hu in the workplace Ar" from the workplace image Ic.

§2. Configuration Example The information processing device 10, the outline of which has been explained above, will now be explained in detail with reference to FIG.

FIG. 1 is a block diagram showing the main components of an information processing device 10. The information processing device 10 illustrated in FIG. 1 includes, as functional blocks, a first acquisition unit 110, a first prediction unit 120, a second acquisition unit 130, a second prediction unit 140, a determination unit 150, an estimation unit 160, and a storage unit 170, for example.

In addition to the above-mentioned functional blocks, the information processing device 10 may include the following units. That is, the information processing device 10 may include a management unit that controls the storage (saving) of the work area image Ic in a predetermined storage area (storage area), a measurement unit that measures the time (stay time) that the worker Hu stays in the work area Ar, an analysis unit that associates the process information Ip with the stay time, and the like. To ensure conciseness of the description, configurations of the information processing device 10 that are not directly related to this embodiment are omitted from the description and block diagram. However, the information processing device 10 may include these omitted configurations in accordance with the actual implementation situation.

The above-mentioned functional blocks of the information processing device 10 can be realized, for example, by the arithmetic unit reading out a program stored in a storage device (storage unit 170) realized by a ROM (read only memory), NVRAM (non-volatile random access memory), etc., into a RAM (random access memory) (not shown) or the like, and executing the program. Examples of devices that can be used as the arithmetic unit include a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), a DSP (Digital Signal Processor), an MPU (Micro Processing Unit), an FPU (Floating point number Processing Unit), a PPU (Physics Processing Unit), a microcontroller, or a combination of these.

First, the first acquisition unit 110, the first prediction unit 120, the second acquisition unit 130, the second prediction unit 140, the determination unit 150, and the estimation unit 160 will be described in detail below.

(Function blocks other than the memory unit)
The first acquisition unit 110 acquires from the cell camera 30 a workplace image Ic captured by the cell camera 30 of the workplace Ar, and outputs the acquired workplace image Ic to the first prediction unit 120.

The first prediction unit 120 performs image analysis on the work place image Ic captured by the cell camera 30, and predicts "the content of the work Op being performed by the worker Hu captured in the work place image Ic on the work object Ob captured in the work place image Ic." The first prediction unit 120 performs, for example, model-based image analysis on the work place image Ic, and predicts "the content of the work Op being performed by the worker Hu captured in the work place image Ic on the work object Ob captured in the work place image Ic."

The first prediction unit 120, which performs model-based image analysis, is realized as, for example, a neural network, and detects a task Op as an HOI by executing a HOI detection algorithm. Specifically, the first prediction unit 120 refers to the storage unit 170 to acquire a task prediction model 171. The task prediction model 171 is a trained model that receives the work place image Ic as input and outputs "the content of the task Op performed by the worker Hu captured in the work place image Ic on the work object Ob captured in the work place image Ic". The first prediction unit 120 uses the task prediction model 171 acquired by referring to the storage unit 170 to predict "the content of the task Op performed by the worker Hu captured in the work place image Ic on the work object Ob captured in the work place image Ic" from the work place image Ic. Hereinafter, "neural network" may be abbreviated as "NN".

For example, the first prediction unit 120 generates the following first prediction Pf(x) from the workplace image Ic(x) using the work prediction model 171. That is, the first prediction unit 120 generates the first prediction Pf(x), which is a prediction of "the content of the work Op(x) being performed by the worker Hu, who is captured in the workplace image Ic(x), on the work object Ob(x), which is captured in the workplace image Ic(x)."

In particular, the first prediction unit 120 performs the following prediction from the workplace image Ic(x) using the imaging time Tp, which is the time when the cell camera 30 captures the workplace Ar(x) and generates the workplace image Ic(x). That is, the first prediction unit 120 predicts from the workplace image Ic(x) the "contents of the work Op(x) being performed by the worker Hu(x) on the work object Ob(x) in the workplace Ar(x) at the imaging time Tp."

The first prediction unit 120 outputs the content of the work Op predicted from the work place image Ic using the work prediction model 171 to the determination unit 150 as a first prediction Pf. For example, the first prediction unit 120 outputs the first prediction Pf(x) generated from the work place image Ic(x) to the determination unit 150.

The second acquisition unit 130 acquires various pieces of information for generating the second prediction Ps from the outside, and outputs the acquired various pieces of information to the second prediction unit 140. The second prediction Ps is a prediction for verifying the first prediction Pf (whether it is correct or not), and includes at least one of a presence/absence prediction Pa and a work prediction Po. The second acquisition unit 130 illustrated in FIG. 1 includes an overall image acquisition unit 131 and a detection information acquisition unit 132.

The overall image acquisition unit 131 acquires an overall image Ia from the ceiling camera 80, which is captured by the ceiling camera 80 overlooking the entire factory Fa, and outputs the acquired overall image Ia to the second prediction unit 140, in particular to the presence/absence prediction unit 141.

The detection information acquisition unit 132 acquires detection information Id (including process information Ip) indicating "at least one state of the tools used in the work Op, the work object Ob, and the work area Ar" from the PLC 20. As described above, the detection information Id is information indicating "at least one state of the tools used in the work Op, the work object Ob, and the work area Ar" detected, for example, by the input device 41 as a detection device (measuring device). The detection information acquisition unit 132 outputs the acquired detection information Id to the first prediction unit 120.

The second prediction unit 140 generates a second prediction Ps including at least one of the presence/absence prediction Pa and the work prediction Po, and outputs the generated second prediction Ps to the determination unit 150. The second prediction unit 140 includes a presence/absence prediction unit 141 and a work prediction unit 142.

The presence/absence prediction unit 141 performs image analysis on the overall image Ia acquired from the second acquisition unit 130 (particularly the overall image acquisition unit 131) to predict "whether a worker Hu is present in the workplace Ar captured in the workplace image Ic," that is, to generate a presence/absence prediction Pa. The presence/absence prediction unit 141, for example, performs model-based image analysis on the overall image Ia to predict "whether a worker Hu is present in the workplace Ar captured in the workplace image Ic."

The presence/absence prediction unit 141, which performs model-based image analysis, is realized, for example, as a neural network, and predicts the presence/absence of a worker Hu in the workplace Ar by executing a movement line detection algorithm (object detection algorithm) such as YOLO. Specifically, the presence/absence prediction unit 141 refers to the memory unit 170 to acquire a presence/absence model 172. The presence/absence model 172 is a trained model that receives the entire image Ia as input and outputs "whether a worker Hu is present in the workplace Ar (more precisely, each of the multiple workplaces Ar in the factory Fa)." The presence/absence prediction unit 141 uses the presence/absence model 172 acquired by referring to the memory unit 170 to predict "whether a worker Hu is present in the workplace Ar captured in the workplace image Ic" from the entire image Ia. For example, the presence/absence prediction unit 141 uses the presence/absence model 172 from the overall image Ia to generate a presence/absence prediction Pa(x), which is a prediction as to whether a worker Hu is present in the workplace Ar(x) captured in the workplace image Ic(x).

In particular, the presence/absence prediction unit 141 performs the following prediction from the overall image Ia using the overall time Ta, which is the time when the ceiling camera 80 captures multiple workplaces Ar, including the workplace Ar(x) captured in the workplace image Ic(x), to generate the overall image Ia. That is, the presence/absence prediction unit 141 predicts from the overall image Ia whether the worker Hu(x) is in the workplace Ar(x) at the overall time Ta, which corresponds to the capture time Tp when the workplace image Ic(x) is captured.

The presence/absence prediction unit 141 may use the presence/absence model 172 to predict "whether a worker Hu is present in the workplace Ar captured in the workplace image Ic," particularly "whether a worker Hu is present in the workplace Ar at the image capture time Tp," from the overall image Ia for a predetermined period (e.g., 5 seconds). The prediction of the presence/absence of the worker Hu by the presence/absence prediction unit 141 using the presence/absence model 172 may result in failure, such as erroneous detection or non-detection. Therefore, if the presence/absence prediction unit 141 predicts that "the worker Hu is present" for only 1 second out of 5 seconds, but predicts that "the worker Hu is not present" for the remaining 4 seconds, the prediction that "the worker Hu is present" may be considered to be a erroneous detection (i.e., an erroneous prediction). Even if it is predicted that "worker Hu is present" for only one second, if it is predicted that "worker Hu is not present" for the remaining four seconds, the presence/absence prediction unit 141 may predict that "worker Hu is not present" from the entire image Ia for those five seconds. In other words, the presence/absence prediction unit 141 may generate a presence/absence prediction Pa that "worker Hu was not present in 'workplace Ar captured in the work place image Ic' during those five seconds" from the entire image Ia for those five seconds.

To summarize, the presence/absence prediction unit 141 may generate a presence/absence prediction Pa that "worker Hu was present" for the entirety of a specified period only when it predicts that "worker Hu was present" for a period of a specified percentage (e.g., 50%) or more of the specified period from the overall image Ia of the specified period. For example, the presence/absence prediction unit 141 may generate a presence/absence prediction Pa that "worker Hu was present during those 5 seconds" only when the period during which it predicted that "worker Hu was present" for a 5-second overall image Ia is 2.5 seconds or more.

The movement line detection algorithm used by the presence/absence prediction unit 141 to predict the presence/absence of the worker Hu is, for example, YOLO, but it is not essential that the movement line detection algorithm used by the presence/absence prediction unit 141 is YOLO, and a movement line detection algorithm other than YOLO may be used. The presence/absence prediction unit 141 only needs to be able to predict, from the overall image Ia, "whether the worker Hu is in the workplace Ar captured in the workplace image Ic" using the movement line detection algorithm.

The presence/absence prediction unit 141 outputs to the determination unit 150 a presence/absence prediction Pa (e.g., presence/absence prediction Pa(x)), which is a prediction as to whether a worker Hu is present in the workplace Ar captured in the workplace image Ic, predicted from the overall image Ia using the presence/absence model 172.

The work prediction unit 142 uses the detection information Id (including the process information Ip) acquired from the second acquisition unit 130 (particularly, the detection information acquisition unit 132) to predict the "contents of the work Op being performed by the worker Hu," that is, to generate a work prediction Po. As described above, the detection information Id is information indicating "the tools (e.g., the output device 42 and the tool 90) used in the work Op, the work object Ob, and at least one state of the work site Ar." For example, the work prediction unit 142 generates the following work prediction Po(x) from the detection information Id(x). That is, the work prediction unit 142 generates a work prediction Po(x), which is a prediction of the "contents of the work Op(x) being performed by the worker Hu(x) in the work site Ar(x)," from the detection information Id(x). As mentioned above, the detection information Id(x) is information that indicates "at least one of the conditions of the tool (e.g., the output device 42(x) or the tool 90(x)) used in the operation Op(x), the work object Ob(x), and the work area Ar(x)."

For example, the work prediction unit 142 predicts that "operation Ac(x) was performed, and work Op(x) corresponding to operation Ac(x) was performed by worker Hu at work site Ar(x)" based on "the magnitude of the current flowing through a tool such as tool 90(x)."

The work prediction unit 142 predicts, for example, from the "power consumption of the tool" that "an operation Ac(x) was executed, and an operation Op(x) corresponding to the operation Ac(x) was performed by a worker Hu at a work site Ar(x)."

For example, the work prediction unit 142 predicts that "work Op(x) was performed by worker Hu in work area Ar(x)" based on the fact that "a tool or work object Ob(x) was grasped."

The work prediction unit 142 predicts that "work Op(x) was performed by a worker Hu at a work site Ar(x)" based on, for example, "the tool, the work object Ob(x), or the opening/closing mechanism of the work site Ar(x) was opened or closed."

The work prediction unit 142 predicts that "work Op(x) was performed by worker Hu at work site Ar(x)" based on, for example, "a tool, work object Ob(x), or a button on work site Ar(x) was pressed."

As described above, the work prediction unit 142 predicts "the content of the work Op being performed by the worker Hu" from the detection information Id, for example, predicts "the content of the work Op(x) being performed by the worker Hu(x) at the workplace Ar(x)" from the detection information Id(x).

In particular, the work prediction unit 142 performs the following prediction using the detection time Td, which is the time when "at least one state of the tool used in the work Op(x), the work object Ob(x), and the work area Ar(x)" contained in the detection information Id(x) was detected. That is, the work prediction unit 142 predicts "the content of the work Op(x) being performed by the worker Hu(x) at the detection time Td corresponding to the imaging time Tp when the work area image Ic(x) was captured" from the detection information Id(x).

The work prediction unit 142 outputs the "contents of the work Op being performed by the worker Hu" predicted from the detection information Id to the determination unit 150 as a work prediction Po. For example, the work prediction unit 142 outputs the work prediction Po(x), which is the "contents of the work Op(x) being performed by the worker Hu(x) at the work site Ar(x)" predicted from the detection information Id(x), to the determination unit 150.

The determination unit 150 verifies the first prediction Pf (whether it is correct or not) using the second prediction Ps, that is, verifies the first prediction Pf (whether it is correct or not) using at least one of the presence/absence prediction Pa and the work prediction Po. Specifically, the determination unit 150 verifies the first prediction Pf(x) (whether it is correct or not) using at least one of the presence/absence prediction Pa(x) and the work prediction Po(x).

As described above, the first prediction Pf(x) is, for example, the "contents of the work Op(x) that a worker Hu(x) is performing on a work object Ob(x) in the workplace Ar(x) at the image capture time Tp" predicted from the workplace image Ic(x). The presence/absence prediction Pa(x) is, for example, a prediction of "whether a worker Hu(x) is in the workplace Ar(x) at the overall time Ta corresponding to the 'image capture time Tp at which the workplace image Ic(x) was captured'" predicted from the overall image Ia. The work prediction Po(x) is, for example, the "contents of the work Op(x) that a worker Hu(x) is performing at the detection time Td corresponding to the 'image capture time Tp at which the workplace image Ic(x) was captured'" predicted from the detection information Id(x).

For example, if the presence/absence prediction Pa(x) predicts that "worker Hu is present in the workplace Ar(x)," the determination unit 150 determines that "the first prediction Pf(x) is correct." In particular, if the determination unit 150 predicts that "worker Hu is present in the workplace Ar(x) at the image capture time Tp, which is the time when the cell camera 30 generates the workplace image Ic(x)," the determination unit 150 determines that "the first prediction Pf(x) is correct."

For example, when the operation Op(x) in the operation prediction Po(x) matches the operation Op(x) in the first prediction Pf(x), the determination unit 150 determines that "the first prediction Pf(x) is correct." In particular, when "the operation Op(x) at the imaging time Tp(x) in the operation prediction Po(x)" matches the "the operation Op(x) at the imaging time Tp(x) in the first prediction Pf(x)," the determination unit 150 determines that "the first prediction Pf(x) is correct."

For example, the determination unit 150 determines that "the worker Hu is present in the work area Ar(x)" when it is predicted that "the worker Hu is present in the work area Ar(x)" and the work Op(x) in the work prediction Po(x) and the work Op(x) in the first prediction Pf(x) match, and determines that "the first prediction Pf(x) is correct". In particular, the determination unit 150 determines that "the first prediction Pf(x) is correct" when both of the following two conditions are satisfied. That is, the first condition is that it is predicted in the work prediction Po(x) that "the worker Hu is present in the work area Ar(x) at the image capture time Tp, which is the time when the cell camera 30 generates the work area image Ic(x)". The second condition is that "the work Op(x at the image capture time Tp(x) in the work prediction Po(x)" and "the work Op(x at the image capture time Tp(x) in the first prediction Pf(x)" match.

The determination unit 150 notifies the estimation unit 160 of the first prediction Pf together with the determination result (verification result). The determination unit 150 may further notify the estimation unit 160 of the second prediction Ps (at least one of the presence/absence prediction Pa and the work prediction Po).

When the determination unit 150 determines that "the first prediction Pf(x) is correct," the estimation unit 160 estimates that the first prediction Pf(x) is "the content of the work Op being performed by the worker Hu at the workplace Ar," and outputs the first prediction Pf(x) to the outside as an estimation result.

When the determination unit 150 determines that the first prediction Pf(x) is incorrect, the estimation unit 160 outputs to the outside the estimation result that there is no worker Hu in the workplace Ar(x) or that there is no content of the work Op(x) that could be estimated with the desired accuracy. The estimation unit 160 may output the estimation result described above together with the workplace image Ic to the video storage server of FIG. 2 or the like.

(Regarding the memory unit)
The storage unit 170 is a storage device that stores various data used by the information processing device 10. The storage unit 170 may non-temporarily store (1) a control program, (2) an OS program, (3) application programs for executing various functions possessed by the information processing device 10, and (4) various data read when executing the application programs, which are executed by the information processing device 10. The above data (1) to (4) are stored in a non-volatile storage device such as a read only memory (ROM), a flash memory, an erasable programmable ROM (EPROM), an electrically EPROM (EEPROM (registered trademark)), or a hard disk drive (HDD). The information processing device 10 may include a temporary storage unit (not shown). The temporary storage unit is a so-called working memory that temporarily stores data used for calculations and calculation results, etc., during the course of various processes executed by the information processing device 10, and is composed of a volatile storage device such as a random access memory (RAM). Which data is to be stored in which storage device is appropriately determined based on the purpose of use, convenience, cost, physical constraints, etc. of the information processing device 10. The storage unit 170 further stores an operation prediction model 171 and a presence/absence model 172.

The work prediction model 171 is a model that the first prediction unit 120 uses to predict "the content of the work Op being performed by the worker Hu captured in the workplace image Ic on the work object Ob captured in the workplace image Ic" from the workplace image Ic. The work prediction model 171 is, for example, a trained model that receives the workplace image Ic as input and outputs information indicating "the content of the work Op being performed by the worker Hu captured in the workplace image Ic on the work object Ob captured in the workplace image Ic".

The presence/absence model 172 is a model that the presence/absence prediction unit 141 uses to predict "whether a worker Hu is present in the workplace Ar captured in the workplace image Ic" from the overall image Ia. The presence/absence model 172 is a trained model that receives the overall image Ia as input and outputs "whether a worker Hu is present in the workplace Ar (more precisely, each of the multiple workplaces Ar in the factory Fa)."

§3. Operational Example Figures 3, 4, and 5 are flow diagrams each illustrating an overview of the process executed by the information processing device 10. The information processing device 10 can estimate "the content of the work Op being performed by the worker Hu" with high accuracy by verifying the first prediction Pf (whether it is correct or not) using the second prediction Ps.

(Processing example when verification is performed using only the work forecast Po)
3 is a flow diagram illustrating an overview of a process executed by the information processing device 10 when verifying (the correctness of) the first prediction Pf using only the work prediction Po of the second prediction Ps. As illustrated in FIG. 3, the information processing device 10 (particularly, the determination unit 150) may verify (the correctness of) the first prediction Pf using only the work prediction Po.

As shown in FIG. 3, the determination unit 150 first checks whether the "contents of the work Op being performed by the worker Hu" are predicted in the work prediction Po, that is, whether "it is predicted from the detection information Id that some work Op is being performed" (S110).

When it is confirmed that "the detection information Id predicts that some work Op is being performed" (YES in S110), the determination unit 150 further checks whether or not "the content of the work Op being performed by the worker Hu" has been predicted in the first prediction Pf. In other words, the determination unit 150 checks "whether or not it has been predicted that some work Op is being performed using the work prediction model 171" (S120).

When it is confirmed that "the work prediction model 171 has been used to predict that some work Op is being performed" (YES in S120), the determination unit 150 executes the following determination. That is, the determination unit 150 determines whether "the content of the work Op predicted from the detection information Id matches the content of the work Op predicted using the work prediction model 171" (S130). That is, the determination unit 150 determines whether "the work Op in the work prediction Po" matches "the work Op(x) in the first prediction Pf".

When the determination unit 150 determines that "the content of the work Op predicted from the detection information Id matches the content of the work Op predicted using the work prediction model 171" (YES in S130), the estimation unit 160 outputs the following estimation result. That is, the estimation unit 160 outputs the content of the work Op predicted by the first acquisition unit 110 using the work prediction model 171 from the workplace image Ic as the estimation result (S140), that is, outputs the first prediction Pf(x) as the estimation result.

When it is confirmed that "the detection information Id did not predict that any work Op was being performed" (NO in S110), that is, when the work prediction unit 142 was unable to predict the content of the work Op from the detection information Id, the estimation unit 160 outputs the following estimation result. In other words, the estimation unit 160 outputs the estimation result that "the content of the work Op could not be estimated with the desired accuracy" (S150).

When it is confirmed that "using the work prediction model 171, it was not predicted that any work Op was being performed" (NO in S120), the estimation unit 160 outputs the following estimation result. That is, the estimation unit 160 outputs the estimation result that "the contents of the work Op were not able to be estimated with the desired accuracy" (S150). In other words, if the first prediction unit 120 cannot predict the contents of the work Op from the workplace image Ic, the estimation unit 160 outputs the estimation result that "the contents of the work Op were not able to be estimated with the desired accuracy."

When it is determined that "the contents of the work Op predicted from the detection information Id do not match the contents of the work Op predicted using the work prediction model 171" (NO in S130), the estimation unit 160 outputs the following estimation result. That is, the estimation unit 160 outputs the estimation result that "the contents of the work Op could not be estimated with the desired accuracy" (S150).

The information processing device 10 verifies the first prediction Pf (whether it is correct or not) using the work prediction Po, that is, uses the HOI detection algorithm in combination with "detection information Id from the input device 41, etc." The information processing device 10 includes a first prediction unit 120 that executes the HOI detection algorithm and a work prediction unit 142 that generates the work prediction Po from the detection information Id, thereby canceling false detections (prediction errors) by the first prediction unit 120. Therefore, the information processing device 10 can cancel false detections (prediction errors) by the HOI detection algorithm and achieve highly accurate estimation.

The work prediction unit 142 predicts the "contents of the work Op being performed by the worker Hu" from the detection information Id from the input device 41, etc., i.e., generates a work prediction Po. The prediction accuracy of the "contents of the work Op being performed by the worker Hu" by the work prediction unit 142 using the detection information Id is higher than the prediction accuracy of the "contents of the work Op being performed by the worker Hu" by the first prediction unit 120 that executes the HOI detection algorithm. In other words, the prediction accuracy of the work prediction Po is higher than the prediction accuracy of the first prediction Pf.

Therefore, by using the first prediction Pf and the work prediction Po, the information processing device 10 can estimate the "contents of the work Op being performed by the worker Hu" with high accuracy. The types of "work Op being performed by the worker Hu" that the first prediction unit 120 can predict by executing the HOI detection algorithm may be the same as the types of "work Op being performed by the worker Hu" that the work prediction unit 142 can predict from the detection information Id, or the former may be greater than the latter.

In general, HOI detection by a HOI detection algorithm (i.e., prediction of the "contents of the work Op") fails if at least one of the worker Hu and the work object Ob cannot be detected (object detection) from the work site image Ic. Therefore, failure of HOI detection, that is, "the work prediction model 171 was not used to predict that some work Op was being performed," may be regarded as "failure to detect an object (e.g., worker Hu) (i.e., absence)."

(Processing example when verification is performed using only the presence/absence prediction Pa)
4 is a flow diagram illustrating an overview of a process executed by the information processing device 10 when verifying the first prediction Pf (correctness or incorrectness) using only the presence/absence prediction Pa of the second prediction Ps. As illustrated in FIG. 4, the information processing device 10 (particularly, the determination unit 150) may verify the first prediction Pf (correctness or incorrectness) using only the presence/absence prediction Pa.

As shown in FIG. 4, the determination unit 150 first checks whether the presence/absence prediction Pa predicts that "the worker Hu is present in the workplace Ar," that is, whether "the presence/absence model 172 is used to predict that the worker Hu is present in the workplace Ar" (S210).

As described above, the presence/absence prediction unit 141 may predict "whether a worker Hu is present in the workplace Ar captured in the workplace image Ic during a predetermined period of time" from the overall image Ia for a predetermined period of time (e.g., 5 seconds). The presence/absence prediction unit 141 may predict "that a worker Hu was present in the workplace Ar captured in the workplace image Ic during the predetermined period of time" only if it predicts that "a worker Hu is present" for a period of at least a predetermined percentage (e.g., 50%) of the predetermined period of time.

When it is confirmed that "using the presence model 172, it has been predicted that the worker Hu is present in the work area Ar" (YES in S210), the determination unit 150 further checks whether or not "the contents of the work Op being performed by the worker Hu" have been predicted in the first prediction Pf. In other words, the determination unit 150 checks whether "using the work prediction model 171, it has been predicted that some work Op is being performed" (S220).

When the determination unit 150 confirms that "using the work prediction model 171, it has been predicted that some work Op is being performed" (YES in S220), the estimation unit 160 outputs the following estimation result. That is, the estimation unit 160 outputs the content of the work Op predicted by the first acquisition unit 110 using the work prediction model 171 from the workplace image Ic as the estimation result (S230), that is, outputs the first prediction Pf(x) as the estimation result.

When it is confirmed that "using the presence/absence model 172, it was predicted that the worker Hu is not present in the workplace Ar" (NO in S210), the estimation unit 160 outputs the following estimation result. That is, the estimation unit 160 outputs the estimation result that "the worker Hu is not present in the workplace Ar" (S250). In other words, when the presence/absence prediction unit 141 predicts from the overall image Ia that "the worker Hu is not present in the workplace Ar captured in the workplace image Ic", the estimation unit 160 outputs the estimation result that "the worker Hu is not present in the workplace Ar".

When it is confirmed that "using the work prediction model 171, it was not predicted that any work Op was being performed" (NO in S220), the estimation unit 160 outputs the following estimation result. That is, the estimation unit 160 outputs the estimation result that "the contents of the work Op were not able to be estimated with the desired accuracy" (S240). In other words, if the first prediction unit 120 could not predict the contents of the work Op from the workplace image Ic, the estimation unit 160 outputs the estimation result that "the contents of the work Op were not able to be estimated with the desired accuracy."

The information processing device 10 verifies the first prediction Pf (whether it is correct or not) using the presence/absence prediction Pa, that is, uses the HOI detection algorithm in combination with a "movement line detection algorithm such as YOLO". That is, the information processing device 10 is equipped with a first prediction unit 120 that executes the HOI detection algorithm and a presence/absence prediction unit 141 that executes the movement line detection algorithm, thereby being able to cancel false detections (prediction errors) by the first prediction unit 120. The movement line detection algorithm can only predict (detect) the presence/absence of the worker Hu, but the prediction accuracy of the movement line detection algorithm is higher than that of the HOI detection algorithm. Therefore, by using the first prediction Pf and the presence/absence prediction Pa, the information processing device 10 can estimate the "contents of the work Op being performed by the worker Hu" with high accuracy.

(Processing example when verification is performed using both presence/absence prediction Pa and work prediction Po)
5 is a flow diagram illustrating an overview of a process executed by the information processing device 10 when verifying the first prediction Pf (whether it is correct or not) using both the presence/absence prediction Pa and the work prediction Po. As illustrated in FIG. 5, the information processing device 10 (particularly, the determination unit 150) may verify the first prediction Pf (whether it is correct or not) using both the presence/absence prediction Pa and the work prediction Po.

As shown in FIG. 5, the determination unit 150 first checks whether the presence/absence prediction Pa predicts that "the worker Hu is present in the workplace Ar," that is, whether "the presence/absence model 172 is used to predict that the worker Hu is present in the workplace Ar" (S310).

As described above, the presence/absence prediction unit 141 may predict, from the overall image Ia for a specified period, "whether a worker Hu is present in the workplace Ar captured in the workplace image Ic during that specified period." The presence/absence prediction unit 141 may predict that "a worker Hu was present in the workplace Ar captured in the workplace image Ic during that specified period" only if it predicts that "a worker Hu is present" for at least a specified percentage of the specified period.

When it is confirmed that "using the presence/absence model 172, it has been predicted that the worker Hu is present in the work area Ar" (YES in S310), the determination unit 150 then checks whether or not "the contents of the work Op being performed by the worker Hu" have been predicted in the work prediction Po. In other words, the determination unit 150 checks "whether or not it has been predicted from the detection information Id that some work Op is being performed" (S320).

When it is confirmed that "the detection information Id predicts that some work Op is being performed" (YES in S320), the determination unit 150 checks whether or not "the content of the work Op being performed by the worker Hu" has been predicted in the first prediction Pf. In other words, the determination unit 150 checks "whether or not it has been predicted that some work Op is being performed using the work prediction model 171" (S330).

When it is confirmed that "the work prediction model 171 has been used to predict that some work Op is being performed" (YES in S330), the determination unit 150 executes the following determination. That is, the determination unit 150 determines whether "the content of the work Op predicted from the detection information Id matches the content of the work Op predicted using the work prediction model 171" (S340). That is, the determination unit 150 determines whether "the work Op in the work prediction Po" matches "the work Op(x) in the first prediction Pf".

When the determination unit 150 determines that "the content of the work Op predicted from the detection information Id matches the content of the work Op predicted using the work prediction model 171" (YES in S340), the estimation unit 160 outputs the following estimation result. That is, the estimation unit 160 outputs the content of the work Op predicted by the first acquisition unit 110 using the work prediction model 171 from the workplace image Ic as the estimation result (S350), that is, outputs the first prediction Pf(x) as the estimation result.

When it is confirmed that "using the presence/absence model 172, it was predicted that the worker Hu is not present in the workplace Ar" (NO in S310), the estimation unit 160 outputs the following estimation result. That is, the estimation unit 160 outputs the estimation result that "the worker Hu is not present in the workplace Ar" (S360). In other words, when the presence/absence prediction unit 141 predicts from the overall image Ia that "the worker Hu is not present in the workplace Ar captured in the workplace image Ic", the estimation unit 160 outputs the estimation result that "the worker Hu is not present in the workplace Ar".

When it is confirmed that "the detection information Id did not predict that any work Op was being performed" (NO in S320), that is, when the work prediction unit 142 was unable to predict the content of the work Op from the detection information Id, the estimation unit 160 outputs the following estimation result. In other words, the estimation unit 160 outputs the estimation result that "the content of the work Op could not be estimated with the desired accuracy" (S370).

When it is confirmed that "using the work prediction model 171, it was not predicted that any work Op was being performed" (NO in S330), the estimation unit 160 outputs the following estimation result. That is, the estimation unit 160 outputs the estimation result that "the contents of the work Op were not able to be estimated with the desired accuracy" (S370). In other words, if the first prediction unit 120 could not predict the contents of the work Op from the workplace image Ic, the estimation unit 160 outputs the estimation result that "the contents of the work Op were not able to be estimated with the desired accuracy."

When it is determined that "the content of the work Op predicted from the detection information Id does not match the content of the work Op predicted using the work prediction model 171" (NO in S340), the estimation unit 160 outputs the following estimation result. That is, the estimation unit 160 outputs the estimation result that "the content of the work Op could not be estimated with the desired accuracy" (S370).

As mentioned above, events that can be predicted from the detection information Id (including process information Ip) are different from events that can be predicted using a movement line detection algorithm. Specifically, the work prediction Po generated by the work prediction unit 142 from the detection information Id is a prediction of "the content of the work Op being performed by the worker Hu." In contrast, the presence/absence prediction Pa generated by the presence/absence prediction unit 141 using the movement line detection algorithm is a prediction of "whether the worker Hu is in the work area Ar captured in the work area image Ic."

Therefore, the information processing device 10 (particularly, the determination unit 150) can use at least one of the work prediction Po and the presence/absence prediction Pa to verify the first prediction Pf (whether it is correct or not). Specifically, the determination unit 150 may verify the first prediction Pf (whether it is correct or not) using only the work prediction Po as shown in FIG. 3, or may verify the first prediction Pf (whether it is correct or not) using only the presence/absence prediction Pa as shown in FIG. 4.

The determination unit 150 may also verify the first prediction Pf (whether it is correct or not) using both the presence/absence prediction Pa and the work prediction Po. In particular, as shown in FIG. 5, the determination unit 150 may first filter the presence/absence of the worker Hu using the presence/absence prediction Pa, and if it predicts that the worker Hu is present, it may further verify the first prediction Pf (whether it is correct or not) using the work prediction Po.

The processing performed by the information processing device 10 described above using Figures 3, 4, and 5 can be summarized as follows. That is, the control method performed by the information processing device 10 is a control method of a work estimation device that estimates "the content of the work Op being performed by the worker Hu at the workplace Ar." The control method performed by the information processing device 10 includes prediction steps (S120, S220, S330), determination steps (S130, S210, S310, S340), and estimation steps (S140, S230, S350).

The prediction step predicts "the content of the work Op being performed by the worker Hu on the work object Ob" from "the workplace image Ic or the feature information generated from the workplace image Ic" using the work prediction model 171. The work prediction model 171 is a trained model that receives as input "the workplace image Ic captured of the workplace Ar" or "the feature information generated from the workplace image Ic" and outputs the following information. In other words, the work prediction model 171 is a trained model that outputs "the content of the work Op being performed by the worker Hu captured in the workplace image Ic on the work object Ob captured in the workplace image Ic".

The determination step determines whether the prediction in the prediction step is correct or not by using at least one of the content of the work Op predicted without using the work prediction model 171 and the determination result of whether or not the worker Hu is present in the workplace Ar. In other words, the determination step determines whether the first prediction Pf is correct or not by using at least one of the work prediction Po and the presence/absence prediction Pa.

In the estimation step, if it is determined in the determination step that "the prediction in the prediction step is correct," the content of the work Op predicted in the prediction step is estimated to be "the content of the work Op being performed by the worker Hu at the workplace Ar." In other words, in the estimation step, if it is verified that the first prediction Pf is correct using at least one of the work prediction Po and the presence/absence prediction Pa, the first prediction Pf is output as an estimation result for "the content of the work Op being performed by the worker Hu at the workplace Ar."

According to the above configuration, the control method determines whether the content of the work Op predicted using the work prediction model 171 is correct or not using at least one of the content of the work Op predicted without using the work prediction model 171 and the determination result of the presence or absence of the worker Hu in the workplace Ar. Then, when it is determined that the prediction using the work prediction model 171 is correct, the control method estimates that the content of the work Op predicted using the work prediction model 171 is "the content of the work Op being performed by the worker Hu in the workplace Ar."

In other words, the control method verifies the accuracy of the content of the work Op predicted using the work prediction model 171 using at least one of the content of the work Op predicted without using the work prediction model 171 and the determination result of whether or not a worker Hu is present in the workplace Ar.

The control method thus has the effect of being able to estimate with high accuracy the "contents of the work Op being performed by the worker Hu in the workplace Ar" from the workplace image Ic.

(Example of the first prediction unit and the presence/absence prediction unit realized as a neural network (NN))
As described above, at least one of the first prediction unit 120 and the presence/absence prediction unit 141 may be realized as an NN. In particular, at least one of the first prediction unit 120 and the presence/absence prediction unit 141 may be realized as a Convolutional Neural Network (CNN) or a Deep Neural Network (DNN).

For example, the first prediction unit 120 (work prediction model 171) outputs a first prediction Pf from the work place image Ic, which indicates "the content of the work Op that the worker Hu captured in the work place image Ic is performing on the work object Ob captured in the work place image Ic." Also, for example, the presence/absence prediction unit 141 (presence model 172) outputs a presence/absence prediction Pa from the overall image Ia, which indicates "whether the worker Hu is in the work place Ar captured in the work place image Ic."

Next, the learning process for constructing the trained models, the task prediction model 171 and the presence/absence model 172, will be described below.

(Learning process)
(About the work prediction model)
The first prediction unit 120 realized as a neural network constructs an activity prediction model 171, which is a trained model, by, for example, supervised learning on a training data set DS1, which is a collection of training data Dt1. The training data Dt1 is data in which at least information indicating "the content of an activity Op being performed by a worker Hu captured in the workshop image Ic on a task object Ob captured in the workshop image Ic" is attached as a label to the workshop image Ic. The "training data" is also called "training data". In addition, the process in which "the first prediction unit 120 constructs the activity prediction model 171 by supervised learning on the training data set DS1" is also called "learning process SS1".

In the learning process SS1, the first prediction unit 120 realized as a neural network constructs, through supervised learning on the learning dataset DS1, an operation prediction model 171 that outputs the following information in response to an input of a workplace image Ic, as a learned model. That is, the operation prediction model 171 is a learned model that receives an input of the workplace image Ic, and outputs "the content of the operation Op being performed by a worker Hu captured in the workplace image Ic on a work object Ob captured in the workplace image Ic."

In other words, the workplace image Ic in the teacher data Dt1 is labeled with information indicating "the content of the work Op being performed by the worker Hu captured in the workplace image Ic on the work object Ob captured in the workplace image Ic." The information attached as a label to the workplace image Ic in the teacher data Dt1 is the same as the information output in response to the input of the workplace image Ic by the work prediction model 171 constructed by machine learning on the learning dataset DS1, which is a collection of the teacher data Dt1.

The first prediction unit 120, implemented as a neural network, stores the task prediction model 171 constructed by the learning process SS1 in the memory unit 170.

(Regarding preparation of training data)
FIG. 6 is a diagram for explaining an example of annotations required for generating the teacher data Dt1. In the teacher data Dt1 illustrated in FIG. 6, the work place image Ic is given the following labels (annotations). That is, the work place image Ic is given a label indicating the "range of the bounding box for detecting the worker Hu" surrounded by a dashed line. In addition, the work place image Ic is given a label indicating the "range of the bounding box for detecting the work object Ob" surrounded by a dashed line. Furthermore, the work place image Ic is given a straight line with arrows on both ends as a label indicating "the content of the work Op performed by the worker Hu captured in the work place image Ic on the work object Ob captured in the work place image Ic". The straight line with arrows on both ends connects the approximate center of the "range of the bounding box for detecting the worker Hu" and the approximate center of the "range of the bounding box for detecting the work object Ob".

To prepare the training data Dt1 as exemplified in FIG. 6, the following five annotations are required for the work place image Ic, even if it is not necessary to set a bounding box for detecting the worker Hu. That is, it is necessary to set the X and Y coordinates of the lower left and upper right points of the four points that indicate the range of the bounding box for detecting the work object Ob for the work place image Ic, which means that four annotations are required. In addition, one annotation is required for the work place image Ic that indicates "the content of the work Op that the worker Hu, who is captured in the work place image Ic, is performing on the work object Ob, which is captured in the work place image Ic."

Here, supervised learning generally requires a large amount of training data Dt1, and in order to prepare a large amount of training data Dt1, it is necessary to perform the above-mentioned five annotations for each of the large number of workplace images Ic. Therefore, if a human is to be made to determine "the content of the work Op that the worker Hu captured in the workplace image Ic is performing on the work object Ob captured in the workplace image Ic" for each of the large number of workplace images Ic, the cost required for this is extremely large.

Therefore, in this embodiment, when preparing the teacher data Dt1, the "contents of the work Op" predicted from the detection information Id, rather than the "contents of the work Op" determined by a human from the workplace image Ic, is attached to the workplace image Ic as a label indicating the "contents of the work Op". In other words, in the teacher data Dt1, the "contents of the work Op" predicted by the work prediction unit 142 from the detection information Id, rather than the "contents of the work Op" determined by a human from the workplace image Ic, is attached as a label indicating the "contents of the work Op".

Therefore, the cost required to construct the work prediction model 171 can be significantly reduced compared to when "content determined by a human from the work site image Ic" is used as a label indicating the "content of the work Op". In particular, the annotation cost required to generate the learning dataset DS1 can be significantly reduced compared to when "content determined by a human from the work site image Ic" is used as a label indicating the "content of the work Op". For example, one of the above-mentioned five annotations required for the work site image Ic is changed from "content determined by a human" to "content predicted by the work prediction unit 142 from the detection information Id", thereby reducing the annotation cost by 20%.

For example, the following "contents of work Op(x)" are attached as labels to a work site image Ic(x) generated by the cell camera 30 capturing an image of the work site Ar(x) at a certain image capturing time Tp, thereby generating teacher data Dt1(x). That is, teacher data Dt1(x) is generated in which the "contents of work Op(x) performed by the worker Hu(x) in the work site Ar(x) at the image capturing time Tp" predicted by the work prediction unit 142 from the detection information Id(x) are attached as labels to the work site image Ic(x). The detection information Id(x) is information indicating "the tool (e.g., the output device 42(x) or the tool 90(x) used in the work Op(x), the work object Ob(x), and at least one state of the work site Ar(x)" at the image capturing time Tp.

Specifically, the task prediction unit 142 predicts, for example, that "at the image capture time Tp, the worker Hu performed task Op(x): screw tightening in the work area Ar(x)" from "the magnitude of the current flowing through the tool 90(x): electric screwdriver at the image capture time Tp." This predicted content (i.e., "content of task Op(x): screw tightening") is added to the work area image Ic(x) to generate the teacher data Dt1(x).

As described above, in the information processing device 10, the work prediction model 171 is a trained model constructed by machine learning using the training dataset DS1. In the training dataset DS1, the "workplace image Ic or feature information generated from the workplace image Ic" is labeled with the following information. That is, the "contents of the work Op being performed by the worker Hu in the workplace Ar" predicted from the detection information Id of the input device 41 that detects "the tools used in the work Op, the work object Ob, and at least one state of the workplace Ar" is labeled.

According to the above configuration, the information processing device 10 constructs an operation prediction model 171 by machine learning using a learning dataset labeled with the "contents of the operation Op being performed by the worker Hu in the workplace Ar" predicted from the detection information Id of the input device 41.

It is known that supervised learning requires a large amount of training data (training dataset), and the labeling process required to generate such training data and the labels (metadata) attached to the training data are also called "annotations."

For example, even if the teacher data Dt1 learned when constructing the work prediction model 171 does not require the setting of a bounding box for detecting the worker Hu, the following five annotations are required for the work place image Ic. That is, it is necessary to set the X and Y coordinates of the lower left and upper right points of the four points that indicate the range of the bounding box for detecting the work object Ob for the work place image Ic, that is, four annotations are required. Furthermore, it is necessary to assign information (label) to the work place image Ic that indicates "the content of the work Op that the worker Hu, who is captured in the work place image Ic, is performing on the work object Ob, which is captured in the work place image Ic," that is, one annotation is required.

The cost of having a human determine "the content of the work Op being performed by the worker Hu captured in the workplace image Ic on the work object Ob captured in the workplace image Ic" for each of the large number of workplace images Ic is extremely high.

Therefore, when generating the teacher data Dt1, the "contents of the work Op" predicted from the detection information Id of the input device 41 is assigned as a label indicating the "contents of the work Op", rather than the "contents of the work Op" determined by a human from the workplace image Ic. In other words, in the teacher data Dt1, the "contents of the work Op" predicted from the detection information Id of the input device 41 is assigned as a label indicating the "contents of the work Op", rather than the "contents of the work Op" determined by a human from the workplace image Ic.

Therefore, the cost required to construct the work prediction model 171 can be significantly reduced compared to when "content determined by a human from the work site image Ic" is used as a label indicating the "content of the work Op". In particular, the annotation cost required to generate the learning dataset DS1 can be significantly reduced compared to when "content determined by a human from the work site image Ic" is used as a label indicating the "content of the work Op". For example, one of the above-mentioned five annotations required for the work site image Ic is changed from "content determined by a human" to "content predicted by a device such as the information processing device 10 from the detection information Id", thereby reducing the annotation cost by 20%.

Therefore, according to the above configuration, the information processing device 10 has the effect of being able to estimate with high accuracy the "contents of the work Op being performed by the worker Hu" from the workplace image Ic using the work prediction model 171 in which annotation costs are significantly reduced.

(Regarding the existence model)
The presence/absence prediction unit 141 realized as a neural network constructs the presence/absence model 172, which is a trained model, by, for example, supervised learning on a data set DS2, which is a collection of teacher data Dt2. The teacher data Dt2 is data in which information indicating "whether a worker Hu is present in a workplace Ar (more precisely, in each of the multiple workplaces Ar in the factory Fa)" is attached as a label to the entire image Ia. The process in which "the presence/absence prediction unit 141 constructs the presence/absence model 172 by supervised learning on the data set DS2" is also referred to as "learning process SS2".

In the learning process SS2, the presence/absence prediction unit 141 implemented as a neural network constructs, through supervised learning on the dataset DS2, a presence/absence model 172 that outputs the following information in response to an input of the entire image Ia, as a learned model. That is, the presence/absence model 172 is a learned model that receives an input of the entire image Ia and outputs "whether a worker Hu is present in the workplace Ar (more precisely, in each of the multiple workplaces Ar in the factory Fa)."

In other words, in the teacher data Dt2, the whole image Ia is labeled with information indicating whether a worker Hu is present in the work area Ar (more precisely, in each of the multiple work areas Ar in the factory Fa). The information labeled for the whole image Ia in the teacher data Dt2 is the same as the information output by the presence/absence model 172 constructed by machine learning for the dataset DS2, which is a collection of the teacher data Dt2, in response to the input of the whole image Ia.

The presence/absence prediction unit 141, implemented as a neural network, stores the presence/absence model 172 constructed by the learning process SS2 in the memory unit 170.

However, it is not essential that the presence/absence prediction unit 141 constructs the presence/absence model 172 by the learning process SS2. The presence/absence prediction unit 141 may use a general learned model used in a traffic line detection algorithm as the presence/absence model 172 to predict "whether a worker Hu is in the workplace Ar (more precisely, in each of the multiple workplaces Ar in the factory Fa)" from the overall image Ia.

§4. Variations (machine learning)
So far, an example has been described in which the task prediction model 171 and the presence/absence model 172 are constructed as learned data by machine learning on supervised data. However, the machine learning for constructing the task prediction model 171 and the presence/absence model 172 is not limited to machine learning on supervised data. The machine learning for constructing the task prediction model 171 and the presence/absence model 172 is not limited to supervised learning, and may be other machine learning such as unsupervised learning or reinforcement learning.

(About the whole image)
So far, an example has been described in which the presence/absence prediction unit 141 predicts "whether the worker Hu is in the workplace Ar captured in the workplace image Ic" from the entire image Ia. However, the presence/absence prediction unit 141 may predict "whether the worker Hu is in the workplace Ar captured in the workplace image Ic" from the workplace image Ic. That is, it is not essential for the information processing device 10 to acquire the entire image Ia separately from the workplace image Ic. The information processing device 10 (particularly, the determination unit 150) can verify the correctness of the first prediction Pf generated from the workplace image Ic by the presence/absence prediction Pa generated from at least one of the entire image Ia and the workplace image Ic. For example, if the presence/absence prediction Pa(x) generated from at least one of the entire image Ia and the workplace image Ic(x) is a prediction that "the worker Hu is present in the workplace Ar(x)", the determination unit 150 determines that "the first prediction Pf(x) is correct".

(Regarding work forecasts)
So far, an example has been described in which the information processing device 10 is the entity that predicts "the contents of the work Op being performed by the worker Hu" from the detection information Id. That is, an example has been described in which the work prediction unit 142 grasps "at least one state of the tool used in the work Op, the work target Ob, and the work site Ar" from the detection information Id, and generates a work prediction Po from the grasped state.

However, the entity that grasps "at least one state of the tool used in the work Op, the work object Ob, and the work area Ar" from the detection information Id and predicts "the content of the work Op being performed by the worker Hu" from the grasped state does not have to be the information processing device 10. It is sufficient for the information processing device 10 to acquire or generate "the content of the work Op being performed by the worker Hu" predicted from at least one state of the tool, the work object Ob, and the work area Ar.

For example, the PLC 20 may generate a work prediction Po from the detection information Id. In this case, the information processing device 10 (particularly the detection information acquisition unit 132) acquires the result of the prediction performed by the PLC 20, that is, the work prediction Po, from the PLC 20. The determination unit 150 then uses the work prediction Po generated by the PLC 20 to verify the first prediction Pf (whether it is correct or not).

In other words, when the information processing device 10 (particularly the determination unit 150) uses the work prediction Po to verify the first prediction Pf (whether it is correct or not), the entity that generates the work prediction Po from the detection information Id may be the information processing device 10, or it may not be the information processing device 10.

(Regarding the second prediction)
Up to this point, an example has been described in which the information processing device 10 includes both the presence/absence prediction unit 141 and the work prediction unit 142, but it is not essential that the information processing device 10 includes both the presence/absence prediction unit 141 and the work prediction unit 142. The determination unit 150 only needs to be able to verify the first prediction Pf (whether it is correct or not) using the second prediction Ps, that is, it only needs to be able to verify the first prediction Pf (whether it is correct or not) using at least one of the presence/absence prediction Pa and the work prediction Po. Therefore, if the presence/absence prediction Pa is not used to verify the first prediction Pf (whether it is correct or not), the information processing device 10 does not need to include the presence/absence prediction unit 141. Also, if the work prediction Po is not used to verify the first prediction Pf (whether it is correct or not), the information processing device 10 does not need to include the work prediction unit 142.

[Software implementation example]
The functional blocks of the information processing device 10 (specifically, the first acquisition unit 110, the first prediction unit 120, the second acquisition unit 130, the second prediction unit 140, the determination unit 150, and the estimation unit 160) may be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like. Furthermore, these functional blocks may be realized by software using a CPU, a GPU, a DSP, or the like.

In the latter case, the information processing device 10 includes a CPU, GPU, DSP, etc. that executes the commands of a program, which is software that realizes each function, a ROM or storage device (these are referred to as "recording media") in which the program and various data are recorded so as to be readable by a computer (or CPU), and a RAM or the like that deploys the program. The object of the present invention is achieved by having the computer (or CPU, GPU, DSP, etc.) read and execute the program from the recording media. The recording media may be "non-transient tangible media," such as tapes, disks, cards, semiconductor memories, programmable logic circuits, etc. The program may also be supplied to the computer via any transmission medium (such as a communication network or broadcast waves) that can transmit the program. The present invention may also be realized in the form of a data signal embedded in a carrier wave in which the program is embodied by electronic transmission.

The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. The technical scope of the present invention also includes embodiments obtained by appropriately combining the technical means disclosed in different embodiments.

10 Information processing device (work estimation device)
41 Input device (sensor)
42 Output device (tool)
90 Tools
120 First prediction unit (prediction unit)
141 Presence prediction unit 142 Work prediction unit (work detection unit)
150 Determination unit 160 Estimation unit 171 Work prediction model 172 Presence model Ar Work place DS1 Learning data set Hu Worker Ia Whole image Ic Work place image Id Detection information Ob Work object Op Work S120 (Prediction step)
S130 (Determination step)
S140 (Estimation step)
S210 (Determination step)
S220 (Prediction step)
S230 (Estimation step)
S310 (Determination step)
S330 (Prediction step)
S340 (Determination step)
S350 (Estimation step)

Claims (6)

A task estimation device that estimates the content of a task performed by a worker in a workplace,
a prediction unit that predicts the content of the work that the worker is performing on the work object, based on the workplace image or the feature information, using a work prediction model that is a trained model that receives an input of a workplace image obtained by capturing an image of the workplace or feature information generated from the workplace image, and outputs the content of the work that the worker, who is captured in the workplace image, is performing on the work object, which is captured in the workplace image, from the workplace image or the feature information;
a determination unit that determines whether the prediction by the prediction unit is correct or not, using at least one of the work content predicted without using the work prediction model and a determination result of whether the worker is present in the work area; and
an estimation unit that, when the determination unit determines that the prediction by the prediction unit is correct, estimates the content of the work predicted by the prediction unit as the content of the work being performed by the worker in the workplace; and
In addition to providing
a presence/absence prediction unit that predicts whether the worker is present in the workplace from the overall image, or the workplace image , or the feature information generated from the overall image or the workplace image, using a presence/absence model that is a trained model that inputs (A) an overall image of a plurality of workplaces including the workplace, or (B) an image of the workplace, or (C) feature information generated from the overall image or the workplace image, and outputs whether the worker is present in the workplace,
The determination unit determines that the prediction by the presence prediction unit is correct when the presence prediction unit predicts that the worker is present in the work area. a work detection unit that predicts the content of the work being performed by the worker in the work area based on detection information from a sensor that detects at least one state of a tool used in the work, the work object, and the work area;
Further equipped with
2. The work estimation device according to claim 1, wherein the determination unit determines that the prediction by the prediction unit is correct when the presence prediction unit predicts that the worker is present in the work area and the content of the work predicted by the prediction unit matches the content of the work predicted by the work detection unit. 3. The work estimation device according to claim 1 or 2, wherein the work prediction model is constructed by machine learning using a learning dataset in which content of the work being performed by the worker in the workplace, predicted from detection information of a sensor that detects at least one state of a tool used in the work, the work object, and the workplace, is assigned as a label to the workplace image or feature information generated from the workplace image. A control method for an activity estimation device that estimates the content of an activity being performed by a worker in a workplace, comprising:
a prediction step of predicting the content of the work being performed by the worker on the work object, based on the workplace image or the feature information, using a work prediction model which is a trained model that receives an input of a workplace image of the workplace or feature information generated from the workplace image and outputs the content of the work being performed by the worker, whose image is captured in the workplace image, on the work object, whose image is captured in the workplace image;
a determination step of determining whether the prediction in the prediction step is correct or not, using at least one of the work content predicted without using the work prediction model and a determination result of whether the worker is present in the work area;
an estimation step of estimating, when it is determined in the determination step that the prediction in the prediction step is correct, the content of the work predicted in the prediction step as the content of the work being performed by the worker in the workplace;
Including,
(A) an overall image of a plurality of workplaces including the workplace is input , or (B) the workplace image is input, or (C) feature information generated from the overall image or the workplace image is input, and a presence/absence model is used as a trained model that outputs whether the worker is present in the workplace from the overall image, or the workplace image, or the feature information generated from the overall image or the workplace image,
The control method, wherein the judging step judges that the prediction made in the predicting step is correct when the presence of the worker in the work area is predicted by the presence predicting step. An information processing program for causing a computer to function as the task estimation device according to any one of claims 1 to 3, the information processing program causing the computer to function as each of the units. A computer-readable recording medium having the information processing program according to claim 5 recorded thereon.

Images