JP7681826B2 - Inspection device, learning device, inspection method, learning device production method, and program - Google Patents

Description

The present invention relates to an inspection device that uses video of a series of tasks that have two or more steps to inspect whether the tasks are appropriate and outputs the inspection results.

Conventionally, there have been inspection methods that detect defects by processing captured images of an object to be inspected (see, for example, Patent Document 1).

Patent No. 6218094

However, in conventional technology, it was not possible to photograph a task that had two or more steps and determine whether the task was being performed correctly according to the planned steps.

The inspection device of the first invention is an inspection device that includes an inspection video acquisition unit that acquires an inspection video, which is a video captured while an inspection target work, which is an inspection target work having two or more steps, is being performed, a judgment unit that judges whether or not the procedure of the inspection target work for the inspection video is correct using teacher data based on a normal video, which is a video captured while a normal work, which is a correct work having two or more steps, is being performed, and an output unit that outputs judgment result information regarding the judgment result of the judgment unit.

With this configuration, by photographing a task that has two or more steps, it is possible to determine whether the task is being performed correctly according to the planned steps.

The inspection device of the second invention is an inspection device in which, compared to the first invention, the inspection video acquisition unit sequentially acquires inspection videos while the work to be inspected is being performed, and the determination unit determines whether or not work corresponding to a procedure that is currently being performed is being performed for one or more fields contained in the inspection videos sequentially acquired by the inspection video acquisition unit, by using the training data corresponding to the procedure that is currently being performed among the training data.

With this configuration, by photographing a task that has two or more steps, it is possible to determine in real time whether the task is being performed correctly and to inform the worker of the result of the determination.

The inspection device of the third invention is an inspection device according to the first or second invention, further comprising a current procedure storage unit in which a current procedure identifier that identifies the procedure currently being inspected among the two or more procedures is stored, the teacher data has one field corresponding to each of the two or more procedures, and the judgment unit comprises an inspection image acquisition means that acquires one field of the inspection video acquired by the inspection video acquisition unit, and a judgment means that uses the one field acquired by the inspection image acquisition means and the one field corresponding to the current procedure identifier to judge whether the procedure corresponding to the one field acquired by the inspection image acquisition means is a procedure identified by the current procedure identifier.

With this configuration, by photographing a task that has two or more steps, it is possible to quickly determine whether the task is being performed correctly.

The inspection device of the fourth invention is an inspection device according to the third invention, further comprising a learning device storage unit in which a learning device constructed by performing a learning process on the two or more teaching data using a machine learning algorithm is stored, and the judgment means performs a machine learning prediction process using the one field and the learning device acquired by the inspection image acquisition means, acquires a procedure identifier corresponding to the one field, and judges whether the procedure identifier matches the current procedure identifier.

With this configuration, by photographing a task that has two or more steps, it is possible to quickly and flexibly determine whether the task is being performed correctly.

The inspection device of the fifth invention is an inspection device in which, compared to the third invention, the judgment means calculates a similarity between a field corresponding to each of two or more procedures and a field acquired by the inspection image acquisition means, and uses the two or more similarities to judge whether the procedure corresponding to the field acquired by the inspection image acquisition means is a procedure identified by the current procedure identifier.

With this configuration, by photographing a task that has two or more steps, it is possible to quickly determine whether the task is being performed correctly.

The inspection device of the sixth invention is an inspection device according to the first or second invention, further comprising a current procedure storage unit in which a current procedure identifier that identifies the procedure currently being inspected among the two or more procedures is stored, the teacher data has two or more fields corresponding to each of the two or more procedures, and the judgment unit comprises an inspection image acquisition means that acquires two or more fields of the inspection video acquired by the inspection video acquisition unit, and a judgment means that uses the two or more fields acquired by the inspection image acquisition means and the two or more fields corresponding to the current procedure identifier to judge whether the procedure corresponding to the two or more fields acquired by the inspection image acquisition means is a procedure identified by the current procedure identifier.

With this configuration, by photographing a task that has two or more steps, it is possible to accurately and flexibly determine whether the task is being performed correctly.

The inspection device of the seventh invention is an inspection device according to the sixth invention, further comprising a learning module storage unit in which a learning module constructed by performing a learning process on the two or more teaching data using a machine learning algorithm is stored, and the judgment means performs a prediction process using a machine learning algorithm using the two or more fields and the learning module acquired by the inspection image acquisition means, acquires a procedure identifier corresponding to the two or more fields, and judges whether the procedure identifier matches the current procedure identifier.

With this configuration, by photographing a task that has two or more steps, it is possible to accurately and flexibly determine whether the task is being performed correctly.

The inspection device of the eighth invention is an inspection device according to any one of the first to seventh inventions, in which the judgment unit acquires one or more fields that are a portion of fields from the inspection video that satisfy a predetermined condition, and judges whether the procedure of the work to be inspected for the inspection video is correct using the one or more fields and the teacher data.

With this configuration, by photographing a task that has two or more steps, it is possible to quickly and flexibly determine whether the task is being performed correctly.

The inspection device of the ninth invention is an inspection device according to any one of the first to eighth inventions, in which the judgment unit performs a machine learning prediction process using a second learning device for determining whether or not each field in the inspection video is to be used as a field for determining whether or not the procedure is correct, obtains one or more fields corresponding to the prediction result that they will be used, and uses the one or more fields and the teacher data to judge whether or not the procedure of the work to be inspected in the inspection video is correct.

With this configuration, by photographing a task that has two or more steps, it is possible to determine with high accuracy whether the task is being performed correctly.

The inspection device of the tenth invention is an inspection device according to any one of the first to ninth inventions, further comprising an equipment-related information acquisition unit that acquires equipment-related information, which is information input to the device for the work being inspected, or information generated within the device, or information output from the device, and the judgment unit uses teacher data based on normal equipment-related information, which is equipment-related information when normal work is performed, and normal video, to judge whether the procedure of the work being inspected is correct for the inspection video and the equipment-related information.

With this configuration, it is possible to determine whether the work is being performed correctly by also using device-related information when work is performed on the device used for that work.

The inspection device of the eleventh invention, compared to the tenth invention, further includes a normal device-related information storage unit in which normal device-related information, which is device-related information when normal work is performed, is stored, and the judgment unit includes a first judgment means for judging whether the procedure of the work to be inspected is correct using the device-related information and normal device-related information acquired by the device-related information acquisition unit, and a second judgment means for judging whether the procedure of the work to be inspected for the inspection video is correct using teacher data based on the normal video when the judgment result of the first judgment means is that the procedure is correct.

With this configuration, after making a judgment using the device-related information, only if the judgment is normal is a judgment made using the inspection image, so that the normality of the work can be judged accurately and efficiently.

The inspection device of the twelfth invention is an inspection device further comprising a normal device-related information storage unit in which normal device-related information, which is device-related information when normal work is performed, is stored, and the judgment unit comprises a second judgment means for judging whether the procedure of the work to be inspected for the inspection video is correct using teacher data based on the normal video, and a first judgment means for judging whether the procedure of the work to be inspected is correct using the device-related information and normal device-related information acquired by the device-related information acquisition unit when the judgment result of the second judgment means is correct.

With this configuration, after making a judgment using the inspection image, only if the judgment is normal is a judgment made using the device-related information, so that the normality of the work can be judged accurately and efficiently.

The inspection device of the thirteenth invention is an inspection device according to the tenth invention, further comprising a learning device storage unit in which the teacher data is a set of one field corresponding to each of two or more procedures, normal device-related information, and a procedure identifier for identifying each procedure, and in which a learning device constructed by performing a learning process on the two or more teacher data using a machine learning algorithm is stored, and the judgment unit acquires one field of the inspection video acquired by the inspection video acquisition unit, and acquires a procedure identifier for identifying the procedure corresponding to the one field by machine learning prediction processing using the one field, the device-related information acquired by the device-related information acquisition unit, and the learning device, and judges whether the procedure identifier matches the current procedure identifier.

With this configuration, the machine learning algorithm can be used to determine whether the work is being performed correctly, using device-related information when the work is performed on the device used for that work.

The learning device of the fourteenth invention is a learning device that includes a learning storage unit that stores a normal video, which is a video captured of a normal task being performed, which is a correct task having two or more steps; a teacher data acquisition unit that acquires one or more fields that satisfy a predetermined condition from the normal video for each of the two or more steps, and acquires two or more teacher data that are pairs of a step identifier that identifies the step corresponding to the field and the field; and a learning unit that performs machine learning learning processing on the two or more teacher data, acquires a learner, and accumulates the learner.

This configuration results in a learning device that can be used by inspection equipment.

The learning device of the fifteenth invention is a learning device that includes a teacher data storage unit that stores one or more teacher data based on a normal video, which is a video captured of a normal operation, which is a correct operation having two or more steps, being performed, and normal device-related information, which is device-related information when the normal operation is performed; a teacher data acquisition unit that acquires, for each of the two or more steps, two or more teacher data that are a combination of one or more fields that satisfy a predetermined condition, normal device-related information, and a step identifier from the normal video; and a learning unit that performs a machine learning learning process using the two or more teacher data, acquires a learner, and accumulates the learner.

This configuration results in a learning device that can be used by inspection equipment.

The inspection device of the present invention can photograph a task that has two or more steps and determine whether the task is being performed correctly.

1 is a conceptual diagram of an inspection system A according to a first embodiment. Block diagram of the inspection system A A flowchart for explaining an example of the operation of the inspection device 2. A flowchart illustrating an example of the field acquisition process. 11 is a flowchart illustrating an example of the first determination process. 11 is a flowchart illustrating an example of the second determination process. 11 is a flowchart illustrating an example of the third determination process. 11 is a flowchart illustrating an example of the fourth determination process. FIG. 11 is a block diagram of a learning device 3 according to the second embodiment. 1 is a flowchart illustrating a first operation example of the learning device 3. 11 is a flowchart illustrating a second operation example of the learning device 3. A flowchart illustrating an example of the field acquisition process. A diagram showing the field management table 1 is a conceptual diagram of an inspection system B according to a third embodiment. Block diagram of the inspection system B A flowchart for explaining an example of the operation of the inspection device 4 11 is a flowchart illustrating a first example of the determination process. 11 is a flowchart illustrating a second example of the determination process. 11 is a flowchart illustrating a third example of the determination process. 11 is a flowchart illustrating the operation of a fourth example of the determination process. 5 is a flowchart illustrating a fifth example of the determination process. A flowchart illustrating an example of the first determination process. Block diagram of a learning device 5 according to the fourth embodiment. 1 is a flowchart illustrating a first operation example of the learning device 5. 11 is a flowchart illustrating a second operation example of the learning device 5. Overview of the computer system according to the above embodiment. Block diagram of the computer system

Embodiments of the inspection device and the like will be described below with reference to the drawings. Note that components with the same reference numerals in the embodiments perform similar operations, and therefore may not be described again.

(Embodiment 1)
In this embodiment, an inspection device is described that uses an inspection video captured during a series of tasks having two or more steps and normative teacher data to inspect whether the tasks are being performed according to the planned steps and outputs the inspection results.

In this embodiment, we also describe an inspection device that sequentially accepts inspection videos during work and checks in real time whether the work is being performed according to the planned procedure.

In addition, in this embodiment, an inspection device that performs inspections using one or more fields corresponding to two or more procedures will be described.

In this embodiment, we also describe an inspection device that uses a machine learning algorithm to check whether work is being performed according to the planned procedure.

In addition, in this embodiment, an inspection device that uses a machine learning algorithm when selecting fields to be used for inspection from an inspection video of a series of tasks having two or more steps is described.

Figure 1 is a conceptual diagram of an inspection system A in this embodiment. The inspection system A includes one or more cameras 1 and an inspection device 2.

Camera 1 captures a series of tasks having two or more steps and obtains an inspection video. An inspection video is a video of the object of inspection. The inspection is an inspection to see whether a series of tasks having two or more steps is being performed according to the planned steps. Note that camera 1 may be fixed, or may be placed in a moving environment, such as being installed on a moving object.

The inspection device 2 is a device that inspects the inspection video captured by the camera 1, inspects whether a series of tasks having two or more steps is being performed according to the planned steps, and outputs the inspection results. The inspection device 2 is, for example, a so-called personal computer, a tablet terminal, a smartphone, a server, etc., and the type does not matter. The server is, for example, a cloud server or an ASP server, and the type does not matter.

The camera 1 and the inspection device 2 may be integrated into a single device. Such a device may be, for example, a tablet terminal or a smartphone equipped with a camera.

Figure 2 is a block diagram of the inspection system A in this embodiment. The inspection device 2 includes a storage unit 21, a processing unit 22, and an output unit 23.

The storage unit 21 includes a teacher data storage unit 211, a learning device storage unit 212, a threshold storage unit 213, and a current procedure storage unit 214. The processing unit 22 includes an inspection video acquisition unit 221, and a judgment unit 222. The judgment unit 222 includes an inspection image acquisition means 2221 and a judgment means 2222.

Camera 1 captures images of the work to be inspected, which is work to be inspected that has two or more steps, and obtains a video. The video is usually an inspection video. The work to be inspected may be video that includes or does not include a worker. It is preferable that camera 1 is installed in a location where the results of the work in each step can be captured well. The steps may also be referred to as steps, etc. The worker may be a person or a device. The device is, for example, a robot or a computer. The robot may be an industrial robot used in the manufacture of some kind of item, or it may be a humanoid robot, etc., and the type is not important.

Various types of information are stored in the storage unit 21 that constitutes the inspection device 2. Examples of the various types of information include teacher data (described later), a learning device (described later), a threshold (described later), a current procedure identifier (described later), and procedure information (described later).

Procedure information is information that specifies two or more procedures that make up a task. Procedure information has, for example, two or more procedure identifiers in the order of the procedures. The procedure identifier is information that identifies a procedure. The procedure identifier is, for example, an ID or a procedure name. Procedure information has, for example, two or more pairs of information indicating the order of the procedures and the procedure identifier. For example, procedure information for the task of making ramen is, for example, "Pour in the soup, pour in the noodles, pour in the toppings (such as green onions)" and "(1, pour in the soup) (2, pour in the noodles) (3, pour in the toppings)."

Two or more pieces of teacher data are stored in the teacher data storage unit 211. Teacher data is information used for comparison with part or all of the inspection video. Teacher data is usually data based on a normal video (referred to as a positive example as appropriate). A normal video is a video captured of normal work being performed. A normal work is a correct work that has two or more steps. Data based on a normal video is information obtained using a normal video. Teacher data includes, for example, one or more fields that make up a normal video.

The teacher data storage unit 211 typically has one or more teacher data for each procedure. The teacher data for each procedure is typically associated with a procedure identifier. The teacher data is, for example, one field that represents the procedure (which may also be called a representative field). The teacher data is, for example, two or more fields that correspond to each procedure. The teacher data is, for example, a normal scene video that corresponds to each procedure. A normal scene video is a video that has been shot from the beginning to the end of a procedure. The teacher data may have a procedure identifier. The teacher data may be associated with a procedure identifier. Note that being associated with a procedure identifier may be considered to mean having a procedure identifier.

The one or more pieces of training data may include data based on an abnormal video, which is a video of an abnormal task (referred to as a negative example, as appropriate). A negative example is information obtained using an abnormal video. A negative example includes, for example, one or more fields that make up an abnormal video. A negative example may include, for example, a flag indicating that the task is abnormal. A negative example may have, for example, a procedure identifier, or may be associated with a procedure identifier.

When the worker is a device such as a robot or a personal computer, the normal video may be information in which a simulation device (not shown) simulates normal operations performed by the device and records the resulting video. The abnormal video may be information in which a simulation device (not shown) simulates abnormal operations that may be performed by the device and records the resulting video. Note that a simulation device that simulates operations that may be performed by device C and outputs the simulation results is a publicly known technology, so a detailed description will be omitted.

The learning device storage unit 212 stores a first learning device in addition to or instead of the teacher data storage unit 211. The learning device storage unit 212 may also store a second learning device.

The first learning device is a learning device used when determining whether or not the procedure of the work to be inspected for the inspection video is correct. The first learning device is a learning device configured by performing a learning process on two or more pieces of teacher data using a machine learning algorithm. The learning device may also be called a classifier, a model, etc.

The learning device storage unit 212 may have a first learning device for each procedure. In other words, each of the two or more first learning devices may be associated with a procedure identifier.

The second learning device is a learning device for selecting a field from the inspection video to be used for determining whether the procedure of the work to be inspected is correct or not. The second learning device is, for example, a learning device configured by performing a learning process on one or more inspection selection positive example fields and one or more inspection selection negative example fields using a machine learning algorithm. The inspection selection positive example field is a field used for inspection. The inspection selection negative example field is a field not used for determining whether the procedure of the work to be inspected is correct or not. The inspection selection negative example field is a field not used for inspection. The inspection selection positive example field and the inspection selection negative example field are, for example, fields manually selected from normal videos and/or abnormal videos.

Furthermore, it is preferable that the first learning device and the second learning device are learning devices acquired by the learning device 3 described below.

Furthermore, examples of machine learning algorithms include deep learning, random forests, decision trees, SVMs, etc. However, the type of machine learning algorithm is not important. The fact that the type of machine learning algorithm is not important applies to both the learning process and the prediction process.

The threshold storage unit 213 stores a threshold value related to the similarity with the teacher data. For example, if the similarity between the field acquired from the inspection video and the teacher data of a certain procedure is equal to or greater than the threshold value, it is determined that the procedure corresponding to the field acquired from the inspection video is the certain procedure. Note that the threshold value may be stored for each procedure. In other words, each of the one or more threshold values may be associated with a procedure identifier.

The current procedure storage unit 214 stores a current procedure identifier. The current procedure identifier is information that identifies the procedure currently being inspected out of two or more procedures. If the procedure information is, for example, "Pour soup, Pour noodles, Pour toppings", the current procedure identifier is, for example, "Pour soup". Also, if the procedure information is, for example, "(1, Pour soup) (2, Pour noodles) (3, Pour toppings)", the current procedure identifier is, for example, "1" or "Pour soup".

The processing unit 22 performs various types of processing. The various types of processing are, for example, processing performed by the inspection video acquisition unit 221 and the judgment unit 222.

The inspection video acquisition unit 221 acquires the inspection video. The inspection video is a video captured while the work to be inspected is being performed. The work to be inspected is a task that is the subject of inspection and has two or more steps. The work to be inspected is, for example, the task of preparing a dish (e.g., ramen noodles) or the task of manufacturing a product. The inspection video acquisition unit 221 receives a video from, for example, the camera 1 capturing the scene where the work to be inspected is being performed. The inspection video acquisition unit 221 reads out, for example, inspection images stored in a recording medium (not shown) by the camera 1 capturing the scene where the work to be inspected is being performed.

It is preferable that the inspection video acquisition unit 221 sequentially acquires the inspection video while the work to be inspected is being performed. In other words, it is preferable that the inspection video acquisition unit 221 sequentially acquires the inspection video in real time while the work to be inspected is being performed.

The judgment unit 222 uses teacher data based on the normal video to judge whether the procedure of the work to be inspected for the inspection video is correct. Note that a normal video is a video captured of normal work being performed. Also, normal work is correct work that has two or more steps. An inspection video is a video of the work to be inspected. The work to be inspected is the work that is the subject of inspection.

The judgment unit 222 judges whether or not work corresponding to a procedure to be performed is currently being performed for one or more fields in the inspection video sequentially acquired by the inspection video acquisition unit 221, by using the teacher data corresponding to the procedure to be performed among the teacher data. Note that a field is a still image in the video, and may also be called a frame or a still image. Also, the teacher data corresponding to the procedure to be performed is the teacher data that pairs with the current procedure identifier.

The judgment unit 222 acquires one or more fields, which are a portion of fields from the inspection video that satisfy predetermined conditions, and uses the one or more fields and the teacher data to judge whether the procedure of the work to be inspected for the inspection video is correct.

In other words, the judgment unit 222 performs a judgment process to judge whether or not the procedure of the work to be inspected for the inspection video is correct, for example, by using one or more fields contained in the inspection video and the teacher data. In addition, as a pre-processing of the judgment process, the judgment unit 222 usually performs a field acquisition process to acquire some fields that satisfy predetermined conditions from the inspection video.

The predetermined condition is, for example, that a predetermined object (e.g., a ramen bowl) is recognized by image recognition. The predetermined condition is, for example, that a machine learning prediction process is used to determine whether or not a procedure is correct.

The inspection image acquisition means 2221 that constitutes the judgment unit 222 performs field acquisition processing.

The inspection image acquisition means 2221 acquires, for example, one field of the inspection video acquired by the inspection video acquisition unit 221. It is preferable that the one field is a field that satisfies a predetermined condition. The predetermined condition is, for example, that a person's hand does not overlap with the frame of the ramen bowl, and that the bowl is present within the field.

The inspection image acquisition means 2221, for example, acquires two or more fields from the inspection video acquired by the inspection video acquisition unit 221. It is preferable that the two or more fields are fields that satisfy a predetermined condition. The predetermined condition is, for example, that a person's hand does not overlap with the frame of the ramen bowl, and that the bowl is present within the field.

The field acquisition process performed by the inspection image acquisition unit 2221 is, for example, one of the following two.
(1) Image recognition method

The inspection image acquisition means 2221 uses image recognition to determine whether or not each field in the inspection video contains a predetermined object (e.g., a ramen bowl). Then, the inspection image acquisition means 2221 acquires two or more fields that contain the predetermined object. Note that two or more fields means one or more fields for each of two or more steps. Also, since this image recognition technology is publicly known, detailed explanation will be omitted.

The inspection image acquisition means 2221, for example, determines whether or not a predetermined object (e.g., a ramen bowl) is present through image recognition, regardless of the procedure.

For example, the inspection image acquisition unit 2221 determines, for each step, whether or not each field of the video of the scene corresponding to the step identified by the current step identifier has a different object by image recognition.
(2) Machine learning method

The inspection image acquisition means 2221, for example, performs a usability determination for each field in the inspection video using machine learning prediction processing, and acquires one or more fields that correspond to the usability determination result of the usability determination.

The use/non-use judgment is a process of performing a machine learning prediction process using a second learning device and obtaining a prediction result of whether or not to use the device. The second learning device is a learning device for judging whether or not to use the device as a field for judging whether or not the procedure is correct.

The inspection image acquisition means 2221, for example, uses one second learning device regardless of the procedure to determine whether each field in the inspection video can be used, and acquires one or more fields that correspond to the result of the usability determination that the field can be used.

The inspection image acquisition means 2221, for example, for each step, uses a second learning device that is paired with the current step identifier to determine whether each field in the inspection video can be used, and acquires one or more fields that correspond to the result of the usability determination that the field can be used.

As a preprocessing step of (1) or (2) above, partial videos (scenes) for each step may be detected from the inspection video. Scene detection means detecting scene divisions. The inspection image acquisition means 2221, for example, determines whether a field in the video satisfies a predetermined scene division condition. If the scene division condition is satisfied, the field is determined to be the start field of a scene. Then, the field immediately before the start field of a scene is the end field of the previous scene.

The scene division conditions are, for example, that the similarity between the immediately preceding field and the current field of interest is below a threshold, that the magnitude of the motion vector obtained based on the difference between the immediately preceding field and the current field of interest is above a threshold, and that an object corresponding to the scene division can be recognized by image recognition processing. Note that various publicly known techniques can be used to divide scenes in video.

The judgment means 2222 performs judgment processing. The judgment processing is processing for judging whether or not the procedure of the work to be inspected for the inspection video is correct. In the judgment processing, the judgment means 2222 may use one field acquired by the inspection image acquisition means 2221 for each procedure, or may use two or more fields.

In other words, the judgment means 2222, for example, uses a field acquired by the inspection image acquisition means 2221 and a field (teacher data) corresponding to the current procedure identifier to judge whether the procedure corresponding to the field acquired by the inspection image acquisition means 2221 is a procedure identified by the current procedure identifier.

The judgment means 2222 also uses the two or more fields acquired by the inspection image acquisition means 2221 and the two or more fields corresponding to the current procedure identifier to judge whether the procedure corresponding to the two or more fields acquired by the inspection image acquisition means 2221 is a procedure identified by the current procedure identifier.

The determining unit 2222 performs the following two types of determination processing.
(1) Machine learning

The judgment means 2222 performs a machine learning prediction process using, for example, a field acquired by the inspection image acquisition means 2221 and a first learning device, and acquires a procedure identifier corresponding to the one field. Next, the judgment means 2222 judges whether the acquired procedure identifier matches the current procedure identifier in the current procedure storage unit 214. If they do not match, the judgment means 2222 acquires an error judgment result. Note that an error judgment result is a judgment result that the procedures are different. Note that the first learning device is given to a module that performs machine learning prediction processing together with the one field, and is a learning device for acquiring a procedure identifier by executing the module. Also, if it is determined that the result of the prediction process matches the current procedure identifier, the judgment means 2222 rewrites the current procedure identifier in the current procedure storage unit 214 to the procedure identifier of the next procedure. Note that the judgment means 2222 refers to the procedure information in the storage unit 21 to acquire the procedure identifier of the next procedure from the current procedure identifier. If it is determined to match the current procedure identifier, the procedure identified by the current procedure identifier has been performed.

The judgment means 2222 performs prediction processing by a machine learning algorithm using the two or more fields acquired by the inspection image acquisition means 2221 and the first learning device, and acquires a procedure identifier corresponding to the two or more fields. Next, the judgment means 2222 judges whether the acquired procedure identifier matches the current procedure identifier. If they do not match, the judgment means 2222 acquires an error judgment result. Note that the first learning device is given to a module that performs machine learning prediction processing together with the two or more fields, and is a learning device for acquiring a procedure identifier by executing the module. Also, if it is determined that the result of the prediction processing matches the current procedure identifier, the judgment means 2222 rewrites the current procedure identifier in the current procedure storage unit 214 to the procedure identifier of the next procedure. Note that the judgment means 2222 refers to the procedure information in the storage unit 21 and acquires the procedure identifier of the next procedure from the current procedure identifier. Also, the module may be an execution module, a function, a method, etc.

It is preferable that the judgment means 2222 does not judge an error when it judges that a procedure identified by the current procedure identifier has been performed and then acquires a procedure identifier corresponding to the same procedure.

Furthermore, when the determining means 2222 determines that the procedure identified by the current procedure identifier has been performed, it overwrites the current procedure identifier with the procedure identifier of the next procedure.
(2) When using field similarity

The determination means 2222 calculates the similarity between the fields (teacher data) corresponding to each of two or more procedures and one field for each procedure acquired by the test image acquisition means 2221, and uses the similarity to determine whether the procedure corresponding to the one field acquired by the test image acquisition means 2221 is a procedure identified by the current procedure identifier. If it is a procedure identified by the current procedure identifier, this means that the procedure has been performed. Note that the process of calculating the similarity between two fields is a publicly known technique, so a detailed explanation will be omitted.

The judgment means 2222, for example, calculates the similarity between a field of the current procedure acquired by the inspection image acquisition means 2221 and a field (teacher data) corresponding to the current procedure identifier. The judgment means 2222 also calculates the similarity between a field of the current procedure acquired by the inspection image acquisition means 2221 and a field (teacher data) corresponding to the procedure identifiers of each procedure after the current procedure. The judgment means 2222 then compares the two or more calculated similarities and judges whether the similarity with the field (teacher data) corresponding to the current procedure identifier is the maximum. If it is not the maximum, the judgment means 2222 obtains a judgment result that the procedures are different.

In addition, if the similarity with the field (teacher data) corresponding to the current procedure identifier is equal to or less than the threshold, the judgment means 2222 obtains an error judgment result.

It is preferable that the judgment means 2222 does not judge an error when it determines that a procedure identified by the current procedure identifier has been performed and then acquires a procedure identifier corresponding to the same procedure.

In addition, when the judgment means 2222 judges that the procedure identified by the current procedure identifier has been performed, it overwrites the current procedure identifier with the procedure identifier of the next procedure.

The output unit 23 outputs judgment result information regarding the judgment result of the judgment unit 222. The judgment result information is, for example, "normal" or "error." Here, output is a concept that includes displaying on a display, projecting using a projector, printing on a printer, outputting sound, transmitting to an external device, storing on a recording medium, and passing on the processing result to another processing device or another program, etc.

The storage unit 21, teacher data storage unit 211, learning device storage unit 212, threshold storage unit 213, and current procedure storage unit 214 are preferably non-volatile recording media, but can also be realized with volatile recording media.

The process by which information is stored in the storage unit 21, etc. is not important. For example, information may be stored in the storage unit 21, etc. via a recording medium, information transmitted via a communication line, etc. may be stored in the storage unit 21, etc., or information inputted via an input device may be stored in the storage unit 21, etc.

The processing unit 22, the inspection video acquisition unit 221, the judgment unit 222, the inspection image acquisition means 2221, and the judgment means 2222 can usually be realized by a processor, memory, etc. The processing procedures of the processing unit 22, etc. are usually realized by software, and the software is recorded in a recording medium such as a ROM. However, they may also be realized by hardware (dedicated circuitry). The processor may be a CPU, MPU, GPU, etc.

The output unit 23 may or may not include an output device such as a display or speaker. The output unit 23 may be realized by driver software for an output device, or by a combination of driver software for an output device and an output device, etc.

Next, an example of the operation of inspection system A will be described. First, the operation of camera 1 will be described. Camera 1 captures an image of an inspection target work having two or more steps being performed, and transmits the acquired images to inspection device 2 in real time. Camera 1 may also store the acquired images in a recording medium accessible by inspection device 2.

Next, an example of the operation of the inspection device 2 will be described using the flowchart in Figure 3.

(Step S301) The processing unit 22 determines whether or not to start the test. If the test is to be started, the process proceeds to step S302. If the test is not to be started, the process returns to step S301.

(Step S302) The processing unit 22 assigns 1 to the counter i.

(Step S303) The processing unit 22 refers to the procedure information in the storage unit 21 and determines whether or not the i-th procedure exists. If the i-th procedure exists, the process proceeds to step S304, and if the i-th procedure does not exist, the process proceeds to step S308. Note that at this point, it is preferable for the processing unit 22 to rewrite the current procedure identifier in the current procedure storage unit 214 to the procedure identifier of the i-th procedure.

(Step S304) The inspection video acquisition unit 221 performs field acquisition processing. An example of the field acquisition processing is described using the flowchart in FIG. 4.

(Step S305) The determination means 2222 performs a determination process. An example of the determination process is described using the flowcharts in Figures 5, 6, 7, and 8.

(Step S306) The processing unit 22 judges whether the result of the judgment process in step S305 is "normal" or "error". If it is "normal", the process proceeds to step S307, and if it is "error", the process proceeds to step S309.

(Step S307) The processing unit 22 increments the counter i by 1. Return to step S303.

(Step S308) The judgment unit 222 assigns the value "normal" to the judgment result information. Proceed to step S310.

(Step S309) The judgment unit 222 assigns the value "error" to the judgment result information.

(Step S310) The output unit 23 outputs the judgment result information. Return to step S301.

In the flowchart in Figure 3, processing ends when the power is turned off or an interrupt occurs to end processing.

In addition, in the flowchart of FIG. 3, the current procedure identifier in the current procedure storage unit 214 is updated to the next procedure identifier as i is incremented.

Next, an example of the field acquisition process in step S304 will be described using the flowchart in FIG.

(Step S401) The inspection video acquisition unit 221 determines whether or not a field of the candidate inspection target has been acquired. For example, the inspection video acquisition unit 221 determines whether or not a field of the candidate inspection target has been received from camera 1. If a field of the candidate inspection target has been acquired, the process proceeds to step S402, and if a field of the candidate inspection target has not been acquired, the process returns to step S401. Note that it is preferable for the inspection video acquisition unit 221 to, for example, sequentially receive the fields that make up the inspection video captured by camera 1. It is also preferable for the inspection video acquisition unit 221 to, for example, sequentially read out the fields of the video in a recording medium (not shown).

(Step S402) The inspection image acquisition means 2221 determines whether the field acquired in step S401 is a field to be inspected. If it is a field to be inspected, the process proceeds to step S403, and if it is not a field to be inspected, the process returns to step S401.

The inspection image acquisition means 2221 makes the determination, for example, by the above-mentioned field acquisition process (1) using a method based on image recognition or (2) using a method based on machine learning. When acquiring two or more fields, the inspection image acquisition means 2221 determines, for example, a field that has a predetermined relationship with the first acquired field to be the field to be inspected. Note that a field with a predetermined relationship is, for example, one or more consecutive fields within a time period within a threshold from the field that was initially determined to be the field to be inspected. A field with a predetermined relationship is, for example, a field whose similarity to the field that was initially determined to be the field to be inspected is equal to or greater than a threshold.

(Step S403) The inspection image acquisition means 2221 acquires the field acquired in step S401 and temporarily stores it in a buffer (not shown).

(Step S404) The inspection image acquisition means 2221 determines whether or not to acquire two or more fields. If two or more fields are to be acquired, the process proceeds to step S405, and if two or more fields are not to be acquired, the process returns to the upper level process. Note that whether or not to acquire two or more fields may be determined in advance.

(Step S405) The inspection image acquisition means 2221 determines whether or not to end the acquisition of fields. If it does, it returns to the upper process, and if it does not, it returns to step S401. Note that, for example, in one procedure, the number of fields to be acquired is determined, and when this number is reached, the inspection image acquisition means 2221 determines to end the acquisition of fields.

Next, an example of the first determination process in step S305 will be described with reference to the flowchart in FIG. 5. The first determination process is a case in which machine learning for multi-value classification is used.

(Step S501) The judgment means 2222 obtains the first learning device from the learning device storage unit 212.

(Step S502) The judgment means 2222 obtains one or more fields obtained in step S304.

(Step S503) The judgment means 2222 provides the first learning device acquired in step S501 and one or more fields acquired in step S502 to a module that performs machine learning prediction processing, and executes the module.

(Step S504) The judgment means 2222 obtains the procedure identifier that is the execution result in step S503.

(Step S505) The judgment means 2222 judges whether the procedure identifier acquired in step S504 matches the current procedure identifier in the current procedure storage unit 214. If they match, the process proceeds to step S506, and if they do not match, the process proceeds to step S507.

(Step S506) The judgment means 2222 assigns "normal" to the judgment result information. It returns to the upper process.

(Step S507) The judgment means 2222 assigns "error" to the judgment result information. It returns to the upper level process.

In the flowchart of FIG. 5, machine learning for multi-value classification was performed using a first learning device. That is, in the flowchart of FIG. 5, the judgment means 2222 used a common first learning device that is not related to the procedure. However, the judgment means 2222 may obtain a first learning device that pairs with the current procedure identifier. Such a first learning device is a learning device for binary classification (determines whether or not the procedure corresponds). Then, when the judgment means 2222 obtains a prediction result that corresponds to the procedure identified by the current procedure identifier as a result of the prediction process, it assigns "normal" to the judgment result information, and when the judgment means 2222 obtains a prediction result that corresponds to the procedure identified by the current procedure identifier, it assigns "error" to the judgment result information.

Next, an example of the second determination process in step S305 will be described with reference to the flowchart in FIG. 6. The second determination process is a case where machine learning for binary classification is used. Note that in the flowchart in FIG. 6, a description of the same steps as in the flowchart in FIG. 5 will be omitted.

(Step S601) The judgment means 2222 assigns 1 to counter i.

(Step S602) The judgment means 2222 judges whether or not a first learning device corresponding to the i-th procedure exists in the learning device storage unit 212. If the i-th first learning device exists, the process proceeds to step S603, and if the i-th first learning device does not exist, the process proceeds to step S608. Each of the two or more first learning devices corresponds to a procedure identifier.

(Step S603) The judgment means 2222 obtains the i-th first learning device from the learning device storage unit 212.

(Step S604) The judgment means 2222 obtains one or more fields obtained in step S304.

(Step S605) The judgment means 2222 provides the i-th first learning device and one or more fields to a module that performs machine learning binary classification prediction processing.

(Step S606) The judgment means 2222 executes the module. Then, the judgment means 2222 obtains the identification result (yes or no) of whether one or more fields correspond to the i-th procedure and a score, and temporarily stores the results in a buffer (not shown). Note that the score is output by the module that performs the prediction process, and is a value that indicates the likelihood of the identification result.

(Step S607) The judgment means 2222 increments the counter i by 1. Return to step S602.

(Step S608) The judgment means 2222 obtains a procedure identifier using the identification result and score temporarily accumulated in step S605. The judgment means 2222 obtains the procedure identifier corresponding to the first learning device with the highest score, which corresponds to the identification result that is applicable. Go to step S505.

Next, an example of the third judgment process of step S305 will be described with reference to the flowchart in FIG. 7. The third judgment process is a method that uses the similarity between the training data of the current procedure and the field to be inspected.

(Step S701) The determination means 2222 acquires teacher data paired with the current procedure identifier from the teacher data storage unit 211. Note that such teacher data is a field that constitutes a normal video.

(Step S702) The judgment means 2222 assigns 1 to counter i.

(Step S703) The judgment means 2222 judges whether the i-th field to be inspected exists. If the i-th field to be inspected exists, the process proceeds to step S704; if not, the process proceeds to step S706.

(Step S704) The judgment means 2222 calculates the similarity between the training data acquired in step S701 and the i-th field to be inspected, and temporarily stores the similarity in a buffer (not shown).

(Step S705) The judgment means 2222 increments the counter i by 1. Return to step S703.

(Step S706) The judgment means 2222 judges whether or not the one or more similarities acquired in step S704 satisfy a predetermined similarity condition. If the similarity condition is satisfied, the process proceeds to step S707, and if the similarity condition is not satisfied, the process proceeds to step S708. Note that the similarity condition is, for example, that the representative similarity is equal to or greater than a threshold value. Note that, if there is one similarity acquired in step S704, the representative similarity is that similarity, and if there are two or more similarities acquired in step S704, the representative value of the two or more similarities (for example, the average value, the top value, the median value, the bottom value).

(Step S707) The judgment means 2222 assigns "normal" to the judgment result information. It returns to the upper process.

(Step S708) The judgment means 2222 assigns "error" to the judgment result information. It returns to the upper level process.

Next, an example of the fourth judgment process of step S305 will be described with reference to the flowchart in FIG. 8. The fourth judgment process is a method that uses the similarity between the training data of each of two or more procedures and the field to be inspected.

(Step S801) The judgment means 2222 assigns 1 to counter i.

(Step S802) The judgment means 2222 judges whether or not the i-th procedure to be compared exists. If the i-th procedure exists, the process proceeds to step S803, and if not, the process proceeds to step S809. The i-th procedure to be compared is, for example, the current procedure and any subsequent procedures.

(Step S803) The determination means 2222 acquires training data that is paired with the procedure identifier of the i-th procedure. Note that the training data is, for example, a field that constitutes a normal video.

(Step S804) The judgment means 2222 assigns 1 to counter j.

(Step S805) The judgment means 2222 judges whether the jth field to be inspected exists. If the jth field to be inspected exists, the process proceeds to step S806; if not, the process proceeds to step S808.

(Step S806) The judgment means 2222 calculates the similarity between the training data acquired in step S803 and the jth field to be inspected.

(Step S807) The judgment means 2222 increments the counter j by 1. Return to step S805.

(Step S808) The judgment means 2222 increments the counter i by 1. Return to step S802.

(Step S809) The judgment means 2222 judges whether the two or more similarities calculated in step S806 satisfy the similarity condition. If the similarity condition is satisfied, the process proceeds to step S810, and if the similarity condition is not satisfied, the process proceeds to step S811. Note that the similarity condition is, for example, that the procedure identifier paired with the training data corresponding to the maximum similarity matches the current procedure identifier. The similarity condition is, for example, that the procedure identifier of the procedure with the largest representative value (e.g., average, median, maximum, minimum) of the two or more similarities corresponding to each procedure matches the current procedure identifier.

(Step S810) The judgment means 2222 assigns "normal" to the judgment result information. It returns to the upper process.

(Step S811) The judgment means 2222 assigns "error" to the judgment result information. It returns to the upper level process.

As described above, according to this embodiment, by photographing a task that has two or more steps, it is possible to determine whether the task is being performed correctly according to the planned steps.

The processing in this embodiment may be realized by software. This software may be distributed by software download or the like. This software may also be recorded on a recording medium such as a CD-ROM and distributed. This also applies to other embodiments in this specification. The software that realizes the inspection device 2 in this embodiment is a program as follows. In other words, this program causes a computer to function as an inspection video acquisition unit that acquires an inspection video, which is a video of an inspection target work that is an inspection target work having two or more steps, being performed, a judgment unit that uses teacher data based on a normal video, which is a video of a normal work that is a correct work having two or more steps, to judge whether the procedure of the inspection target work for the inspection video is correct or not, and an output unit that outputs judgment result information regarding the judgment result of the judgment unit.

(Embodiment 2)
In this embodiment, a description will be given of a learning device 3 that acquires a learning device used by the inspection device 2. Note that the learning device here is the above-mentioned first learning device or second learning device.

Figure 9 is a block diagram of the learning device 3 in this embodiment. The learning device 3 includes a learning storage unit 31 and a learning processing unit 32. The learning processing unit 32 includes a teacher data acquisition unit 321 and a learning unit 322.

The learning storage unit 31 stores various types of information. The various types of information include, for example, one or more normal videos, teacher data, a first learning device, and a second learning device. A normal video is a video captured when a normal task, which is a correct task having two or more steps, is being performed.

The learning storage unit 31 may store two or more teacher data for constructing a second learning device. That is, such teacher data is, for example, a field selected by the user from a normal video, and is a field (positive example) used in the judgment process of the judgment means 2222. Also, such teacher data is, for example, a field selected by the user from a normal video, and is a field (negative example) not used in the judgment process of the judgment means 2222.

Each of the two or more pieces of teacher data for constituting the second learning device may be associated with a procedure identifier. Such teacher data may have a procedure identifier. Furthermore, such teacher data may be a pair of a field selected by the user from the normal video and a procedure identifier input by the user.

The learning processing unit 32 performs various types of processing. For example, various types of processing are processing performed by the teacher data acquisition unit 321 and the learning unit 322.

The teacher data acquisition unit 321 acquires one or more fields that satisfy a predetermined condition from the normal video for each of two or more procedures. The teacher data acquisition unit 321 also acquires a procedure identifier that identifies the procedure corresponding to the one or more fields. Next, the teacher data acquisition unit 321 acquires two or more pieces of teacher data that are pairs of the acquired one or more fields and procedure identifiers for each of two or more procedures. Note that the one or more fields that make up the teacher data are fields that make up the video captured during the work of the procedure identified by the paired procedure identifier.

The process by which the teacher data acquisition unit 321 acquires fields may be the same as the field acquisition process performed by the test image acquisition means 2221. That is, the teacher data acquisition unit 321 acquires one or more fields to be used for learning from a normal video, for example, by either (1) the image recognition method or (2) the machine learning method of the test image acquisition means 2221 described above.

The learning unit 322 performs a machine learning learning process on two or more pieces of teacher data, obtains a first learning device, and stores the first learning device. Note that the first learning device may be stored in, for example, the learning storage unit 31, but the storage location is not critical.

The learning unit 322 performs a machine learning learning process on, for example, two or more pieces of teacher data, acquires a first learning device, and accumulates the first learning device. Such a first learning device is a learning device used for prediction processing that accepts a field and outputs a procedure identifier. Such a first learning device is a learning device used when performing multi-value classification. As described above, any machine learning algorithm is acceptable.

The learning unit 322, for example, learns, for each procedure identifier, one or more fields (positive examples) that constitute the video of the procedure identified by the procedure identifier and one or more fields (negative examples) that do not constitute the video of the procedure by using a machine learning learning process, acquires a second learning device, and accumulates the second learning device. Note that the second learning device is accumulated in pairs with the procedure identifier for each procedure identifier. The second learning device is a learning device used for prediction processing that accepts a field, determines whether or not it corresponds to the procedure corresponding to the second learning device, and outputs a score. The second learning device is a learning device used when performing binary classification. Note that, as described above, any machine learning algorithm is used.

The learning unit 322 performs a machine learning learning process on two or more pieces of teacher data, acquires a second learning device, and stores the second learning device. The second learning device may be stored in, for example, the learning storage unit 31, but it does not matter. The two or more pieces of teacher data are teacher data for constituting the second learning device. The second learning device is a learning device for classifying whether or not to use the procedure in the judgment process for determining the procedure, and is a learning device for performing binary classification.

The learning unit 322, for example, acquires a second learning device for each procedure and accumulates the second learning device. In such a case, the learning unit 322, for example, for each procedure, provides two or more pieces of teacher data to a module that performs machine learning learning processing, with one or more pieces of teacher data paired with a procedure identifier that identifies the procedure as positive examples and one or more pieces of teacher data that are not paired with a procedure identifier that identifies the procedure as negative examples, executes the module, and acquires a second learning device. Note that, as described above, any machine learning algorithm is acceptable.

The learning storage unit 31 is preferably a non-volatile recording medium, but can also be realized as a volatile recording medium.

The process by which information is stored in the learning storage unit 31, etc. is not important. For example, information may be stored in the learning storage unit 31, etc. via a recording medium, information transmitted via a communication line, etc. may be stored in the learning storage unit 31, etc., or information inputted via an input device may be stored in the learning storage unit 31, etc.

The learning processing unit 32, teacher data acquisition unit 321, and learning unit 322 can usually be realized by a processor, memory, etc. The processing procedures of the learning processing unit 32, etc. are usually realized by software, and the software is recorded on a recording medium such as a ROM. However, they may also be realized by hardware (dedicated circuitry). The processor may be a CPU, MPU, GPU, etc.

Next, a first operation example of the learning device 3 will be described with reference to the flowchart in FIG. 10. In the flowchart in FIG. 10, a first learning device for multi-value classification is obtained. Such a first learning device is a learning device that receives one or more fields as input and outputs a procedure identifier.

(Step S1001) The learning processing unit 32 assigns 1 to counter i.

(Step S1002) The learning processing unit 32 determines whether or not the i-th video exists in the learning storage unit 31. If the i-th video exists, the process proceeds to step S1003; if not, the process proceeds to step S1009. Note that the i-th video is usually the i-th normal video.

(Step S1003) The teacher data acquisition unit 321 assigns 1 to counter j.

(Step S1004) The teacher data acquisition unit 321 determines whether the jth procedure exists. If the jth procedure exists, the process proceeds to step S1005; if not, the process proceeds to step S1008.

(Step S1005) The teacher data acquisition unit 321 performs a field acquisition process corresponding to the jth procedure. Note that the field acquisition process here is a process for acquiring one or more fields corresponding to the jth procedure. An example of the field acquisition process here is described using the flowchart in FIG. 12.

(Step S1006) The teacher data acquisition unit 321 associates the one or more fields acquired in step S1005 with the procedure identifier of the jth procedure and accumulates them in the learning storage unit 31.

(Step S1007) The learning processing unit 32 increments the counter j by 1. Return to step S1004.

(Step S1008) The learning processing unit 32 increments the counter i by 1. Return to step S1002.

(Step S1009) The learning unit 322 provides two or more pieces of teacher data stored in the learning storage unit 31 to a machine learning learning processing module, executes the module, and obtains a first learning device.

(Step S1010) The learning unit 322 accumulates the first learning device acquired in step S1009. The process ends.

Next, a second operation example of the learning device 3 will be described with reference to the flowchart of FIG. 11. In the flowchart of FIG. 11, a first learning device for binary classification is obtained for each procedure identifier. Such a first learning device is a learning device that receives one or more fields as input and outputs whether or not the procedure corresponds to the first learning device. In the flowchart of FIG. 11, the description of the same steps as those in the flowchart of FIG. 10 will be omitted.

(Step S1101) The learning unit 322 assigns 1 to the counter k.

(Step S1102) The learning unit 322 determines whether the kth procedure exists. If the kth procedure exists, the process proceeds to step S1103; if the kth procedure does not exist, the process ends.

(Step S1103) The learning unit 322 obtains one or more fields (positive examples) that are paired with the procedure identifier of the kth procedure from the learning storage unit 31.

(Step S1104) The learning unit 322 retrieves one or more fields (negative examples) that are not paired with the procedure identifier of the kth procedure from the learning storage unit 31.

(Step S1105) The learning unit 322 provides one or more positive examples and one or more negative examples to a module that performs machine learning learning processing, executes the module, and obtains a first learning device.

(Step S1106) The learning unit 322 stores the first learning device acquired in step S1105 in a pair with the procedure identifier of the kth procedure.

(Step S1107) The learning unit 322 increments the counter k by 1. Return to step S1102.

Next, an example of the field acquisition process in step S1005 will be described with reference to the flowchart in FIG.
(Step S1201) The teacher data acquisition unit 321 assigns X to a counter k. Note that X is the counter value in the i-th moving image to be processed. X is not initialized in the loop processing from S1004 to S1007 in Fig. 10 and Fig. 11. The initial value of X is "1".
(Step S1202) The teacher data acquisition unit 321 determines whether or not the kth field exists in the moving image to be processed. If the kth field exists, the process proceeds to step S1203, and if the kth field does not exist, the process returns to the upper process.
(Step S1203) The teacher data acquisition unit 321 acquires the k-th field of the moving image to be processed.
(Step S1204) The teacher data acquisition unit 321 judges whether the k-th field is a field to be learned. If it is a field to be learned, the process proceeds to step S1205, and if it is not a field to be learned, the process proceeds to step S1207.

The method for determining whether the kth field is a field to be learned is (1) an image recognition method or (2) a machine learning method by the inspection image acquisition means 2221 described above, and detailed explanation will be omitted.
(Step S1205) The teacher data acquiring unit 321 temporarily stores the k-th field in a buffer (not shown).
(Step S1206) The teacher data acquisition unit 321 judges whether or not to end the acquisition of fields in the jth procedure. If it is to end, it returns to the upper process, and if it is not to end, it goes to step S1207. If the maximum number of fields to be acquired in the jth procedure is determined, the teacher data acquisition unit 321 judges that the acquisition of fields in the jth procedure is to end when the maximum number of fields is accumulated in step S1205. However, the maximum number of fields to be acquired does not have to be determined. In addition, after accumulating a field to be learned one or more times, if it is determined in step S1204 that the field is not to be learned, it is preferable that the teacher data acquisition unit 321 judges that the acquisition of fields in the jth procedure is to end as a transition to the next procedure.
(Step S1207) The teacher data acquiring unit 321 increments the counter k by 1. The process returns to step S1202.

The specific operation of the learning device 3 is described below.

Now, let us say that the learning storage unit 31 of the learning device 3 stores the procedure information (1, pour in the soup), (2, pour in the noodles), and (3, pour in the toppings). In other words, the procedure information is information in which the procedure identifiers "1", "2", and "3" correspond to the character strings "Pour in the soup", "Pour in the noodles", and "Pour in the toppings", which indicate the three procedures that make up the work of making ramen.

The learning storage unit 31 of the learning device 3 also stores a field management table shown in FIG. 13. The field management table stores fields (still images) that are teacher data for configuring the first learning device. The fields in FIG. 13 are fields acquired by the field acquisition process of the learning processing unit 32. The fields correspond to procedure identifiers. In FIG. 13, "ID" is information that identifies a record.

In such a situation, the learning unit 322 obtains two or more pieces of training data having pairs of fields and procedure identifiers from the field management table shown in FIG. 13.

Next, the learning unit 322 provides a large amount of teacher data to a machine learning learning processing module, executes the module, and acquires and accumulates a first learning device. The first learning device is a learning device for multi-value classification. The first learning device is a learning device used for prediction processing that accepts a field and outputs a procedure identifier. Here, it is assumed that the first learning device is accumulated in the learning device storage unit 212 of the inspection device 2.

Next, a specific example of the operation of the above-mentioned inspection system A will be described. A conceptual diagram of the inspection system A is shown in FIG. 1. Assume that the task to be inspected is the task of making ramen. Also, assume that the storage unit 21 stores procedure information indicating the task of making ramen (1, pour in the soup) (2, pour in the noodles) (3, pour in the toppings).

Furthermore, camera 1 is installed in a position where it can capture the ramen making process (particularly, the inside of the ramen bowl).

In this situation, the following two specific examples will be described. Specific Example 1 is a case where the work is performed normally. Specific Example 2 is a case where the work is not performed normally. It is assumed that the storage unit 21 of the inspection device 2 stores the procedure information (1, pour soup), (2, pour noodles), (3, pour toppings). In other words, the normal work is to perform the steps in the order "Pour soup", "Pour noodles", and "Pour toppings".

(Specific Example 1)
Now, worker A is making ramen. He puts the soup, noodles, and toppings (green onions, boiled eggs, etc.) into the bowl in that order.

The camera 1 then captures images of worker A's work and transmits the captured video to the inspection device 2 in sequence.

Next, the inspection video acquisition unit 221 of the inspection device 2 sequentially receives the videos of the work to be inspected.

The determination unit 222 of the inspection device 2 operates as follows (1) to (9).
(1) The inspection image acquisition means 2221 of the judgment unit 222 acquires a field that meets a condition (including the entire frame of the bowl) through image recognition processing. Next, the judgment means 2222 provides the first learning device in the learning device storage unit 212 and the field acquired by the inspection image acquisition means 2221 to a machine learning prediction processing module, executes the module, and acquires a step identifier "1." Note that "1" is the identifier for the step "Pour soup in."

Next, the determination means 2222 obtains the current procedure identifier "1" from the current procedure storage unit 214. Next, the determination means 2222 determines that the procedure identifier "1" obtained by the prediction process matches the current procedure identifier "1", and assigns "normal" to the determination result information. Next, the determination means 2222 refers to the procedure information in the storage unit 21 and determines that the next procedure exists. Next, the determination means 2222 updates the current procedure identifier to the next procedure identifier "2".
(2) Next, the inspection image acquisition unit 2221 acquires the next field that meets the conditions from the inspection target video. Next, the determination unit 2222 provides the first learning device in the learning device storage unit 212 and the field acquired by the inspection image acquisition unit 2221 to a machine learning prediction processing module, executes the module, and acquires a procedure identifier "1".

Next, the judgment means 2222 acquires the current procedure identifier "2" from the current procedure storage unit 214. Next, since the procedure identifier "1" acquired by the prediction process does not match the current procedure identifier "2", but matches the immediately previous current procedure identifier "1", the judgment means 2222 assigns "normal" to the judgment result information. In other words, when a field indicating the continuation of an operation within one procedure is detected, "normal" is assigned to the judgment result information. Note that after the procedure identifier acquired by the prediction process matches the current procedure identifier and after the current procedure identifier is updated, if the judgment means 2222 acquires a procedure identifier by the prediction process that matches the current procedure identifier immediately before the update, it judges that the normal procedure is continuing, judges that it is normal, and does not judge that it is an error.
(3) Usually, the process of (2) is repeated.
(4) The inspection image acquisition unit 2221 does not acquire fields that do not meet the conditions through image recognition processing (for example, a field in which the edge of a pot overlaps with the worker's hand). In addition, there are usually consecutive fields that do not meet the conditions.
(5) Next, the inspection image acquisition means 2221 acquires a field that meets the conditions from the inspection target video. Next, the determination means 2222 provides the first learning device in the learning device storage unit 212 and the field acquired by the inspection image acquisition means 2221 to a machine learning prediction processing module, executes the module, and acquires a step identifier "2." Note that "2" is the identifier of the step "adding noodles."

Next, the determination means 2222 obtains the current procedure identifier "2" from the current procedure storage unit 214. Next, the determination means 2222 determines that the procedure identifier "2" obtained by the prediction process matches the current procedure identifier "2", and assigns "normal" to the determination result information. Next, the determination means 2222 refers to the procedure information in the storage unit 21 and determines that the next procedure exists. Next, the determination means 2222 updates the current procedure identifier to the next procedure identifier "3".
(6) Next, the inspection image acquisition unit 2221 acquires the next field that meets the conditions from the inspection target video. Next, the determination unit 2222 provides the first learning device in the learning device storage unit 212 and the field acquired by the inspection image acquisition unit 2221 to a machine learning prediction processing module, executes the module, and acquires a procedure identifier "2".

Next, the determination means 2222 obtains the current procedure identifier "3" from the current procedure storage unit 214. Next, the determination means 2222 assigns "normal" to the determination result information because the procedure identifier "2" obtained by the prediction process does not match the current procedure identifier "3", but matches the immediately preceding current procedure identifier "2".
(7) Usually, the process of (6) is repeated.
(8) The inspection image acquisition means 2221 does not acquire fields that do not meet the conditions through image recognition processing.
(9) Next, the inspection image acquisition means 2221 acquires a field that meets the conditions from the inspection target video. Next, the determination means 2222 provides the first learning device in the learning device storage unit 212 and the field acquired by the inspection image acquisition means 2221 to a machine learning prediction processing module, executes the module, and acquires a step identifier "3." Note that "3" is the identifier of the step "adding ingredients."

Next, the judgment means 2222 obtains the current procedure identifier "3" from the current procedure storage unit 214. Next, the judgment means 2222 determines that the procedure identifier "3" obtained by the prediction process matches the current procedure identifier "3", and assigns "normal" to the judgment result information. Next, the judgment means 2222 refers to the procedure information in the storage unit 21, and determines that all procedures have been completed.

Next, the output unit 23 outputs the judgment result information.

(Specific Example 2)
Now, worker B is making ramen. First, worker B puts the noodles into the ramen bowl.

Next, the inspection video acquisition unit 221 of the inspection device 2 sequentially receives the videos of the work to be inspected.

Then, the inspection device 2 operates as follows:

In other words, the inspection image acquisition means 2221 of the judgment unit 222 acquires a field that meets the condition (the entire frame of the bowl is included) through image recognition processing. Next, the judgment means 2222 provides the first learning device in the learning device storage unit 212 and the field acquired by the inspection image acquisition means 2221 to a machine learning prediction processing module, executes the module, and acquires a procedure identifier "2". Note that "2" is the identifier for the procedure "adding noodles".

Next, the judgment means 2222 obtains the current procedure identifier "1" from the current procedure storage unit 214. Next, the judgment means 2222 determines that the procedure identifier "2" obtained by the prediction process does not match the current procedure identifier "1", and assigns "error" to the judgment result information.

Next, the output unit 23 outputs the judgment result information. The output unit 23 outputs a buzzer sound as the output of the judgment result information "error", to inform the worker B that the procedure is different. In the event of an error, it is preferable for the output unit 23 to output a voice or sound corresponding to the judgment result information "error" and to inform the worker B that an error has occurred.

As described above, according to this embodiment, a learning device that can be used by the inspection device is obtained. The learning device is the first learning device or the second learning device.

Furthermore, according to the above embodiment, by photographing a task that has two or more steps, it is possible to determine in real time whether the task is being performed correctly and to inform the worker of the result of the determination.

The process in this embodiment may be realized by software. This software may be distributed by software download or the like. This software may be recorded on a recording medium such as a CD-ROM and distributed. This also applies to other embodiments in this specification. The software for realizing the learning device 3 in this embodiment is a program as follows. That is, this program includes a computer that can access a teacher data storage unit in which one or more teacher data based on a normal video, which is a video captured of a normal operation, which is a correct operation having two or more steps, is stored, a teacher data acquisition unit that acquires one or more fields that satisfy a predetermined condition from the normal video for each of the two or more steps, and acquires two or more teacher data that are pairs of a procedure identifier that identifies a procedure corresponding to the field and the field,
This is a program for functioning as a learning unit that performs machine learning learning processing on the two or more teacher data, acquires a learning device, and accumulates the learning device.

(Embodiment 3)
In this embodiment, an inspection device is described that acquires a video of a series of work steps having two or more steps and equipment-related information, uses the video and the equipment-related information to inspect whether the work is appropriate, and outputs the inspection results.

In this embodiment, we will also describe an inspection device that inspects video after inspecting device-related information.

In this embodiment, we will also describe an inspection device that inspects device-related information after inspecting a video.

Furthermore, in this embodiment, we will explain an inspection device that uses a machine learning algorithm to inspect whether work is appropriate and output the inspection results.

Figure 14 is a conceptual diagram of an inspection system B in this embodiment. Inspection system B includes one or more cameras 1, an inspection device 4, and one or more devices C. Camera 1 corresponds to device C. However, the camera 1 and device C do not have to have a one-to-one correspondence, and may have, for example, a many-to-one or one-to-many correspondence.

The device C is a device that performs work. The device C is, for example, a robot or a computer. The robot may be an industrial robot, a humanoid robot, or the like, and the type does not matter. The robot is, for example, a so-called drone. The drone is an air vehicle. The drone here is preferably equipped with a camera 1 that takes pictures. The drone here is preferably equipped with a function to obtain and transmit position information that specifies the flying position. The position information is preferably information that indicates a position in three-dimensional space (e.g., (latitude, longitude, altitude)). The drone here is preferably equipped with a function to transport luggage and carry it to a specified location. The computer is, for example, a so-called personal computer, a tablet terminal, a smartphone, a server, or the like, and the type does not matter. The device C may be an apparatus that is installed in a fixed location, or may be a moving apparatus (moving body). The device C may be integrated with the camera 1.

Figure 15 is a block diagram of an inspection system B in this embodiment. The inspection device 4 constituting the inspection system B includes a storage unit 41, a processing unit 42, and an output unit 23. The storage unit 41 includes a teacher data storage unit 211, a learning device storage unit 212, a threshold storage unit 213, a normal device related information storage unit 411, and a current procedure storage unit 214. The processing unit 42 includes an inspection video acquisition unit 221, a device related information acquisition unit 421, and a judgment unit 422. The judgment unit 422 includes a first judgment means 4221 and a second judgment means 4222.

Various types of information are stored in the storage unit 41 of the inspection device 4. Examples of the various types of information include teacher data, a learning device, a threshold value, information related to normal devices (described below), a current procedure identifier, and procedure information.

Two or more pieces of teacher data are stored in the teacher data storage unit 211. Typically, one or more pieces of teacher data exist for each procedure in the teacher data storage unit 211. The teacher data for each procedure is typically associated with a procedure identifier.

The training data is, for example, information used to compare with part or all of the test video. The training data is, for example, data based on a normal video (referred to as a positive example as appropriate). The training data includes, for example, one or more fields that make up the normal video.

The training data (positive examples) here is, for example, information based on one or more pieces of normal device-related information. The training data here is, for example, a vector whose elements are two or more pieces of normal device-related information. Normal device-related information is device-related information when normal work is performed.

The training data (positive examples) here may be information having data based on a normal video and data based on one or more pieces of normal device-related information. The training data is, for example, a vector whose elements are one or more feature amounts obtained from one or more fields constituting a normal video and one or more pieces of normal device-related information. The positive examples may include a flag indicating that the task is normal. The positive examples may, for example, have a procedure identifier or may be associated with the procedure identifier.

The normal video may be information in which a simulation device (not shown) simulates normal operations performed by device C and records the resulting video. The abnormal video may be information in which a simulation device (not shown) simulates abnormal operations that may be performed by the device and records the resulting video. Note that the simulation device that simulates operations that may be performed by device C and outputs the simulation results is a publicly known technology, so a detailed description will be omitted.

The device-related information is input device-related information, generator-related information, or output device-related information. The input device-related information is information input to the device C for the performance of the work to be inspected. The generator-related information is information generated within the device C in the performance of the work. The output device-related information is information output from the device C as a result of the performance of the work. The device-related information is, for example, an instruction or data. The input device-related information is, for example, an instruction given to the device C, or an instruction or information input to the device C by a user. The generator-related information is, for example, an event generated by the information input to the device C. When the device C is a moving body such as a drone, the generator-related information is, for example, a signal generated internally by the device C to place the baggage at the scheduled transportation location (for example, a signal to open the arm that is catching the baggage). The output device-related information is, for example, information transmitted from the device C to another device, a screen or information displayed by the device C, or one or more pieces of location information acquired by the device C (for example, location information identifying the location where the drone flew).

The one or more teacher data may be information (referred to as negative examples as appropriate) including data based on an abnormal video, which is a video of an abnormal operation, and abnormal device-related information. The negative examples are, for example, information acquired using an abnormal video and abnormal device-related information. The negative examples are, for example, one or more fields constituting an abnormal video and abnormal device-related information. The negative examples are, for example, a vector whose elements are one or more feature amounts acquired from one or more fields constituting an abnormal video and one or more pieces of abnormal device-related information. The negative examples may include, for example, a flag indicating that the operation is abnormal. The negative examples may have, for example, a procedure identifier, or may be associated with the procedure identifier. The abnormal device-related information is abnormal information input to device C, abnormal information that has occurred within device C, or abnormal information output from device C.

The learning device storage unit 212 stores a third learning device in addition to or instead of the teacher data storage unit 211. The learning device storage unit 212 may also store the first learning device or the second learning device described above.

The learning device storage unit 212 may include a third learning device for each procedure. In other words, each of the two or more third learning devices may be associated with a procedure identifier.

The third learning device is a learning device used when determining whether the procedure of the work to be inspected is correct. The third learning device is a learning device configured by performing a learning process on two or more pieces of training data using a machine learning algorithm. The training data here includes, for example, normal device related information or abnormal device related information. In other words, the training data that is a positive example includes, for example, data based on normal videos and normal device related information. Furthermore, the training data that is a negative example includes, for example, data based on abnormal videos and abnormal device related information.

When only one third learner is stored in the learner storage unit 212, the third learner is, for example, a learner that uses each element of a vector obtained from video-based data and one or more pieces of device-related information as explanatory variables and a procedure identifier as an objective variable. However, the third learner may also be a learner that uses each element of a vector obtained from video-based data as explanatory variables and a procedure identifier as an objective variable. The third learner may also be a learner that uses each element of a vector obtained from two or more pieces of device-related information as explanatory variables and a procedure identifier as an objective variable.

In addition, when two or more third learners paired with a procedure identifier are stored in the learner storage unit 212, each third learner is, for example, a learner that uses each element of a vector obtained from video-based data and one or more pieces of device-related information as an explanatory variable, and the determination result of whether or not the procedure is relevant as an objective variable. Furthermore, such a third learner may be, for example, a learner that uses each element of a vector obtained from video-based data as an explanatory variable, and the determination result of whether or not the procedure is relevant as an objective variable. Furthermore, such a third learner may be, for example, a learner that uses each element of a vector obtained from two or more pieces of device-related information as an explanatory variable, and the determination result of whether or not the procedure is relevant as an objective variable.

It is also preferable that the third learning device is a learning device acquired by the learning device 5 described below.

As described above, the machine learning algorithm may be, for example, deep learning, random forest, decision tree, SVM, etc. However, the machine learning algorithm is not important. The fact that the machine learning algorithm is not important applies to both the learning process and the prediction process.

The threshold storage unit 213 stores a threshold. The threshold is, for example, a threshold regarding the similarity between a vector obtained from a field obtained from the inspection video and one or more pieces of device-related information obtained by the device-related information acquisition unit 421, and the teacher data which is a vector. The threshold is, for example, a threshold regarding the similarity between a field obtained from the inspection video and a field included in the normal video. The threshold is, for example, a threshold regarding the similarity between a vector whose elements are one or more pieces of device-related information obtained by the device-related information acquisition unit 421, and a vector whose elements are one or more pieces of normal device-related information.

The thresholds may be stored for each procedure. That is, each of the one or more thresholds may be associated with a procedure identifier.

The normal device related information storage unit 411 stores one or more pieces of normal device related information. The normal device related information is device related information when normal work is performed. The normal device related information corresponds to a procedure identifier, for example. The normal device related information may be information constituting the teacher data of positive examples. In other words, the normal device related information storage unit 411 may be included in the teacher data storage unit 211.

The normal device related information storage unit 411 stores, for example, a vector whose elements are two or more pieces of device related information obtained when normal work is performed.

The processing unit 42 performs various types of processing. The various types of processing are, for example, processing performed by the inspection video acquisition unit 221, the device-related information acquisition unit 421, and the judgment unit 422.

The inspection video acquisition unit 221 may acquire, for example, an image that is a capture of the screen of device C. The inspection video acquisition unit 221 may acquire, for example, two or more images that are captures of the screen of device C. In such a case, camera 1 is not required.

For example, if device C is a moving object such as a drone, inspection video acquisition unit 221 may acquire both images acquired by an external camera 1 and an image acquired by a camera 1 equipped on the drone, or may acquire only images acquired by a camera 1 equipped on the drone. For example, inspection video acquisition unit 221 acquires images taken by camera 1 that are synchronized with the timing at which a signal indicating that the drone has released the cargo it is carrying (placed at a specified location on the ground or a building) is received, or that are one or more images close to the timing and that satisfy a predetermined condition.

The device-related information acquisition unit 421 acquires one or more pieces of device-related information from device C. The device-related information acquisition unit 421 typically acquires device-related information from an operating device C. The technology for acquiring device-related information from device C is publicly known, and therefore a detailed description thereof will be omitted.

The device-related information acquisition unit 421 acquires, for example, one or more pieces of input device-related information input to device C. The device-related information acquisition unit 421 acquires, for example, one or more pieces of generating device-related information generated within device C. The device-related information acquisition unit 421 acquires, for example, one or more pieces of output device-related information output by device C.

The device-related information acquisition unit 421 may receive device-related information from device C, or may receive the information from a device (not shown) that receives the information transmitted from device C. There is no restriction on the method by which the device-related information acquisition unit 421 acquires the device-related information.

It is preferable that the device-related information acquisition unit 421 acquires device-related information for each procedure in association with a procedure identifier.

It is preferable that the device-related information acquisition unit 421 acquires the field that the judgment unit 422 uses for judgment and the device-related information at the time corresponding to the time of the field. In other words, it is preferable that the field that the judgment unit 422 uses for judgment and the device-related information are synchronized.

The judgment unit 422 judges whether the procedure of the work to be inspected is correct using teacher data based on the normal video and normal device-related information. The judgment unit 422 judges whether the procedure of the work to be inspected is correct for the inspection video and device-related information using teacher data based on the normal device-related information and normal video.

For example, the judgment unit 422 preferably performs a machine learning prediction process using a second learning device to determine whether or not each field in the inspection video should be used as a field for determining whether or not the procedure is correct, obtains one or more fields that correspond to the prediction result that the field will be used, and performs the following judgment process using the one or more fields.

When the device C is a mobile object such as a drone, the determination unit 422 determines whether the transport of the luggage has been performed normally, for example, using one or more images acquired by the inspection video acquisition unit 221 and a positive example image. The determination unit 422 determines that the transport of the luggage has been performed normally, for example, when an image having a similarity to the positive example image equal to or greater than a threshold is included in one or more images acquired by the inspection video acquisition unit 221. The determination unit 422 also performs a learning process using a machine learning algorithm, for example, using each of the one or more images acquired by the inspection video acquisition unit 221 and a learning device, and obtains a prediction result indicating whether each image is normal.

When device C is a moving object such as a drone, for example, the determination unit 422 may determine whether device C has properly held the luggage using, for example, one or more images synchronized with a signal for holding the luggage and acquired by the inspection video acquisition unit 221 and a positive example image. Such a determination method may use the similarity of the images as described above, or may use a machine learning algorithm.

When the device C is a mobile object such as a drone, the determination unit 422 determines whether the device C is moving normally along the planned flight route by using, for example, one or more pieces of position information that are the output device related information of the device C and one or more pieces of position information of the positive example. The one or more pieces of position information that are the output device related information of the device C are information that specifies the route that has actually been flown. The one or more pieces of position information of the positive example are information that specifies the planned flight route. The determination unit 422, for example, calculates the similarity between a vector having one or more pieces of position information that are the output device related information and a vector having one or more pieces of position information of the positive example, and determines that the similarity is normal if the similarity is equal to or greater than a threshold value. The determination unit 422, for example, calculates the distance between each piece of position information that is the output device related information and a line in three-dimensional space that specifies the planned flight route, and determines that the distance is normal if the distance is within a threshold value or less than the threshold value. Note that the method by which the determination unit 422 determines whether the flight route is normal is not important.

The judgment unit 422 judges whether or not the procedure of the work to be inspected is correct, for example, by using one of the following algorithms. The first method is a method in which both the video and the equipment-related information are used at the same time. The second method is a method in which a first inspection is performed using the equipment-related information, and then a second inspection is performed using the video. The third method is a method in which a second inspection is performed using the video, and then a first inspection is performed using the equipment-related information. Each of these methods will be explained below.
(1) First Method

In the first method, there are a case where a third learning device for each procedure is used, a case where one third learning device is used, and a case where vector similarity is used.
(1-2) When using the third learning module for each step

The judgment unit 422 acquires one field of the inspection video acquired by the inspection video acquisition unit 221 for each procedure, and acquires one or more feature amounts of the field. The judgment unit 422 also acquires one or more pieces of device-related information acquired by the device-related information acquisition unit 421 for each procedure. Then, the judgment unit 422 configures a vector whose elements are one or more feature amounts and one or more pieces of device-related information for each procedure. Next, the judgment unit 422 acquires a third learner for each procedure. Next, the judgment unit 422 performs a machine learning prediction process using the vector and the third learner for each procedure, and acquires a prediction result. Note that the prediction result is information indicating that the work is "normal" or "abnormal". In this case, the third learner is a binary classification learner that identifies whether the work is "normal" or "abnormal".

The one or more feature quantities are image feature quantities. The image feature quantity is, for example, a spatial feature quantity. The spatial feature quantity is, for example, a local image feature (SIFT), a pixel value, an activity, a spatiotemporal correlation, a motion vector, and a frequency distribution. The activity is, for example, a maximum value and a minimum value of multiple pixels, a dynamic range (DR), a difference value between multiple pixels, a pixel value distribution of the entire image, a motion vector distribution of the entire image, and one or more spatiotemporal correlations of the entire image.

It is preferable that the determination unit 422 obtains the prediction result in real time during the procedure currently being performed.
(1-2) When using one third learning module

The judgment unit 422 acquires, for each procedure, one field of the inspection video acquired by the inspection video acquisition unit 221, and acquires a procedure identifier that identifies the procedure corresponding to the one field acquired by the inspection image acquisition means through a machine learning prediction process using the one field, the device-related information acquired by the device-related information acquisition unit 421, and a third learning device, and judges whether the procedure identifier matches the current procedure identifier.

For example, the determination unit 422 acquires one or more feature amounts from one field of the inspection target, and configures a vector whose elements are the one or more feature amounts and one or more pieces of device-related information acquired by the device-related information acquisition unit 421. Next, the determination unit 422 performs a machine learning prediction process using the vector and one third learner, acquires a procedure identifier, and determines whether the procedure identifier matches the current procedure identifier. In this case, it is preferable that the third learner is a multi-value classification learner that determines one procedure identifier from multiple procedure identifiers.
(1-3) When using vector similarity

For each procedure, the determination unit 422 obtains one field of the inspection video obtained by the inspection video obtaining unit 221, and obtains one or more feature amounts of the field. In addition, for each procedure, the determination unit 422 obtains one or more pieces of device-related information obtained by the device-related information obtaining unit 421. Then, for each procedure, the determination unit 422 constructs a vector whose elements are one or more feature amounts and one or more pieces of device-related information.

Next, the judgment unit 422 obtains, for each procedure, a vector that is teacher data corresponding to a normal operation from the storage unit 41. Next, the judgment unit 422 calculates the similarity between the constructed vector and the teacher data vector for each procedure. Next, the judgment unit 422 judges whether the similarity is equal to or greater than a threshold value or is larger than a threshold value (whether or not it is normal).
(2) Second Method

The first judgment means 4221 judges whether the procedure of the work to be inspected is correct or not using the equipment-related information and normal equipment-related information acquired by the equipment-related information acquisition unit 421 (referred to as the "first judgment"). Next, when the judgment result of the first judgment means 4221 is that the procedure is correct, the second judgment means 4222 judges whether the procedure of the work to be inspected for the inspection video is correct or not using teacher data based on the normal video (referred to as the "second judgment").

The following three examples are examples of the first judgment.
(2-1-1) When using the third learning module for each step

In the first judgment, for example, the first judgment means 4221 constructs a vector for each procedure, the elements of which are one or more pieces of device-related information acquired by the device-related information acquisition unit 421, and performs a machine learning prediction process using the vector and a third learner corresponding to each procedure to acquire a prediction result indicating whether the operation of the corresponding procedure is normal or not. In this case, there are two or more third learners corresponding to each procedure identifier. In addition, the third learner is a learner that uses one or more pieces of device-related information as explanatory variables and information indicating whether the operation of the corresponding procedure is normal or not as a target variable.
(2-1-2) When using one third learning module

In the first judgment, for example, the first judgment means 4221 uses a machine learning algorithm to construct a vector whose elements are one or more pieces of device-related information acquired by the device-related information acquisition unit 421 for each procedure, and performs machine learning prediction processing using the vector and one third learning device to acquire a procedure identifier. Then, the first judgment means 4221 judges whether the acquired procedure identifier matches the current procedure identifier. If they match, it is normal, and if they do not match, it is abnormal.
(2-1-3) When using vector similarity

In the first judgment, for example, the first judgment means 4221 constructs, for each procedure, a vector whose elements are one or more pieces of device-related information acquired by the device-related information acquisition unit 421. Next, for each procedure, the first judgment means 4221 calculates the similarity between the vector and a vector corresponding to the procedure identifier of the corresponding procedure (a vector corresponding to normal work). Next, the first judgment means 4221 judges, for each procedure, whether the similarity is equal to or greater than a threshold value (whether it is normal or not).

The following three examples are specific examples of the second judgment.
(2-2-1) When using the third learning module for each step

In the second judgment, for example, the second judgment means 4222 acquires one field of the inspection video acquired by the inspection video acquisition unit 221 for each procedure, acquires two or more feature amounts of the field, and configures a vector having the two or more feature amounts as elements. Next, the second judgment means 4222 acquires a third learning device for each procedure. Next, the second judgment means 4222 performs a machine learning prediction process using the vector and the third learning device for each procedure, and acquires a prediction result. Note that the prediction result is information indicating that the work is "normal" or "abnormal". In such a case, the third learning device is a binary classification learning device that identifies whether the work is "normal" or "abnormal". In addition, the third learning device is a learning device that uses two or more feature amounts of the field as explanatory variables.
(2-2-2) When using one third learning device

In the second judgment, for example, the second judgment means 4222 acquires one field of the inspection video acquired by the inspection video acquisition unit 221 for each procedure, acquires two or more feature amounts of the field, and configures a vector having the two or more feature amounts as elements. Next, the second judgment means 4222 acquires one third learning device. Next, the second judgment means 4222 performs a machine learning prediction process using the vector and the third learning device for each procedure, and acquires a procedure identifier. Then, the second judgment means 4222 judges whether the acquired procedure identifier matches the current procedure identifier.
(2-2-3) When using vector similarity

In the second judgment, for example, the second judgment means 4222 acquires, for each procedure, one field of the inspection video acquired by the inspection video acquisition unit 221, acquires two or more feature amounts of the field, and configures a vector having the two or more feature amounts as elements. Next, the second judgment means 4222 calculates, for each procedure, a similarity between the vector and a vector corresponding to the procedure identifier of the corresponding procedure (a vector corresponding to a normal operation). Next, the second judgment means 4222 judges, for each procedure, whether the similarity is equal to or greater than a threshold value (whether it is normal or not).
(3) The third method

The second judgment means 4222 uses teacher data based on the normal video to judge whether the procedure of the work to be inspected for the inspection video is correct (performs a "second judgment"). Next, if the judgment result of the second judgment means 4222 is that the procedure is correct, the first judgment means 4221 uses the equipment-related information and normal equipment-related information acquired by the equipment-related information acquisition unit 421 to judge whether the procedure of the work to be inspected is correct (performs a "first judgment"). Note that specific processing examples of the second judgment and the first judgment have been described above, so their explanation will be omitted here.

Next, an example of the operation of the inspection system B will be described. First, an example of the operation of the inspection device 4 will be described using the flowchart in FIG. 16. In the flowchart in FIG. 16, the explanation of the steps equivalent to those in the flowchart in FIG. 3 will be omitted.

(Step S1601) The device-related information acquisition unit 421 acquires one or more pieces of device-related information corresponding to the i-th procedure. The device-related information corresponding to the i-th procedure is, for example, device-related information acquired from device C at the time or at the time when the one or more fields acquired in step S304 were photographed. It is preferable that the acquired device-related information is information synchronized with the one or more fields acquired in step S304.

(Step S1602) The judgment unit 422 judges whether the work in the i-th procedure is normal or not, using one or more fields acquired in step S304 and one or more pieces of device-related information acquired in step S1601. The process proceeds to step S306. An example of such a judgment process will be described with reference to the flowcharts in Figs. 17 to 22.

In the flowchart in Figure 16, processing ends when the power is turned off or an interrupt occurs to end processing.

In addition, in the flowchart of FIG. 16, the current procedure identifier in the current procedure storage unit 214 is updated to the next procedure identifier in accordance with the increment of counter i.

Next, a first example of the determination process in step S1602 will be described using the flowchart in FIG. 17. In the flowchart in FIG. 17, the description of steps equivalent to those in the flowchart in FIG. 5 will be omitted.

(Step S1701) The judgment unit 422 obtains the third learning device from the learning device storage unit 212.

(Step S1702) The judgment unit 422 acquires one or more feature amounts from one or more fields acquired in step S304. The feature amount is an image feature amount. The feature amount may be a temporal feature amount in two or more fields. The temporal feature amount is, for example, a motion vector distribution of the entire image, or a spatiotemporal correlation of two or more entire images.

(Step S1703) The judgment unit 422 constructs a vector whose elements are one or more feature amounts acquired in step S1702 and one or more pieces of device-related information acquired in step S1601.

(Step S1704) The judgment unit 422 provides the third learning device obtained in step S1701 and the vector obtained in step S1703 to a module that performs machine learning prediction processing, executes the module, and obtains a procedure identifier. Go to step S505.

Next, a second example of the determination process in step S1602 will be described using the flowchart in FIG. 18. In the flowchart in FIG. 18, the description of steps equivalent to those in the flowcharts in FIG. 5 and FIG. 6 will be omitted.

(Step S1801) The judgment unit 422 judges whether a third learning device that pairs with the i-th procedure identifier exists in the learning device storage unit 212.

(Step S1802) The judgment unit 422 obtains the third learning device that pairs with the i-th procedure identifier from the learning device storage unit 212. Go to step S1702.

(Step S1803) The judgment unit 422 provides the third learning device obtained in step S1802 and the vector obtained in step S1703 to a module that performs machine learning prediction processing, executes the module, and obtains a procedure identifier. Go to step 607.

Next, a third example of the determination process in step S1602 will be described using the flowchart in FIG. 19. In the flowchart in FIG. 19, the description of steps equivalent to those in the flowcharts in FIG. 5 and FIG. 17 will be omitted.

Next, a fourth example of the determination process in step S1602 will be described using the flowchart in FIG. 20. In the flowchart in FIG. 20, the description of steps equivalent to those in the flowcharts in FIG. 7 and FIG. 17 will be omitted.

(Step S2001) The judgment unit 422 acquires teacher data corresponding to the i-th procedure from the teacher data storage unit 211. The teacher data here is a vector constructed using one or more fields in which normal work in the i-th procedure is photographed, and one or more pieces of device-related information during normal work in the i-th procedure.

(Step S2002) The judgment unit 422 calculates the similarity between the vector that is the training data acquired in step S2001 and the vector constructed in step S1703. Proceed to step S706.

Next, a fifth example of the determination process in step S1602 will be described using the flowchart in FIG. 21. In the flowchart in FIG. 21, the description of steps equivalent to those in the flowchart in FIG. 5 will be omitted.

(Step S2101) The first determination means 4221 performs a first determination using one or more pieces of device-related information acquired in step S1601. An example of the first determination process is described using the flowchart in FIG. 22.

(Step S2102) If the result of the first judgment in step S2101 is "normal", the first judgment means 4221 proceeds to step S2103, and if the result is "error", the first judgment means 4221 proceeds to step S507.

(Step S2103) The second determination means 4222 performs a second determination using one or more fields acquired in step S304. Examples of the second determination process are the determination processes in each of Figures 5, 6, 7, and 8 in embodiment 1.

(Step S2104) If the result of the second judgment in step S2103 is "normal", the second judgment means 4222 proceeds to step S506, and if the result is "error", the second judgment means 4222 proceeds to step S507.

Next, an example of the first judgment process in step S2101 will be described using the flowchart in FIG. 22. In the flowchart in FIG. 22, explanations of steps equivalent to those in the flowchart in FIG. 7 will be omitted.

(Step S2201) The first judgment means 4221 acquires teacher data paired with the corresponding procedure identifier from the teacher data storage unit 211. Note that the teacher data here is a vector constructed using two or more pieces of device-related information acquired when normal work is performed in the corresponding procedure.

(Step S2202) The first judgment means 4221 constructs a vector using two or more pieces of device-related information acquired in step S1601.

(Step S2203) The first judgment means 4221 calculates the similarity between the training data acquired in step S2201 and the vector acquired in step S2202. Go to step S706.

As described above, according to this embodiment, it is possible to determine whether or not a task using device C is being performed correctly by using a video obtained by filming a task having two or more steps performed using device C and device-related information when the task is performed using device C.

In addition, according to this embodiment, after making a judgment using the device-related information, only if the judgment is normal is a judgment made using the inspection image, so that the normality of the work can be judged accurately and efficiently.

In addition, according to this embodiment, after making a judgment using the inspection image, only if the judgment is normal is a judgment made using the device-related information, so that the normality of the work can be judged accurately and efficiently.

Furthermore, according to this embodiment, when a task is performed using device C, the machine learning algorithm can also be used to determine whether the task is being performed correctly.

Furthermore, the processing in this embodiment may be realized by software. This software may be distributed by software download or the like. This software may also be recorded on a recording medium such as a CD-ROM and distributed. This also applies to other embodiments in this specification. The software that realizes the inspection device 4 in this embodiment is the following program. In other words, this program causes a computer to function as an inspection video acquisition unit that acquires an inspection video, which is a video of an inspection target work that is an inspection target work having two or more procedures, being performed, an equipment-related information acquisition unit that acquires equipment-related information that is information input to the equipment for the inspection target work, information generated within the equipment, or information output from the equipment, a judgment unit that judges whether the procedure of the inspection target work is correct for the inspection video and the equipment-related information using teacher data based on normal equipment-related information, which is equipment-related information when the normal work is performed, and the normal video, and an output unit that outputs judgment result information regarding the judgment result of the judgment unit.

(Embodiment 4)
In this embodiment, a description will be given of a learning device 5. In this embodiment, the learning device 5 constitutes a learning unit used by the inspection device 4 described in the fourth embodiment.

Figure 23 is a block diagram of the learning device 5 in this embodiment. The learning device 5 includes a learning storage unit 51 and a learning processing unit 52. The learning storage unit 51 includes a teacher data storage unit 511. The learning processing unit 52 includes a teacher data acquisition unit 521 and a learning unit 522.

The learning storage unit 51 stores various types of information. The various types of information include, for example, one or more normal videos, teacher data, and a third learning device.

The teacher data storage unit 511 stores two or more teacher data for constructing a third learning device. Such teacher data is, for example, a field selected by the user from a normal video, and is a vector (vector of positive examples) whose elements are one or more feature amounts obtained from the field used in the judgment process of the judgment means 2222 and one or more pieces of normal device-related information.

The teacher data is, for example, one or more fields selected by the user from the normal video. The teacher data is, for example, one or more pieces of normal device-related information acquired when device C operates normally.

The training data is, for example, one or more fields selected by the user from a video (e.g., a normal video or an abnormal video taken when an abnormal task is performed), and is a vector (vector of negative examples) whose elements are one or more feature amounts and one or more pieces of abnormal device-related information obtained from one or more fields not used in the judgment process of the judgment means 2222.

The teaching data is, for example, one or more fields selected by the user from a video (for example, a normal video or an abnormal video taken when an abnormal task is performed). It is also information related to the teaching abnormality device.

The teacher data in the teacher data storage unit 511 may be stored by the learning processing unit 52 or by a means not shown. There is no restriction on the method of storing the teacher data in the teacher data storage unit 511.

The learning processing unit 52 performs various types of processing. For example, various types of processing are performed by the teacher data acquisition unit 521 and the learning unit 522.

The teacher data acquisition unit 521 acquires, for each of two or more procedures, two or more pieces of teacher data from the normal video, which are combinations of one or more fields that satisfy predetermined conditions, normal device related information, and a procedure identifier.

The teacher data acquisition unit 521 acquires, for example, one or more fields that satisfy predetermined conditions and one or more pieces of normal device related information from the normal video for each of two or more steps. The teacher data acquisition unit 521 then acquires one or more feature amounts using the one or more fields. Next, the teacher data acquisition unit 521 constructs teacher data (positive examples) that is a vector whose elements are one or more feature amounts and one or more pieces of normal device related information. Next, the teacher data acquisition unit 521 associates the vector with the step identifier of the corresponding step and accumulates it in the teacher data storage unit 511.

Furthermore, the teacher data acquisition unit 521 acquires, for example, one or more fields that satisfy a predetermined condition and one or more pieces of abnormal device-related information from the abnormal video for each of two or more procedures. Next, the teacher data acquisition unit 521 acquires one or more feature amounts from the one or more fields. Next, the teacher data acquisition unit 521 acquires a negative example vector having one or more feature amounts and one or more pieces of abnormal device-related information as elements, and accumulates the vector in the teacher data storage unit 511.
The teacher data acquisition unit 521 reads, for example, two or more pieces of teacher data from the teacher data storage unit 511 .

The process of obtaining the above vectors may be performed by the learning unit 522.

The learning unit 522 performs a machine learning learning process using two or more pieces of teacher data acquired by the teacher data acquisition unit 521, acquires a third learning device, and accumulates the third learning device. As described above, the machine learning algorithm is not important.

Next, a first operation example of the learning device 5 will be described with reference to the flowchart in FIG. 24. In the flowchart in FIG. 24, the description of the same steps as those in FIG. 10 will be omitted.

(Step S2401) The teacher data acquisition unit 321 acquires one or more pieces of device-related information. The method for acquiring one or more pieces of device-related information may be the same as that used by the device-related information acquisition unit 421.

(Step S2402) The teacher data acquisition unit 321 acquires one or more feature amounts from one or more fields acquired in step S1005. The teacher data acquisition unit 321 then constructs a vector whose elements are the one or more feature amounts and one or more pieces of device-related information acquired in step S2401, associates the vector with the j-th procedure identifier, and accumulates it in the teacher data storage unit 511. Proceed to step S1007. Note that the vector is teacher data.

(Step S2403) The learning unit 522 provides the two or more pieces of teacher data stored in the teacher data storage unit 511 in step S2402 to a machine learning learning processing module, executes the module, and obtains a third learning device. The third learning device here is a learning device that uses each element of the vector as an explanatory variable and the procedure identifier as an objective variable.

(Step S2404) The learning unit 522 stores the third learning device acquired in step S2403 in the learning storage unit 51. The process ends.

Next, a second operation example of the learning device 5 will be described with reference to the flowchart in FIG. 25. In the flowchart in FIG. 24, the same steps as those in FIG. 10, FIG. 11, and FIG. 24 will not be described.

(Step S2501) The learning unit 522 provides one or more positive examples and one or more negative examples to a module that performs machine learning learning processing, executes the module, and obtains a third learning device.

(Step S2502) The learning unit 522 stores the third learning device acquired in step S2501 in a pair with the procedure identifier of the kth procedure. Note that the third learning device here uses each element of the vector as an explanatory variable and information indicating whether or not the procedure is relevant as an objective variable. The third learning device here is a learning device for performing binary classification.

As described above, according to this embodiment, a learning device that can be used by the inspection device 4 is obtained. The learning device is a third learning device. The inspection device 4 may also acquire the first learning device or the second learning device acquired by the inspection device 2. More specifically, according to this embodiment, by photographing an operation having two or more steps and acquiring device-related information from the device C, a learning device for determining whether the operation is being performed correctly in real time can be acquired.

The program for realizing the learning device 5 in this embodiment is a program for making a computer that can access a teacher data storage unit that stores one or more teacher data based on a normal video, which is a video captured of a normal operation, which is a correct operation having two or more steps, being performed, and normal device-related information, which is device-related information when the normal operation is performed, function as a teacher data acquisition unit that acquires, for each of the two or more steps, from the normal video, two or more partial teacher data that are combinations of one or more fields that satisfy a predetermined condition, the normal device-related information, and a step identifier, and a learning unit that performs machine learning learning processing on the two or more partial teacher data, acquires a learner, and accumulates the learner.

FIG. 26 also shows the appearance of a computer that executes the programs described in this specification to realize the inspection device 2 or learning device 3 of the various embodiments described above. The above-mentioned embodiments can be realized by computer hardware and a computer program executed thereon. FIG. 26 is an overview of this computer system 300, and FIG. 27 is a block diagram of the system 300.

In FIG. 26, computer system 300 includes computer 301, which includes a CD-ROM drive, keyboard 302, mouse 303, and monitor 304.

In FIG. 27, in addition to a CD-ROM drive 3012, the computer 301 includes an MPU 3013, a bus 3014 connected to the CD-ROM drive 3012 etc., a ROM 3015 for storing programs such as a boot-up program, a RAM 3016 connected to the MPU 3013 for temporarily storing instructions for application programs and providing temporary storage space, and a hard disk 3017 for storing application programs, system programs, and data. Although not shown here, the computer 301 may further include a network card that provides a connection to a LAN.

The program that causes the computer system 300 to execute the functions of the inspection device 2 and the like of the above-mentioned embodiment may be stored on the CD-ROM 3101, inserted into the CD-ROM drive 3012, and then transferred to the hard disk 3017. Alternatively, the program may be sent to the computer 301 via a network (not shown) and stored on the hard disk 3017. The program is loaded into the RAM 3016 when executed. The program may be loaded directly from the CD-ROM 3101 or the network.

The program does not necessarily have to include an operating system (OS) that causes the computer 301 to execute the functions of the inspection device 2 of the above-described embodiment, or a third-party program. The program only needs to include an instruction portion that calls appropriate functions (modules) in a controlled manner to obtain the desired results. How the computer system 300 operates is well known, and a detailed description will be omitted.

In the above program, the steps of transmitting information and receiving information do not include processing performed by hardware, such as processing performed by a modem or interface card in the transmission step (processing that is performed only by hardware).

The computer that executes the above program may be a single computer or multiple computers. In other words, centralized processing or distributed processing may be performed. In other words, the inspection device 2, etc. may be a standalone device or may be composed of two or more devices.

Furthermore, in each of the above embodiments, it goes without saying that two or more communication means present in one device may be realized physically by one medium.

In addition, in each of the above embodiments, each process may be realized by centralized processing in a single device, or may be realized by distributed processing in multiple devices.

As described above, the inspection device of the present invention has the effect of being able to determine whether an operation having two or more steps has been performed correctly by photographing the operation, and is useful as an inspection device, etc.

A, B Inspection system C Apparatus 1 Camera 2, 4 Inspection apparatus 3, 5 Learning apparatus 21, 41 Storage unit 22, 42 Processing unit 23 Output unit 31, 51 Learning storage unit 32, 52 Learning processing unit 211, 511 Teacher data storage unit 212 Learning device storage unit 213 Threshold value storage unit 214 Current procedure storage unit 221 Inspection video acquisition unit 222, 422 Judgment unit 321, 521 Teacher data acquisition unit 322, 522 Learning unit 411 Normal apparatus related information storage unit 421 Apparatus related information acquisition unit 2221 Inspection image acquisition means 2222 Judgment means 4221 First judgment means 4222 Second judgment means

Claims (12)

a current procedure storage unit in which a current procedure identifier for identifying a procedure currently being inspected among the two or more procedures is stored;
an inspection video acquisition unit that acquires an inspection video, which is a video captured while an inspection target work, which is an inspection target work having two or more procedures, is being performed;
an apparatus-related information acquisition unit that acquires apparatus-related information, which is information input to the apparatus for the work to be inspected, information generated within the apparatus, or information output from the apparatus;
a learning module storage unit in which a learning module configured by performing a learning process using a machine learning algorithm on two or more pieces of teacher data, each of which is a set of one field corresponding to each of two or more procedures, normal device-related information which is device-related information when normal work is performed, and a procedure identifier which identifies each procedure, is stored;
a determination unit that obtains one field of the inspection video obtained by the inspection video obtaining unit, obtains a procedure identifier that identifies a procedure corresponding to the one field by a machine learning prediction process using the one field, the device-related information obtained by the device-related information obtaining unit, and the learning device, and determines whether the procedure identifier matches the current procedure identifier ;
and an output section that outputs judgment result information relating to the judgment result of the judgment section. a storage unit for storing procedure information for identifying two or more procedures constituting a task;
a current procedure storage unit in which a current procedure identifier that identifies a procedure currently being inspected among the two or more procedures is stored;
an inspection video acquisition unit that acquires an inspection video, which is a video captured while an inspection target work, which is an inspection target work having two or more procedures, is being performed;
A judgment unit that judges whether or not the procedure of the inspection target work for the inspection video is correct using teacher data based on a normal video, which is a video captured of a normal work being performed, which is a correct work having two or more procedures;
an output unit that outputs judgment result information regarding the judgment result of the judgment unit,
The teacher data is
having one or more fields corresponding to each of the two or more procedures,
The determination unit is
an inspection image acquisition means for acquiring one or more fields of the inspection video acquired by the inspection video acquisition unit;
a determination means for determining whether or not a procedure corresponding to the one or more fields acquired by the inspection image acquisition means is a procedure identified by the current procedure identifier, using the one or more fields acquired by the inspection image acquisition means and one or more fields corresponding to the current procedure identifier;
The determination means is
An inspection device that, when the procedure corresponding to the one or more fields acquired by the inspection image acquisition means matches the procedure identified by the current procedure identifier, overwrites the current procedure identifier with the procedure identifier of the next procedure indicated by the procedure information. The inspection video acquisition unit
The inspection device according to claim 2 , wherein the inspection video is sequentially acquired while an inspection target operation is being performed. The teacher data is
A pair of a field corresponding to each of the two or more procedures and a procedure identifier for identifying each procedure,
Further comprising a learning module storage unit in which a learning module configured by performing a learning process on two or more pieces of teacher data using a machine learning algorithm is stored;
The determination means is
4. The inspection device according to claim 2, further comprising: a machine learning prediction process using the one field acquired by the inspection image acquisition means and the learning device, to acquire a procedure identifier corresponding to the one field, and to determine whether the procedure identifier matches the current procedure identifier. The determination means is
An inspection device as described in claim 2 or claim 3, which calculates a similarity between a field corresponding to each of the two or more procedures and the one field acquired by the inspection image acquisition means, and uses the two or more similarities to determine whether the procedure corresponding to the one field acquired by the inspection image acquisition means is a procedure identified by the current procedure identifier. The determination unit is
An inspection device as described in any one of claims 2 to 5, which obtains one or more fields that are a portion of the fields from the inspection video that satisfy predetermined conditions, and uses the one or more fields and the teacher data to determine whether the procedure of the work to be inspected for the inspection video is correct . The determination unit is
An inspection device as described in any one of claims 1 to 6, which performs a machine learning prediction process using a second learning device to determine whether or not each field in the inspection video will be used as a field for determining whether or not it is a correct procedure, obtains one or more fields corresponding to the prediction result that they will be used, and uses the one or more fields and the teacher data to determine whether or not the procedure of the work to be inspected for the inspection video is correct. a teacher data storage unit in which two or more teacher data are stored based on a normal video , which is a video of a normal operation, which is a correct operation having two or more steps, being performed on a device, the normal video having two or more fields corresponding to a step identifier that identifies each of the two or more steps, and normal device-related information, which is device-related information of the device on which the normal operation was performed;
a teacher data acquisition unit that acquires, for each of the two or more procedures, from the normal video, two or more teacher data that are a combination of an explanatory variable having one or more fields that satisfy a predetermined condition and the normal device related information, and a target variable that is a procedure identifier corresponding to the one or more fields;
A learning device comprising: a learning unit that performs machine learning learning processing using the two or more teacher data acquired by the teacher data acquisition unit , acquires a learning device, and accumulates the learning device. An inspection method comprising all of the processes performed by the inspection apparatus according to any one of claims 1 to 7. 9. A learning method comprising all the processes performed by the learning device according to claim 8. Computer,
A program for causing the inspection device according to any one of claims 1 to 7 to function as such an inspection device. Computer,
A program for causing the learning device according to claim 8 to function.

Images