JP7708040B2 - Teacher data collection device - Google Patents

Description

This disclosure relates to a teacher data collection device.

Patent Document 1 discloses a technology that automatically generates training data to be used in machine learning to generate a trained model that detects abnormalities in factories. This technology acquires information (such as weight) necessary to determine whether an object is abnormal along with a video, and links the video to the determination result to generate training data.

JP 2020-194270 A

However, in an actual factory, the probability of defective products occurring is extremely low, and it was not possible to collect a sufficient amount of training data.

The present disclosure has been made in consideration of the above, and aims to provide a teacher data collection device capable of collecting large amounts of teacher data.

The teacher data collection device according to the present disclosure includes a processor configured to acquire a video of a worker assembling a part to a workpiece, the video being captured with the worker, the part, and the workpiece positioned so that they are reflected in a predetermined area at the start of the work, extract positional information of the worker's hands, the part, and the workpiece from the video, determine whether the work was performed according to a predetermined procedure based on the positional information, and link the video with the determination result for the video and store the result as teacher data to be used in machine learning.

This disclosure makes it possible to realize a teacher data collection device that can collect large amounts of teacher data.

FIG. 1 is a block diagram showing a schematic configuration of a teacher data collection device according to an embodiment. FIG. 2 is a flowchart showing an example of a processing procedure of a teacher data collection method executed by the teacher data collection device according to the embodiment. FIG. 3 is a diagram for explaining the state of work.

A teacher data collection device according to an embodiment of the present disclosure will be described with reference to the drawings. Note that the components in the following embodiments include those that are easily replaceable by a person skilled in the art, or those that are substantially identical.

(Teacher data collection device)
The teacher data collection device is a device that collects teacher data used for machine learning to generate a trained model that judges whether or not a worker has performed an appropriate procedure for assembling a part to a workpiece in a factory or the like. The teacher data is a group of data that links a video of the work being performed with a judgment result of whether or not the work in the video was performed in an appropriate procedure. By having an AI perform machine learning using this teacher data, a trained model that judges whether or not the work in the video was performed in an appropriate procedure can be generated.

The teacher data collection device can collect large amounts of teacher data by collecting video footage of work processes from many factories.

Various specific configurations of the teacher data collection device are envisioned. Examples of teacher data collection devices include server systems, personal computers, workstations, mobile phones, smartphones, tablet terminal devices, vehicles, and other machine tools. In addition, the teacher data collection device may be equipped with a trained model consisting of the learning results of, for example, deep learning or other machine learning within the configurations listed above.

As shown in FIG. 1, the teacher data collection device 1 includes a control unit 11, a memory unit 12, an acquisition unit 13, and a display unit 14.

The control unit 11 is realized by a processor such as a CPU (Central Processing Unit) and a memory (main storage unit) such as a RAM (Random Access Memory) and a ROM (Read Only Memory).

The control unit 11 loads a program into the working area of the main memory and executes it, and by controlling each component through the execution of the program, realizes a function that meets a predetermined purpose. Through the execution of the program, the control unit 11 functions as an extraction unit 111 and a determination unit 112.

Note that while FIG. 1 shows an example in which the functions of each part of the control unit 11 are realized by one device (e.g., a computer, etc.), the functions of each part may be realized by, for example, multiple devices. In other words, the extraction unit 111 and the determination unit 112 of the control unit 11 may be realized by different devices, and these devices may be connected by a network.

The extraction unit 111 extracts positional information of the worker's hands, parts, and workpiece from a video that captures the worker assembling parts to a workpiece.

The determination unit 112 determines whether the work was performed according to a specified procedure based on the position information of the worker's hands, parts, and work extracted by the extraction unit 111.

The storage unit 12 is realized by a recording medium such as an erasable programmable ROM (EPROM), a hard disk drive (HDD), or a removable medium. Examples of removable media include a universal serial bus (USB) memory, a compact disc (CD), a digital versatile disc (DVD), and a Blu-ray (registered trademark) disc (BD).

The storage unit 12 can store an operating system (OS), various programs, various tables, various databases, and the like. The storage unit 12 stores teacher data 121. The teacher data is data that links a video with the judgment result of the judgment unit 112 for that video. In addition to the above, the storage unit 12 may also store processing results in the control unit 11, etc., as necessary. The storage unit 12 may also store a trained model generated using the teacher data.

The acquisition unit 13 is configured, for example, with a communication module capable of receiving various types of information. The acquisition unit 13 communicates with external devices, for example, via a network (not shown), and receives various types of information. The acquisition unit 13 acquires a video of a worker assembling a part to a workpiece, the video being captured by positioning the worker, the part, and the workpiece so that they are reflected in a predetermined area at the start of the work.

The display unit 14 is configured, for example, with a liquid crystal display or an organic EL display. The display unit 14 displays, for example, the processing results of the control unit 11 and various other information.

(Teacher data collection method)
An example of the processing procedure of the teacher data collection method executed by the teacher data collection device according to the embodiment will be described with reference to FIG. 2. FIG. 3 is a diagram for explaining the state of work. As shown in FIG. 3, in an image Im captured at the start of work, a worker O, parts P1 to P4, and a workpiece W are arranged so as to be reflected in a predetermined area of the image Im. In other words, the camera's angle of view, the positions of the worker O, parts P1 to P4, and workpiece W at the start of work are adjusted so that the image Im shown in FIG. 3 is captured. The procedure of the work in which the worker O assembles the parts P1 to P4 to the workpiece W is set to the order of parts P1, P2, P3, and P4.

Image Im is, for example, an image captured by a top-view camera attached to the ceiling of a factory, but the direction or angle of view in which the image is captured is not particularly limited as long as the areas in the image that show the worker, work, and parts are defined. However, when collecting videos from multiple factories or multiple production lines, it is required that the worker O, parts P1 to P4, and work W are positioned so that they are captured in specified areas of image Im in every factory and every production line.

The parts P1 to P4 and the workpiece W are not particularly limited. In other words, the parts P1 to P4 and the workpiece W may be different for each video of the work, and the finished product may also be different.

First, the acquisition unit 13 acquires a video of the worker O assembling the parts P1 to P4 onto the workpiece W (step S1). In this video, as shown in FIG. 3, the worker O, the parts P1 to P4, and the workpiece W are positioned in advance so that they are visible in a predetermined area at the start of the work.

Next, the extraction unit 111 extracts position information of the hands of the worker O, the parts P1 to P4, and the workpiece W from a video of the worker O assembling the parts P1 to P4 onto the workpiece W (step S2).

Then, the determination unit 112 determines whether the work was performed according to the predetermined procedure based on the position information of the hand of the worker O, parts P1 to P4, and the workpiece W extracted by the extraction unit 111 (step S3). Specifically, the determination unit 112 determines whether the worker O assembled parts P1, P2, P3, and P4 in the order on the workpiece W based on the time change of each piece of position information in the video. At this time, if the assembly order is incorrect or a process for assembling any of parts P1 to P4 is missing, the determination unit 112 determines that the work was not performed according to the predetermined procedure.

Then, the memory unit 12 stores teacher data linking the video with the judgment result of the judgment unit 112 for that video (step S4).

In the teacher data collection device according to the embodiment described above, the positioning of the worker O, parts P1-P4, and workpiece W at the start of the work and the procedure for the work are determined in advance, and then a video of the work is captured. As a result, even if the video captures work involving different parts, workpieces, or finished products, it can be used as teacher data to be used in machine learning to generate a trained model that determines whether the work was performed in the appropriate procedure. Moreover, according to the teacher data collection device 1, by collecting a large amount of video from multiple factories that use different parts, workpieces, or finished products, it is possible to collect a larger amount of teacher data than when collecting videos from factories that use the same parts, workpieces, and finished products.

By performing machine learning using the large amount of training data collected by the training data collection device 1, a trained model can be generated that determines whether work has been performed in the correct procedure. Using this trained model, it is also possible to notify an alert to a worker if the worker makes a mistake in the order of work or skips a step.

Further advantages and modifications may readily occur to those skilled in the art. Thus, the invention in its broader aspects is not limited to the specific details and representative embodiments shown and described above. Thus, various modifications may be made without departing from the spirit or scope of the general inventive concept as defined by the appended claims and equivalents thereof.

REFERENCE SIGNS LIST 1 Teacher data collection device 11 Control unit 111 Extraction unit 112 Determination unit 12 Storage unit 121 Teacher data 13 Acquisition unit 14 Display unit

Claims (1)

A video is acquired by capturing an image of an operator assembling a part to a workpiece, the video being captured while the operator, the part, and the workpiece are positioned so as to be captured in a predetermined area at the start of the operation;
Extracting position information of the worker's hands, the parts, and the workpiece from the video;
determining whether the work has been performed according to a predetermined procedure based on the position information;
A training data collection device comprising: a processor configured to link the video and a judgment result for the video and store them as training data to be used in machine learning.