JP7129673B2 - How to create a control system and trained model - Google Patents

Description

The present invention relates to a control system and a method of creating a learned model.

At factory labor sites that require skilled work, many skilled workers have retired and the number of new recruits has decreased due to the declining birthrate and aging population. Since it is difficult to train skilled workers, passing on the skilled work that has been performed by skilled workers has become a major issue. Skills possessed by experts are called tacit knowledge. Tacit knowledge includes many skills that cannot be expressed in words, such as intuition and tricks, and it has been extremely difficult to replace it with logic to control machines.

On the other hand, among experts and researchers in FA research, there has long been a preconceived notion that FA control is difficult with AI (Artificial Intelligence) that has learned the knowledge of experts. In the background, from the 1980s to the 1990s, AI research at the time, such as expert systems that learned the knowledge of experts and could make decisions like experts, was blocked by the barrier of tacit knowledge of experts, and it did not work as expected. The fact that it did not lead to practical application without obtaining results has had a large impact. For example, an expert knowledge base generation method using an expert system is known (see Patent Document 1, for example).

JP-A-4-262424

Since the known techniques such as the above-mentioned prior art documents use rule-based reasoning, it is difficult to deal with the tacit knowledge of experts. Also, in the near future, it will be necessary to remotely consolidate multiple skilled tasks under the dwindling number of skilled workers. have not been sufficiently considered. In addition, sufficient consideration has not been given as to how to educate general workers and new workers about the skills that are the tacit knowledge of experts.

The present invention has been made in consideration of such circumstances, and is a control system capable of controlling a machine using a learned model obtained by focusing on the tacit knowledge possessed by skilled workers. , and to provide a method for creating a trained model.

The control system of the embodiment acquires data of an image captured by a camera that captures a point of interest of an object of a skilled worker, and also acquires data of an operation signal indicating details of operation of a machine by the skilled worker. and an acquisition unit that generates a signal for operating the machine by inputting the image data to a trained model that has been trained using the image data as learning data and the operation signal data as teacher data. and a control unit for providing signals generated by the reasoning unit to the machine.

A machine can be controlled using a trained model obtained by focusing on the tacit knowledge possessed by skilled workers.

1 is a schematic diagram of a steel material processing system 1 as an example of a control system; FIG. FIG. 10 is a diagram showing an example of centering work by a skilled worker when creating a trained model; 1 is a cross-sectional view showing a functional configuration of a steel material processing AI system 2; FIG. 2 is a block diagram showing an example of the functional configuration of the control device 100; FIG. FIG. 10 is a diagram showing an example of processing of the control device in a learning scene; FIG. 4 is a diagram showing an example of processing of the control device in a control scene; FIG. 2 is a block diagram showing an example of a functional configuration of a trained model creation device 200; FIG. 8 is a diagram showing an example of learning data extracted from image data of a second camera 72; FIG. It is a figure which shows the structural example of CNN. FIG. 11 is a diagram showing an example of a conceptual diagram of a learning software configuration by a learning processing unit 243. FIG. FIG. 4 is a diagram showing an example of a schematic configuration of an RNN model used by a learning processing unit 243; FIG. It is a figure which shows an example of the result of having performed simulation. FIG. 4 is a diagram showing an example of input data to a trained model extracted from image data; 4 is a conceptual diagram of estimation processing by the control device 100. FIG. 4 is a flowchart showing an example of estimation processing by the control device 100; 4 is a flow chart showing an example of a processing flow of automatic control in the control device 100. FIG. 4 is a flow chart showing an example of a fail-safe and Take Over processing flow in the control device 100. FIG. 4 is a flow chart showing an example of a processing flow for automatic output permission in the control device 100. FIG. It is a figure which shows the comparison of the control signal based on an estimation result, and a skilled worker's operation timing. 4 is a diagram showing an example of a display screen in an interactive user interface section 190; FIG. FIG. 10 is a diagram showing another example of a display screen in the interactive user interface section 190; FIG. FIG. 10 is a diagram for explaining comparison of centering operations;

A control system of the embodiment and a method of creating a learned model will be described below with reference to the drawings.

[First embodiment]
FIG. 1 is a schematic diagram of a steel processing system 1 that is an example of a control system. The steel processing system 1 includes, for example, a coil support section 10, an uncoiler 20, a rough leveler 30, a side trimmer 40, a first camera 71, a second camera 72, a control device 100, and a learned model creation device 200. and Here, the control device 100 and the trained model generation device 200 are separate devices, but the present invention is not limited to this. The control device 100 and the learned model creating device 200 may be formed integrally.

The steel material processing system 1 is an apparatus for drawing out a coiled steel material 10A (an example of an object), conveying it by a conveying means (not shown) such as a belt conveyor, and processing it into a flat steel material of a predetermined size. be. When arranging in the order of arrangement from the upstream side to the downstream side in the conveying direction, the order is the coil support section 10, the uncoiler 20, the rough leveler 30, and the side trimmer 40. As shown in FIG. An example in which the target object of the control system is a steel material will be described below, but the present invention is not limited to this. The target object of the control system may be a member made of material such as rubber or paper as long as it is a sheet-like member wound in a coil shape.

The coil support portion 10 is inserted into a cavity along the central axis of the coiled steel material 10A and rotatably supports the steel material 10A. In addition, the coil support section 10 is connected to the control device 100, and is rotated in the axial direction by the control device 100 to pull out or pull back the steel material 10A.

An adjustment device 12 is provided around the coil support portion 10 , and the adjustment device 12 is connected to the control device 100 . The adjustment device 12 includes, for example, actuators provided at both one end 10R and the other end 10L of the coil support portion 10 . The adjustment device 12 operates based on a control signal from the control device 100 to move the coil support 10 in the axial direction. That is, the adjustment device 12 moves the coil support portion 10 toward the one end 10R by a predetermined amount, or moves the coil support portion 10 toward the one end 10L by a predetermined amount. The steel material 10A is centered by moving the coil support 10 in the axial direction. Centering refers to transporting the steel material 10A so as to align the center (the center in the width direction) of the steel material 10A drawn out to the uncoiler 20 with the center (hereinafter referred to as the transport center line) in the direction perpendicular to the transport direction of the transport means. It is to adjust the posture of the direction. This centering may be manually controlled by an operator, or may be automatically controlled using a learned model created by the learned model creation device 200 .

The uncoiler operator A monitors the automatic control of the adjustment device 12 by the control device 100 . Also, the uncoiler operator A may operate the control lever 110A of the control device 100 to operate the adjustment device 12 and adjust the posture of the steel material 10A in the conveying direction, thereby performing manual centering work. Details will be described later.

An important purpose of the centering operation is to prevent edge splitting from occurring in the process using the side trimmer 40 . Edge cut means that the width of the edge portion (edge) of the steel material 10A cut off by the side trimmer 40 is too small and the edge is cut into a distorted shape, or the edge cannot be cut. Edge cuts occur, for example, when the posture of the steel material 10A in the conveying direction shifts. It means that the width direction of steel material 10A is not cut|disconnected to the regular size because edge|edge cutting generate|occur|produces. When edge splitting occurs, it is necessary to stop the steel material processing system 1, resulting in poor manufacturing efficiency. In addition, loss occurs because the steel material in the part where the edge is cut cannot be sold as a product. In this way, it is necessary to carry out the centering work on the upstream side while foreseeing edge splitting from not only the situation on the upstream side but also the situation on the downstream side, and this largely depends on the tacit knowledge of skilled workers. The trained model creation device 200 creates a trained model by learning the tacit knowledge of the skilled worker.

The uncoiler 20 is a device that includes a plurality of rollers that sandwich the steel material 10A from above and below, and pulls out the coiled steel material 10A toward the conveying path. The rough leveler 30 is provided with a plurality of rollers that sandwich the steel material 10A from above and below, and is a device that reduces the distortion of the steel material 10A pulled out by the uncoiler 20 . The side trimmer 40 is a device that cuts both sides of the steel material 10A and trims the length in the width direction (the direction perpendicular to the conveying direction) to a specified size. In this embodiment, the side trimmer 40 cuts both ends of the steel material 10A along the conveying direction. Not limited to this, the side trimmer 40 may cut one end of the steel material 10A along the conveying direction.

A first camera 71 is attached around the side trimmer 40 to capture an image of a predetermined area including the cutting position P<b>1 by the side trimmer 40 . The cutting position P1 is a position where the side trimmer 40 cuts both ends (one end 10P1R and the other end P1L) of the steel material 10A being conveyed. The first camera 71 images, for example, a predetermined area including one end P1R and a predetermined area including the other end P1L.

A second camera 72 is attached around the rough leveler 30 to capture an image of a predetermined area including the uncoiler position P2 at the entrance of the rough leveler 30 (on the upstream side of the rough leveler 30 in the conveying direction). The second camera 72 captures an image of a predetermined area including, for example, a portion where one end 10R of the steel material 10A and the rough leveler 30 contact each other.

The first camera 71 and the second camera 72 are an example of a camera that captures the target point of the object of the skilled worker. A focus point is a physical location that a skilled worker focuses on when performing a centering operation. For example, the point of interest captured by the first camera 71 is the cutting position P1, and the point of interest captured by the second camera 72 is the uncoiler position P2. A method of determining the point of interest, a specific example, and the like will be described later. The first camera 71 and the second camera 72 are, for example, waterproof and dustproof cameras. The number of the first camera 71 and the number of the second cameras 72 may be one, or a plurality of them.

Note that the timing at which images are captured by the first camera 71 and the second camera 72 and the timing at which the control device 100 accepts an operation are associated with each other by time stamps expressed in Japan Standard Time, for example. In other words, with the time information as a key, the fact that the operation has been accepted is associated with the image data.

The control device 100 is, for example, a device including an industrial Raspberry Pi (registered trademark) or a digital I/O input/output module. The control device 100 inputs signals necessary for machine learning such as lever operation among the signals used in the factory line, and outputs control signals for controlling the adjusting device 12 .

The learned model creation device 200 creates a learned model based on the collected information, and outputs data of the created learned model to the control device 100 . This learned model is a model that has learned the relationship between the skilled worker's operation of the adjustment device 12 and the conveying situation of the steel material 10A. It is a trained model that has previously learned centering work by a skilled worker Y, which will be described later.

With reference to FIG. 2, an example of centering work by a skilled worker when creating a learned model will be described below. FIG. 2A shows an example in which there are a skilled worker X who is gazing at the cut edge near the side trimmer 40 and a skilled worker Y who operates the control device 100 with reference to the assistance of the skilled worker X. indicates The skilled worker X senses the sign of the cut edge and gives a hand gesture to the skilled worker Y to correct the centering work necessary to prevent the cut edge. Skilled worker X may also operate a buzzer to signal to skilled worker Y on the same line. Information (buzzer signal) indicating that the buzzer has been operated is output to control device 100 . This is a technique for calling the attention of the uncoiler operator A, who is far away in the noise, to notice early signs of ear cracking. On the other hand, if the skilled worker X does not exist, the accuracy of the centering work is lowered, and the possibility of edge cutting increases. With the assistance of the skilled worker X, the skilled worker Y can perform not only the centering work in the uncoiler process but also the work based on the point of view of the skilled worker X who monitors the sign of edge cut in the side trimmer process. becomes possible. A method of acquiring learning data and teacher data in a learning scene in which skilled worker X assists and skilled worker Y operates adjustment device 12 in this manner is referred to as a first learning method.

Further, when the uncoiler operator himself has a skilled skill, an assistant such as the skilled operator X is not required. FIG. 2B shows an example in which the skilled worker Y operates the control device 100 alone. For example, the skilled worker Y operates the control device 100 himself to perform the centering work while monitoring the captured image of the periphery of the side trimmer 40 . The skilled worker Y visually confirms the conveying direction (conveying attitude) of the steel material 10A pulled out by the uncoiler 20, and while confirming the edge width around the side trimmer 40 on the downstream side via the monitor, Centering work is performed while detecting signs of cut and making necessary corrections to prevent edge cut. A method of acquiring learning data and teacher data in a learning scene in which the skilled worker Y operates the adjustment device 12 while monitoring the vicinity of the side trimmer 40 on the monitor is hereinafter referred to as a second learning method.

FIG. 2(C) shows a skilled worker X who is gazing at the edge cut in the vicinity of the side trimmer 40 and an unskilled worker B who operates the control device 100 with reference to the assistance of the skilled worker X. Give an example. The unskilled worker B operates the adjusting device 12 while receiving assistance from the skilled worker X. A method of acquiring learning data and teacher data in such a learning scene is hereinafter referred to as a third learning method. The learned model creation device 200 may create a learned model under any one of the learning scenes described above, or may create a learned model by combining information obtained in different scenes.

The control device 100 automatically controls the adjustment device 12 so as to imitate the instructions of the skilled worker X, for example, using the learned model created by the learned model creation device 200 . By doing so, even if the skilled worker X is absent, it is possible to prevent the edge from being cut. Also, the control device 100 automatically controls the adjustment device 12 so as to imitate the operation of the skilled worker Y, for example, using the learned model created by the learned model creation device 200 . By doing so, even if the uncoiler operator A is an inexperienced operator, it is possible to prevent the edge from being cut.

For example, the control device 100 uses the learned model described above, and based on the image data from the first camera 71 and the image data from the second camera 72, determines the transport posture ( A signal (hereinafter referred to as an automatic centering assist signal) is generated for operating the adjusting device 12 for adjusting the transport position. The automatic centering assist signal is, for example, a command signal for causing the adjusting device 12 to perform control according to similar operations. The controller 100 outputs the generated auto-centering assist signal to the adjusting device 12 to control the adjusting device 12 to adjust the axial deviation of the steel material 10A. Then, the control device 100 outputs the generated automatic centering assist signal as a control signal to the adjustment device 12 to adjust the axial deviation of the steel material 10A.

FIG. 3 is a cross-sectional view showing the functional configuration of the steel processing AI system 2. As shown in FIG. FIG. 3 shows an example of a plurality of rollers not shown in FIG. Although omitted in FIG. 1, a main leveler 50 and a shearing section 60 may be provided downstream of the side trimmer 40 . Like the rough leveler 30, the main leveler 50 is a device that includes a plurality of rollers that sandwich the steel material 10A from above and below, and that reduces strain on the steel material 10A. The shearing unit 60 is a device that cuts the steel material 10A in both axial directions of the coil (a direction orthogonal to the direction of crime) to cut the steel material 10A into a specified size.

[Control device]
FIG. 4 is a block diagram showing an example of the functional configuration of the control device 100. As shown in FIG. The control device 100 includes, for example, a machine operation unit 110, a machine display unit 120, an acquisition unit 130, a processing unit 140, a storage unit 150, a control unit 160, a communication unit 170, and an interactive user interface unit. 190. The control device 100 is connected to the first camera 71 and the second camera 72 by wire.

The machine operation unit 110 includes, for example, some or all of the above-described operation lever 110A, various keys, buttons, dial switches, mouse, and the like. The mechanical display unit 120 is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electroluminescence) display device. The mechanical operation unit 110 and the mechanical display unit 120 may be a touch panel TP integrally formed.

The interactive user interface section 190 is, for example, a tablet terminal, and includes an operation section 191 and a display section 192 . The interactive user interface unit 190 displays how the skilled worker operates, displays points of attention of the skilled worker in another process, and receives labeling information given to data by the worker. In addition, the interactive user interface unit 190 can be operated to start and stop the system, to select the type of the steel material 10A, and to select the thickness of the steel material 10A. The interactive user interface unit 190 receives a selection operation and outputs information indicating the details of the operation.

The storage unit 150 is realized by, for example, RAM (Random Access Memory), ROM (Read Only Memory), flash memory, SD card, register, hard disk drive, and the like. The storage unit 150 stores image data 151, operation history information 152, and learned model data 153, for example. Image data 151 includes image data output from first camera 71 and image data output from second camera 72 . Image data is given a time stamp, for example. The operation history information 152 includes information indicating the details of operations received by the machine operation unit 110. As described above, this information is provided with time stamps synchronized with the time stamps of the image data. The trained model data 153 is data indicating the parameters, connection structure, function characteristics, etc. of the trained model created by the trained model creation device 200 .

The communication unit 170 has hardware for connecting to the network NW using a cellular network, a Wi-Fi network, or the like. For example, the communication unit 170 has an antenna, a transmitting/receiving device, and the like. The communication unit 170 communicates with the trained model generating device 200 and the external server 300 via the network NW. The communication unit 170 is not limited to this, and may be connected to and communicate with the learned model generating apparatus 200 by wire.

The acquisition unit 130 includes, for example, an image acquisition unit 131 and an electrical signal acquisition unit 132 . The processing unit 140 includes, for example, a data storage unit 141, a streaming video transmission unit 142, a display control unit 143, and an inference unit 144. The controller 160 includes, for example, a manual controller 161 and an automatic controller 162 . Some or all of the functions of the processing unit 140 and the control unit 160 are implemented by a processor such as a CPU (Central Processing Unit) executing a program (software) stored in its own storage unit. . In addition, some or all of the functions of these components are implemented in hardware such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), and GPU (Graphics Processing Unit). (including circuitry), or may be realized by cooperation of software and hardware. The program may be stored in advance in a storage device such as a HDD (Hard Disk Drive) or flash memory, or may be stored in a removable storage medium such as a DVD or CD-ROM. It may be installed by being worn.

The image acquisition unit 131 , for example, acquires image data from the first camera 71 and stores it in the storage unit 150 as part of the image data 151 . Also, the image acquisition unit 131 acquires image data from the second camera 72 and stores it in the storage unit 150 as part of the image data 151 . For example, when a trained model is created by the trained model creation device 200, the image acquisition unit 131 acquires image data as learning data for creating a learned model. On the other hand, when automatically controlling the adjustment device 12, the image acquisition unit 131 outputs the image data acquired in real time to the inference unit 144 as input data of the learned model. Further, the image acquisition unit 131 outputs image data acquired in real time to the streaming video transmission unit 142 for monitoring by the operator.

The image acquisition unit 131 may add labeling information indicating that the image data is good learning data to the image data selected by the skilled worker from among the acquired image data. The timing at which the labeling information is added (selected as good learning data by a skilled worker) may be when the captured image is displayed in real time on the interactive user interface unit 190. This may be when the captured image is later read out and displayed on the interactive user interface section 190 . This can facilitate data cleansing when acquiring learning data. Note that these processes related to labeling information may be executed in the trained model creation device 200 .

The electrical signal acquisition unit 132 acquires information (hereinafter referred to as operation history information) indicating the operation content of the adjustment device 12 received via the machine operation unit 110, and stores the information in the storage unit 150 as part of the operation history information 152. store in The operation history information includes, for example, a centering signal, which is an electric signal when the operation lever 110A is operated in the centering operation. Further, the operation history information includes the data of the image captured by the first camera 71 and the second camera 72, and the position information in the image of the point of interest focused on by the skilled worker X and the skilled worker Y while performing the centering work. may include information given to

For example, when a learned model is created by the learned model creation device 200, the electrical signal acquisition unit 132 acquires, as teacher data, operation history information to which the same time stamp as image data as learning data is added. On the other hand, when manually controlling the adjustment device 12 , the electrical signal acquisition unit 132 outputs information indicating the details of the operation acquired in real time to the manual control unit 161 .

For example, the operation history information can be obtained by the skilled worker Y, for example, at the entrance of the rough leveler 30 in a state where the image captured by the first camera 71 is displayed on the machine display unit 120 and the interactive user interface unit 190. This is a history of operations performed while visually confirming the uncoiler position P2 and monitoring the machine display section 120 and the interactive user interface section 190. FIG. Without being limited to this, in a state in which both the image captured by the first camera 71 and the image captured by the second camera 72 are displayed on the machine display unit 120, the machine display unit 120 and the interactive user It may be a history of operations performed by the skilled worker Y while monitoring the interface unit 190 . All of these are examples of operation history information in the second learning method described above. The operation history information is not limited to this, and may be operation history information in the first learning method or the third learning method described above.

In addition, the electric signal acquisition unit 132 is for signaling the (unskilled) uncoiler worker A on the same line based on the output from the operation unit (not shown) operated by the skilled worker X. A buzzer signal may be acquired and included in the operation history information. Furthermore, the electrical signal acquisition unit 132 includes the signal for determining the line stop acquired via the machine operation unit 110 in the operation history information in order to prevent memory tightness due to acquisition of unnecessary data when the line is stopped. good too. In addition, the electric signal acquisition unit 132 is included in the machine operation unit 110 when the steel material 10A starts to flow stably without the operator operating the machine operation unit 110 during the first coil unwinding. When the automatic button is pressed, the position of the steel material 10A at that time may be acquired as an initial reference position that allows stable flow, and may be included in the operation history information. The electrical signal acquisition unit 132 also receives a sensor signal from a sensor that detects whether the line is moving, or a buzzer signal from a buzzer that receives a cue for contacting skilled workers in a distant place in an emergency. may be acquired and included in the operation history information.

The data storage unit 141 transmits the image data acquired by the image acquisition unit 131 and the operation history information acquired by the electrical signal acquisition unit 132 to the trained model creation device 200 and the external server 300 via the communication unit 170. , to be stored.

The streaming video transmission section 142 outputs the image data acquired in real time by the image acquisition section 131 to the interactive user interface section 190 . The interactive user interface unit 190 causes the display unit 192 to display the input image data, thereby providing the operator with an image captured by the first camera 71 (or the second camera 72 is also included) in real time. can be done.

The display control unit 143 causes the machine display unit 120 to display predetermined information. The display control unit 143 also outputs predetermined information to the interactive user interface unit 190 and causes the display unit 192 to display various operation screens, guidance images, and the like.

The inference unit 144 uses the trained model to generate an auto-centering assist signal for the adjustment device 12 based at least on the image data from the first camera 71 . The reasoning unit 144 may also use the trained model to generate an auto-centering assist signal for the adjustment device 12 based on the image data from the second camera 72 .

In addition, the inference unit 144 inputs image data acquired in real time (or information acquired from the image data) to the learned model, thereby performing an operation similar to the operation of the skilled operator on the adjustment device 12 (hereinafter referred to as described as a similar operation) may be estimated. The similar operation is, for example, the content of operation for adjusting the amount of adjustment for the positional deviation of the steel material 10A in the width direction perpendicular to the conveying direction on the upstream side of the cutting position in the conveying direction. Specifically, it includes tilting the operation lever 110A to the right by XX degrees in order to move it to the right in the width direction by XX cm. Specific processing by the inference unit 144 will be described in detail after the processing by the trained model generation device 200 is described.

The display control unit 143 causes the machine display unit 120 and the interactive user interface unit 190 to display similar operations (how skilled workers operate) estimated by the inference unit 144 using the learned model. By doing so, even when the manual control is switched, the unskilled uncoiler operator A can control the adjustment device 12 with reference to the recommended manual lever operation.

In addition, the inference unit 144 may input image data (or information acquired from the image data) acquired in real time to the trained model to estimate the focus of the skilled worker on the adjustment device 12. . In this case, the trained model uses, as training data, positional information of a point of interest in the image that the skilled worker X and the skilled worker Y focused on while performing centering work, using the first camera 71 and the second camera 72. Information attached to the data of the captured image is used. The display control unit 143 causes the machine display unit 120 and the interactive user interface unit 190 to display the point of interest of the skilled worker estimated by the inference unit 144 using the learned model.

The manual control section 161 controls the adjustment device 12 based on information received by the operation lever 110A and other mechanical operation sections 110 . Note that the manual control unit 161 switches from automatic control to manual control when an operator's operation is input via the machine operation unit 110 while automatic control is being performed by the automatic control unit 162 .

Automatic control unit 162 automatically controls adjusting device 12 based on the auto-centering assist signal generated by reasoning unit 144 . For example, the automatic control unit 162 outputs an automatic centering assist signal to a PLC (Programmable Logic Controller) to control the adjusting device 12 and perform automatic centering control. For example, if the operation of the estimated skilled worker is to tilt the operation lever 110A to the right by XX degrees, an automatic centering assist signal for moving XX cm to the right in the width direction is generated and sent to the adjustment device 12. Output. By doing so, it is possible to automatically reproduce the control when a skilled worker operates via the machine operation unit 110 .

Further, the automatic control unit 162 determines the start timing and end timing of the automatic control, starts the automatic control when the start timing is reached, and ends the automatic control when the end timing is reached. For example, the automatic control section 162 starts automatic control when an automatic button included in the machine operation section 110 is pressed. When the coiled steel material 10A approaches the end and the manual button included in the machine operation unit 110 is pressed, the automatic control unit 162 cancels the automatic control and returns to manual operation. The control device 100 repeats the estimation process by the inference section 144 and the control signal output by the automatic control section 162 from the start of the automatic control to the end of the automatic control. It should be noted that the end timing of automatic control also includes a case where a condition for switching to manual control is satisfied.

In addition, the automatic control unit 162 determines whether the control based on the automatic centering assist signal generated by the inference unit 144 is within a specified range. The adjustment device 12 is automatically controlled based on the auto-centering assist signal. On the other hand, if it is not within the specified range, the automatic control unit 162 executes control within the specified range. The control within the specified range may be set in advance, or may be the control based on the automatic centering assist signal generated by the reasoning unit 144, which is changed to the control within the specified range. By doing so, it is possible to guarantee the output even for unknown input data, and it is possible to limit the output obtained from the inference result by deep learning so that it is limited within a specified range. In other words, the automatic control unit 162 has a fail safe function, and can safely perform control even when a failure due to an erroneous operation or malfunction occurs, thereby improving reliability. Therefore, in order to perform control based on the inference result of the inference unit 144, fail-safe measures such as limiting the output obtained from the inference result within a specified range are essential. Details will be described later.

In addition, for the same reason as the fail-safe function, the automatic control unit 162 is designed so that the operator who is monitoring the operation can manually correct the operation correctly when the output result of the inference unit 144 moves unexpectedly. It has a Take Over function for For example, when the automatic control unit 162 receives an operation through the machine operation unit 110 while executing automatic control based on the operation content estimated by the inference unit 144, the operation received through the machine operation unit 110 switch to manual control based on

For example, the automatic control unit 162 controls each unit based on limiters related to the time and number of control outputs set so that an excessive lever operation signal is not erroneously output to the adjustment device 12 when the automatic centering assist signal is output. You can For example, when the uncoiler operator A manually operates the operation lever 110A during automatic control, the automatic control unit 162 immediately reduces the gain of the automatic centering control to interrupt the automatic control in order to avoid a collision with a person. , take over with priority given to manual. Further, when the emergency stop button is pressed in an emergency, the automatic control section 162 may immediately stop automatic control and return to manual operation.

FIG. 5 is a diagram showing an example of processing of the control device in a learning scene. For example, the image acquisition unit 131 acquires image data captured by the first camera 71 and image data captured by the second camera 72 as learning data for creating a trained model. At the same time, the electrical signal acquisition unit 132 acquires an electrical signal (operation history information) that is the result of operating the mechanical operation part 110 as teacher data. The data storage unit 141 transmits the image data acquired by the image acquisition unit 131 and the operation history information acquired by the electrical signal acquisition unit 132 to the trained model creation device 200 and the external server 300 via the communication unit 170. , to be stored.

The data storage unit 141 also outputs the image data acquired by the image acquisition unit 131 to the streaming video transmission unit 142 . Streaming video transmission unit 142 outputs the input image data to interactive user interface unit 190 and causes display unit 192 to display it. Also, the interactive user interface unit 190 outputs the operation details of the skilled worker input via the operation unit 191, the electrical signal acquisition unit 132 acquires them as part of the operation history information, and includes them in the training data. may

FIG. 6 is a diagram illustrating an example of processing of the control device in a control scene. For example, the image acquisition unit 131 outputs image data acquired in real time from the first camera 71 and the second camera 72 to the inference unit 144 as input data for the trained model. When automatic control is being executed, the reasoning unit 144 uses the learned model to generate an automatic centering assist signal based on the input image data and outputs it to the automatic control unit 162 . Automatic control unit 162 automatically controls adjusting device 12 based on the auto-centering assist signal generated by reasoning unit 144 .

In addition, the inference unit 144 inputs the image data obtained in real time to the trained model to estimate a similar operation similar to that of the skilled operator on the adjustment device 12 . The display control unit 143 causes the machine display unit 120 and the interactive user interface unit 190 to display the estimated similar operation. Similar operations may be displayed both during automatic control and during manual control.

When the manual control is being executed, the electrical signal acquisition unit 132 outputs information indicating the operation details acquired in real time to the manual control unit 161 . The manual control section 161 controls the adjustment device 12 according to the input operation content.

[Learned model creation device]
FIG. 7 is a block diagram showing an example of the functional configuration of the trained model creation device 200. As shown in FIG. The trained model creation device 200 includes, for example, a processing unit 240, a storage unit 250, and a communication unit 270. The storage unit 250 and the communication unit 270 may have the same configuration as the storage unit 150 and the communication unit 170 described above, and detailed description thereof will be omitted. Also, the trained model generating device 200 may have the same configuration as the interactive user interface section 190 described above. Also, trained model creation device 200 may be directly connected to first camera 71 , second camera 72 , and machine operation unit 110 .

The processing unit 240 includes, for example, an acquisition unit 241, a data collection unit 242, a learning processing unit 243, and a verification unit 244. Some or all of these functional units are implemented by, for example, a processor such as a CPU executing a program (software) stored in its own storage unit. Some or all of the functions of these components may be realized by hardware (including circuitry) such as LSI, ASIC, FPGA, GPU, etc., or cooperation between software and hardware. It may be realized by work. The program may be stored in advance in a storage device such as an HDD or flash memory, or may be stored in a removable storage medium such as a DVD or CD-ROM. may be installed.

The acquisition unit 241 acquires information (hereinafter referred to as focus information) regarding points on which the skilled worker pays attention, and stores the information in the focus information 252 of the storage unit 250 . For example, the acquisition unit 241 obtains information about what the skilled worker pays attention to in the control device 100 and the steel material 10A and what the skilled worker pays attention to (hereafter referred to as the hearing). content information) is acquired as information of interest. In addition, the acquisition unit 241 detects the line of sight of the skilled worker with a line of sight detection device (not shown), thereby focusing on information (hereinafter referred to as line of sight information) indicating the location where the skilled worker is paying attention. You may acquire it as information. Furthermore, the acquisition unit 241 may acquire hearing content information, including the reason for focusing, by interviewing the skilled worker based on the recorded line-of-sight information of the expert obtained by the line-of-sight detection device. In this way, by combining the detection result of the line-of-sight detection device and the hearing from the skilled worker, it is possible to acquire attention information including the reason why the skilled worker focused on that point, and to obtain more accurate attention. You can also get point information.

Further, the acquisition unit 241 may acquire information obtained by associating an image captured by the first camera 71 or the second camera 72 with information indicating a location focused by the skilled worker as the focused information. Note that the skilled worker here includes the skilled worker X, who is excellent in the technique of giving instructions to the uncoiler worker A while watching the cut edges near the side trimmer 40, and the skilled worker X who operates the control device 100 to perform centering. At least one of skilled workers Y with excellent skills is included.

For example, the skilled worker X pays attention to changes in the length of the ears in the width direction at the cutting position, changes in the balance in the width direction of the left and right ears, and changes in distortion of the cut ears. For example, skilled worker Y pays attention to the relative positional relationship between the steel material 10A and the laser beam that indicates the reference position during transportation of the steel material 10A, and changes in the contact points between the rollers of the rough leveler 30 and the steel material 10A.

Note that the administrator who installs the first camera 71 and the second camera 72 takes an image of the location (including the location that the skilled worker is paying attention to) based on the information acquired by the acquisition unit 241. First, the installation positions of the first camera 71 and the second camera 72 are determined.

For example, the first camera 71 is installed so as to capture an image of a predetermined area (hereinafter referred to as a first area of interest) that the skilled worker pays attention to at the cutting position. The second camera 72 is installed so as to capture an image of a predetermined area (hereinafter referred to as a second area of interest) that a skilled worker pays attention to during the centering work. The first area of interest is, for example, an area where the length in the width direction of the ears at both ends cut by the side trimmer 40 can be obtained. The second area of interest is, for example, an area including the steel plate edge position corresponding to the point of contact between the roller of the rough leveler 30 and the steel material 10A. Details will be described later.

The first area of interest and the second area of interest are determined by the manager based on, for example, interviews with skilled worker X and skilled worker Y and results of a line-of-sight detection test. The area of interest is determined by listening to what skilled workers X and Y pay attention to in machines and materials (steel materials) and what they pay attention to. Note that the area of interest may be determined by the administrator based on the above-described information of interest other than the hearing.

Note that the area of interest may be determined by the learning processing unit 243 by deep learning using image processing results such as image data from the first camera 71 and image data from the second camera 72 and information of interest as input information. For example, the learning processing unit 243 extracts the feature points and rules indicated by the information of interest based on the image recognition result of the image data, and determines the area of interest based on the extraction result.

The acquisition unit 241 also acquires raw data for creating a trained model. For example, the acquisition unit 241 acquires image data captured by the first camera 71 from the external server 300 and outputs the data to the data collection unit 242 . The acquisition unit 241 also acquires image data obtained by imaging with the second camera 72 from the external server 300 and outputs the data to the data collection unit 242 . The acquisition unit 241 also acquires information indicating the details of the operation performed on the adjustment device 12 from the external server 300 and outputs the information to the data collection unit 242 .

Note that the image data output from the first camera 71, the image data output from the second camera 72, and the operation history information received by the machine operation unit 110 of the control device 100 are synchronized as described above. is given a time stamp.

The data collection unit 242 stores the raw data acquired by the acquisition unit 241 in the storage unit 250 as part of the raw data 251 . In addition, the data collection unit 242 extracts, as learning data, information indicating the transition of the point of interest focused by the skilled worker from the image data acquired by the acquisition unit 241, and stores the information as part of the learning data 253. . Further, the data collection unit 242 extracts the centering signal by the skilled worker as teacher data from the operating machine history information acquired by the acquisition unit 241 and stores it as a part of the teacher data 254 . By doing so, the learning processing unit 243 can generate a trained model that has learned the skill (implicit knowledge) possessed by the expert.

Note that the data collection unit 242 may transmit the acquired raw data, the extracted learning data, and the teacher data to the external server 300 for storage, and read them from the external server 300 when creating a trained model. Further, the data collection unit 242 extracts effective data from the raw data by cleansing, for example, based on the labeling information, performs preprocessing on the extracted image data, and performs DL (Deep Learning). ) and given to the neural network model as learning data.

For example, the data collection unit 242 performs image preprocessing by a CNN (Convolutional Neural Network) that obtains a position where a feature (point of interest) can be easily grasped from the image data obtained by imaging with the first camera 71 . For example, the data acquisition unit 242 uses a CNN model (hereinafter referred to as a CNN2 model) to obtain a plurality of positions indicating the length of the left and right ear widths cut from the image data. Information indicating the ear width value (hereinafter referred to as ear width value information) is obtained by image recognition, and information obtained by adding a time stamp to the ear width value information is stored in the storage unit 250 as part of the learning data 253. .

The data collection unit 242 also performs image preprocessing using a CNN model (hereinafter referred to as a CNN1 model) that obtains a position where features can be easily grasped from the image data obtained by imaging with the second camera 72, for example. For example, the second camera 72 is laid out in a place where the amount of axial displacement of the coil support portion 10 of the steel material 10A can be clearly determined, and where there is no reflection of light or shadows and does not interfere with the work of the operator. A standard industrial GigE camera is used.

FIG. 8 is a diagram showing an example of learning data extracted from the image data of the second camera 72. As shown in FIG. The data collection unit 242 cuts the image data D1 of the second camera 72 into an appropriate size to generate the normalized data D2 so that the feature amount of the image becomes clear and easy to learn. For example, the data collection unit 242 refers to the focus information 252, determines the size and position of the normalized data D2 so as to cut out a predetermined size including the focus of the skilled worker, and determines the determined size and position. acquire normalized data D2 normalized by .

For example, the data collection unit 242 generates information (hereinafter referred to as edge position information) obtained by image recognition of the steel plate edge position corresponding to the contact point between the roller of the rough leveler 30 and the steel material 10A using the CNN2 model. For example, the data collection unit 242 acquires edge position information indicating the steel plate edge positions in the image data D1 based on the binarized image D3 in which only the steel plate edge positions are white. The data collection unit 242 adds the time stamps added to the image data D1 to the generated normalized data D2 and the generated edge position information, and stores them in the storage unit 250 as part of the learning data 253. .

FIG. 9 is a diagram illustrating a configuration example of a CNN model. As a machine learning model, which is a trained model, an encoder/decoder CNN model that excels in segmentation was used.

For example, the learning processing unit 243 creates a learned model that has been learned using a plurality of images obtained by imaging a predetermined area (first area of interest) that the skilled worker pays attention to at the cutting position.

For example, the learning processing unit 243 creates a learned model roughly configured by two systems, ie, an uncoiler process and a trimmer process, which have different dead times. FIG. 10 is a diagram showing an example of a conceptual diagram of a learning software configuration by the learning processing unit 243. As shown in FIG.

First, the learning processing unit 243 inputs the edge position information with the same time stamp as learning data and the centering signal T1 as teacher data to the RNN1 model to generate an uncoiler system model with a short dead time. The RNN1 model is, for example, a model for predicting an automatic centering assist signal that reflects the points of interest of skilled worker Y on the uncoiler side. The centering signal T1 is a history of operations performed by a skilled operator or the like in any of the learning methods described above. Note that the learning processing unit 243 may further input the rotation speed of the coil support unit 10 to the RNN1 model as learning data.

Next, the learning processing unit 243 allows, for example, the skilled operator Y to observe the image monitor of the side trimmer at a distance, foresee signs of edge splitting from the image, and manually operate the operation lever 110A to avoid edge splitting. A centering signal T2 during operation is read from the raw data 251. FIG. Note that the centering signal T2 may be a history of operations performed by a skilled operator or the like in any of the learning methods described above.

Then, the learning processing unit 243 inputs the ear width information to which the same time stamp is given as learning data and the centering signal T2 as training data to the RNN2 model, and predicts the ear clipping prevention signal for automatic centering correction by reasoning. Generate a side trimmer system model. The RNN2 model is, for example, a model for predicting an automatic centering assist signal that reflects the attention point of a skilled worker X on the side trimmer side.

In the estimation process by the control device 100, an automatic centering assist signal is generated by synthesizing the estimation result of the uncoiler system model and the estimation result of the side trimmer system model.

The learning processing unit 243 treats the uncoiler process and the side trimmer process as independent models, generates a trained model composed of two models, and synthesizes the estimated results of the respective models to reduce wasted time. Different control systems can be treated as one model. Details will be described later.

Without being limited to this, the learning processing unit 243 may generate a learned model based on the simulation result. Even if learning data is input to the RNN model created as described above and inference is made, the correct answer rate may not increase. In this case, first, the learning processing unit 243 performs a simulation of the RNN model with the synthesized input/output learning data set so as to match the experimental data for which the correct answer is known. You may review the parameters. An input/output learning data set is a set of learning data and teacher data with the same time stamp.

Furthermore, in order to make the results of simulations conducted using actual data more similar to the details of lever operations performed by actual workers, changes in the steel material position over short periods of time are ignored, and predictions are made only for position changes over long periods of time. The LSTM time step may be tuned for output.

Also, the number of data required for learning may be confirmed by simulation. FIG. 11 is a diagram showing an example of a schematic configuration of the RNN model used by the learning processing unit 243. As shown in FIG. As shown in the figure, an LSTM (Long short-term memory) layer is used for the intermediate layer, making it possible to use more past information for learning.

In order to confirm the validity of the RNN model used, the verification unit 244 provides a data set for which the answer is known in advance as simulation data and confirms that the inference result of the learned RNN is correct, thereby confirming the validity of the model. Calculates information indicating For the simulation data, the edge of the simulation steel plate was randomly moved, and when the threshold was exceeded from the no-operation range, input was made to move it within the threshold. Signal input data was created from the calculation formula in physical conditions. .

FIG. 12 is a diagram showing an example of simulation results. The solid line is simulation ideal data, and the dashed line is the AI inference result. As can be seen from the results shown in the figure, it can be seen that the input direction, input time, and input timing of the signals are substantially the same.

On the other hand, the verification unit 244 may determine the minimum required number of learning data by simulation and verify whether the actual data is the correct operation for centering. In this way, data that can be regarded as typical model data can be labeled by the operator using the interactive user interface unit 190, thereby shortening the data cleansing time. Also, when it is determined that the steel material is stably flowing, the uncoiler operator A turns on the automatic button included in the interactive user interface unit 190, and the information indicating this operation content is included in the learning data and teacher data. , the reference position may be acquired.

[Estimation processing by control device 100]
Next, the estimation processing by the control device 100 will be described in detail. FIG. 13 is a diagram showing an example of input data to a trained model extracted from image data. FIG. 14 is a conceptual diagram of estimation processing by the control device 100 . FIG. 15 is a flowchart showing an example of estimation processing by the control device 100. As shown in FIG. First, the operator selects the type of steel material 10A via the interactive user interface section 190 . By doing so, it is possible to create a learned model for each type of steel based on the data acquired for each type of steel.

First, the inference unit 144 performs image preprocessing by CNN to obtain positions where features can be easily grasped from the image data obtained by imaging with the second camera 72 . For example, the inference unit 144 obtains the normalized data D12 by cutting the image data D11 into an appropriate size so that the feature amount of the image becomes clear and easy to learn. Further, the inference unit 144 obtains image recognition information (edge position information ) to generate a binarized image D14 indicating edge position information in the image data D11. That is, the binarized image D14 is an inference result restored to the original data size. The inference unit 144 acquires information obtained by adding a time stamp to the binarized image D14 as input data (step S1).

Next, the inference unit 144 gives the input data acquired in step S1 to the uncoiler system model (learned model) of the RNN to acquire an estimation result.

In addition, the inference unit 144 performs image preprocessing by CNN to obtain a position where features can be easily grasped from image data of the left ear obtained by imaging with the first camera 71 . For example, the inference unit 144 obtains the left ear width value cut by the steel plate edge and the blade of the side trimmer 40 by CNN from the image data of the left ear by image recognition, and the recognition result (left ear width value information) is timed. The stamped information is acquired as input data (step S2).

Furthermore, the inference unit 144 performs image preprocessing by CNN to obtain a position where features can be easily grasped from the image data of the right ear obtained by imaging with the first camera 71 . For example, the inference unit 144 obtains the right ear width value cut by the steel plate edge and the blade of the side trimmer 40 by CNN from the image data of the right ear by image recognition, and the recognition result (right ear width value information) is timed. The stamped information is acquired as input data (step S3).

Next, the inference unit 144 gives the input data acquired in step S2 and the input data acquired in step S3 to the side trimmer model (learned model) of the RNN to acquire an estimation result.

Then, the inference unit 144 generates an automatic centering assist signal by RNN inference by synthesizing the estimation result by the uncoiler system model and the estimation result by the side trimmer system model (step S4).

Similarly, the inference unit 144 may estimate similar operation content.

FIG. 16 is a flow chart showing an example of a processing flow of automatic control in the control device 100. As shown in FIG. The automatic control unit 162 starts the automatic control when the timing for starting the automatic control is reached (step S11). When automatic control is started by the automatic control unit 162, the inference unit 144 inputs image data obtained in real time to the learned model and generates an automatic centering assist signal (step S12). Next, the automatic control unit 162 controls the adjusting device 12 based on the automatic centering assist signal generated by the reasoning unit 144 (step S13). Then, the automatic control unit 162 ends the automatic control when the end timing of the automatic control is reached (step S14). On the other hand, if the automatic control end timing has not been reached, the automatic control unit 162 returns to step S11 and repeats the process.

FIG. 17 is a flow chart showing an example of a fail-safe and Take Over process flow in the control device 100. As shown in FIG. The control device 100 may apply multiple interlocks so as not to output the automatic centering assist signal based on the estimation of the learned model as it is.

For example, when the automatic centering assist signal estimated by the inference unit 144 is input (step S21), the automatic control unit 162 (1) determines whether or not the output is permitted (step S22). If it is in the output permission state, the automatic control unit 162 (2) determines whether or not the automatic output itself is abnormal (step S23). If the automatic output itself is not abnormal, the automatic control unit 162 (3) determines whether or not the input at the time of human operation is the same as the automatic output (step S24). If the input and the automatic output at the time of human operation are the same, the automatic control unit 162 satisfies all of these conditions (1) to (3). The control signal for the automatic control based on this is used as it is (step S25).

On the other hand, if it is determined in step S22 that the output is not permitted, the automatic control unit 162 returns to step S21 and repeats the process. In this case, it must be allowed.

In addition, if the automatic output itself is abnormal in the judgment of step S23, and if the input when the human operation is entered and the automatic output are not the same (including the case where there is no human operation) in the judgment of step S24, the automatic control The unit 162 cancels the output permission (step S26), and can prevent abnormal operation due to the control output.

FIG. 18 is a flow chart showing an example of the automatic output permission processing flow in the control device 100 . In order to prevent automatic output permission from being easily made, it is necessary to follow a predetermined procedure. The predetermined procedure includes, for example, a method in which the operator inputs specific pattern data in a predetermined order via the machine operation unit 110, and a method in which the operator instructs permission for automatic control via the machine operation unit 110. It includes a method of starting automatic control before a certain period of time elapses.

First, the automatic control unit 162 executes a procedure for permitting automatic output (step S31), and determines whether or not a predetermined permission procedure has been set (step S32). In the determination of step S32, if the setting is made according to the predetermined permission procedure, the automatic control unit 162 sets the automatic control output permission state (step S33). On the other hand, if it is determined in step S32 that the setting has not been made according to the predetermined permission procedure, the automatic control unit 162 sets the automatic control output disallowed state (step S34).

By doing so, automatic output can be permitted only when output permission is set according to the procedure. In other words, automatic output is not permitted if you do not follow the steps and make the settings. In addition, automatic output permission can be canceled easily without going through a procedure.

FIG. 19 is a diagram showing a comparison between the control signal based on the estimation result and the operation timing of the skilled worker. A thin solid line is information indicating a feature point (steel plate edge position corresponding to the contact point between the roller of the rough leveler 30 and the steel material 10A) image-recognized from the image captured by the second camera 72 using the learned model. A thick solid line is information indicating a control signal based on the result estimated by the trained model. A thick dashed line is information indicating the operation history of the skilled worker Y. FIG. It can be seen that in the marks M1 to M3, control signals based on inference results are output at timings substantially similar to those of the expert's operation.

[Interactive User Interface]
First, an example of operation functions included in the interactive user interface section 190 will be described. The interactive user interface portion 190 includes a steel plate selection button, a model OK button, an automatic button, and a manual button. When the operator presses the button for labeling the data with the type of steel sheet, the steel plate selection button turns on when it is ON, and turns off when it is further pressed. When the operator presses a button that recognizes that the data is correct, the model OK button turns ON when SW. The automatic button is a button for starting automatic centering assist when the strip starts to flow without frequent lever operation at the beginning of coil unwinding. Then press it again to switch to manual mode and turn off the light. The manual button is a button for stopping the system operation in case of emergency and returning to manual mode. When the manual button is pressed, the switch is turned on to turn on the switch. When pressed again, the switch is turned off to turn off the switch.

Next, an example of display functions included in the interactive user interface section 190 will be described. A display function to be described below is executed under the control of the display control unit 143 .

For example, during automatic centering, the interactive user interface unit 190 displays lever operation and displays information for teaching the operator the lever operation direction. The interactive user interface unit 190 displays the clipped trimmer's ear image to the operator on the clipped trimmer's ear image monitor portion. The interactive user interface portion 190 displays the end of the coil and displays information to inform the operator that the end of the coil is near. The interactive user interface portion 190 displays the data acquisition status and displays the starting, data capturing and finished statuses. The interactive user interface unit 190 displays an error and displays information notifying that an error has occurred during data acquisition.

FIG. 20 is a diagram showing an example of a display screen in the interactive user interface section 190. As shown in FIG. The display screen 400 has a board specification selection button 401 on the upper part of the left screen, and a switching button 403 for switching between manual control and automatic control on the lower part. The middle part of the left screen of the display screen 400 is a display area 402 showing the direction of the machine operation signal learned from the expert. On the right side of the display screen 400, there is a screen that can monitor the width of the scraps (commonly called lugs) of the steel material cut by the W.S. and D.S. trimmers in the distant side trimmer process.

FIG. 21 is a diagram showing another example of the display screen in the interactive user interface section 190. As shown in FIG. On the display screen 410, the expected change in ear width and the prediction of how the ear width will change from now on according to the operator's operation can also be displayed by AR overlaid on the monitor image. Using this display screen 410, a model OK button is provided on the screen for confirming whether or not the currently acquired data is suitable as model data for a skilled worker as a model for work. For example, labeling information is added to the screen data displayed on the display screen 410 when the model OK button is operated. This makes it possible to extract only the learning data recognized as correct by the skilled worker from a huge amount of collected data, which is effective in cleansing the learning data.

FIG. 22 is a diagram for explaining comparison of centering operations. FIG. 22A is a diagram for explaining conventional centering work. In the conventional centering work, the skilled worker X visually monitors the vicinity of the cutting position P1 by the side trimmer 40 from the image of the monitor camera showing the point of interest of the side trimmer 40, recognizes the sign of the cut edge, and , and informed the skilled worker Y of the right direction and the left direction by continuous motions of hand gestures. The skilled worker Y interprets the hand gestures of the skilled worker X to correct the centering and avoid the cut edge.

FIG. 22B is a diagram for explaining a centering operation when collecting teacher data. Here, the skilled worker X is not required, and only the skilled worker Y is required. The interactive user interface section 190 of the control device 100 displays an image of the vicinity of the cutting position P1 by the side trimmer 40 . The skilled operator Y operates the operation lever 110A while visually confirming the uncoiler position P2 at the entrance of the rough leveler 30 while monitoring the interactive user interface section 190. FIG. The operation history information at this time becomes part of the teacher data.

FIG. 22C is a diagram for explaining the centering work that is automatically controlled by the control device 100. FIG. Since the control device 100 stores a learned model, information based on an image taken near the cutting position P1 by the side trimmer 40 and an image taken at the uncoiler position P2 at the entrance of the rough leveler 30 is stored in the learned model. , the operation by the skilled operator is estimated, and the adjustment device 12 is controlled according to the estimated operation content.

According to the embodiments described above, control can be performed using a learned model obtained by paying attention to tacit knowledge possessed by a skilled worker.

Moreover, since most of the skills of skilled workers are tacit knowledge and cannot be verbalized, conventional expert systems cannot acquire many important skills of skilled workers. According to the present embodiment, it is possible to take advantage of the feature of automatically learning the feature amount of deep learning, learn the tacit knowledge of the expert as it is, and control the machine following the expert. In addition, it is possible to learn tacit knowledge distributedly possessed by a plurality of skilled workers, and to control the machine by consolidating the work of a plurality of skilled workers.

(Second embodiment)
The control device 100 may cause the interactive user interface unit 190 to display information (for example, the above-mentioned attention information) regarding the attention point of the steel material 10A of the skilled worker when acquiring the operation history information as the teaching data. good. The information of interest displayed on the interactive user interface unit 190 may be text information, which is superimposed on the screen that displays the image captured by the first camera 71 (or the second camera 72) in real time. , a marker or the like that makes the point of interest conspicuous.

Further, the control device 100 may cause the interactive user interface unit 190 to display a recommended manual lever operation based on the past operation history by a skilled worker when acquiring operation history information as teaching data. The recommended manual lever operation is the operation performed by skilled workers in the past to prevent ear splits. The operation history information displayed on the interactive user interface unit 190 may be text information, which is superimposed on the screen displaying the image captured by the first camera 71 (or the second camera 72) in real time. Alternatively, it may be a marker or the like that highlights the operation history.

In the present embodiment, the control device 100 controls the uncoiler while the interactive user interface unit 190 displays information of interest by the skilled worker or recommended manual lever operation based on the past operation history by the skilled worker. The operation history of operator A or expert operator Y while monitoring the interactive user interface unit 190 is acquired, and the learned model creating device 200 creates a learned model using this operation history as teacher data.

By doing this, it is possible to generate a trained model that has learned the skills (tacit knowledge) possessed by one or more skilled workers, and to generate a series of trained models that have been distributed and possessed as tacit knowledge by a plurality of skilled workers. It is possible to integrate and acquire the proficiency skills of

As described above, the mode for carrying out the present invention has been described using the embodiments, but the present invention is not limited to such embodiments at all, and various modifications and replacements can be made without departing from the scope of the present invention. can be added.

DESCRIPTION OF SYMBOLS 1... Steel material processing system 10... Coil support part 20... Uncoiler 30... Rough leveler 40... Side trimmer 71... First camera 72... Second camera 100... Control device 200... Learned model creation device, DESCRIPTION OF SYMBOLS 110... Machine operation part 120... Machine display part 130... Acquisition part 131... Image acquisition part 132... Electric signal acquisition part 140... Processing part 141... Data storage part 142... Streaming video transmission part 143... Display control unit 144 Reasoning unit 150 Storage unit 160 Control unit 170 Communication unit 190 Interactive user interface unit

Claims (8)

Cutting that cuts the ends of a sheet-like member that is being conveyed along the longitudinal direction without being cut in the direction perpendicular to the conveying direction, which is pulled out from the coiled state to the conveying path. An operation signal indicating operation of an adjustment device for acquiring image data from a camera that captures an image of a predetermined area including a position and adjusting the conveying position of the sheet-like member on the upstream side in the conveying direction of the cutting position. an acquisition unit that acquires the data of
The images acquired by the acquiring unit are added to a trained model trained using data of a plurality of images obtained by imaging a predetermined area focused by a skilled worker at the cutting position as learning data and data of the operation signal as teaching data. a reasoning unit that generates a signal for operating the adjustment device by inputting the data of
A control system comprising: a control unit that controls the adjustment device based on the operation content estimated by the inference unit. a learning processing unit that learns the learned model using the image data as learning data and the operation signal data as teacher data;
2. The control system of claim 1. The control unit displays the data of the image on a display unit that is visible to an operator who operates the adjustment device ,
The acquisition unit further acquires operation signal data indicating details of operation of the adjusting device by the skilled worker and another skilled worker input during the period in which the data of the image is displayed on the display unit. Acquired,
The learning processing unit learns the learned model using data of the image as learning data and data of an operation signal indicating details of operation of the adjusting device by the other skilled worker as teacher data .
3. A control system according to claim 2. The control unit causes a display unit visible to an operator who operates the adjustment device to display the operation content estimated by the inference unit.
A control system according to claim 1 . Further equipped with an operation part,
The control unit assigns labeling information indicating good teacher data to an image selected by a skilled worker through the operation unit from among the plurality of images.
A control system according to claim 1 . The control unit determines whether the control based on the operation content estimated by the inference unit is within a specified range, and if it is not within the specified range, executes the control within the specified range.
A control system according to claim 1 . The control unit switches to manual control based on the operation received through the operation unit when an operation is accepted through the operation unit while performing automatic control based on the operation content estimated by the inference unit.
A control system according to claim 1 . Cutting that cuts the ends of a sheet-like member that is being conveyed along the longitudinal direction without being cut in the direction perpendicular to the conveying direction, which is pulled out from the coiled state to the conveying path. Acquiring a plurality of captured images obtained by capturing a predetermined area focused on by a skilled worker at a position,
Acquiring information indicating an operation history of an adjusting device that adjusts the conveying position of the sheet member on the upstream side in the conveying direction of the cutting position;
Learning in which the relationship between the operation of the adjusting device and the conveying state of the sheet-shaped member is learned by using information obtained based on the plurality of captured images as learning data and information indicating the operation history as teaching data. create a model,
How to create a trained model.

Images