JP7753682B2 - Monitoring device and monitoring method - Google Patents

Description

The present invention relates to an apparatus and method for monitoring an operator performing a specified operation.

A method is known in which the operation of an operating tool placed on a control panel is detected by correlating the coordinates of the operating tool on the control panel with the time when the operating tool was operated by an operator (hereinafter also referred to as "operator"); the coordinates of the operator's line of sight on the control panel are detected by correlating them with the time when the line of sight was detected; an image showing the coordinates of the operator's operating position on the control panel is acquired by correlating it with the time when the image was acquired; and when the operator operated the operating tool, it is determined whether the operator's line of sight and the coordinates of the operating position were aligned with the coordinates of the operating tool, thereby determining whether the operator acted appropriately without looking away (Patent Document 1).

Japanese Patent Application Laid-Open No. 2018-13958

However, with the above-mentioned conventional technology, if the operator's body or operating tools become obstacles and the camera cannot detect the operator's operating position, it is not possible to determine whether the operator's actions were appropriate, resulting in a problem of reduced accuracy in the determination.

The problem that this invention aims to solve is to provide a monitoring device and monitoring method that improves the accuracy of determining whether an operator's behavior is appropriate.

The present invention solves the above problem by using an imaging device attached to the arm of an operator who performs an operation and having an attachment part for attachment to the palm side of the operator's forearm to acquire an image including an operation target, which is an object on which the operator performs an input operation, and part or all of the operator's hand performing the input operation on the object, and determining whether or not an operation identified from the positional relationship between the operator's hand and the operation target in the acquired image is a predetermined operation.The present invention also solves the above problem by using an imaging device attached to the arm of an operator who performs an operation and having an attachment part for attachment to the palm side of the operator's forearm to acquire an image including an operation target, which is an object on which the operator performs an input operation, and part or all of the operator's hand in contact with the object, and determining whether or not the operator's operation is a predetermined operation using the acquired image.

This invention makes it possible to acquire images suitable for determining the operator's actions, thereby improving the accuracy of determining the operator's actions.

1 is a block diagram showing one embodiment of a monitoring system including a monitoring device according to the present invention. 2 is a side view of an operator's arm showing an example of the positional relationship between an imaging device 11, a detection device 12, and a display device 13 shown in FIG. 1. FIG. FIG. 2 is an explanatory diagram illustrating an example of a trained model used in the operation identification unit of FIG. 1 . FIG. 10 is an explanatory diagram showing an example of a predetermined procedure. FIG. 2 is an explanatory diagram illustrating an example of a trained model used in the operation procedure determination unit of FIG. 1 . 2 is a flowchart showing an example of an information processing procedure in the monitoring system of FIG. 1 .

Embodiments of a monitoring device and monitoring method according to the present invention will be described below with reference to the accompanying drawings.

[Surveillance System]
1 is a block diagram showing a monitoring system 1 according to the present invention. The monitoring system 1 is a device that monitors the operation of an operator who performs an operation on a certain operation object, and can be used, for example, to monitor whether a worker engaged in assembly work at a vehicle assembly factory has operated a tool in accordance with a predetermined procedure, or to monitor whether a mechanic engaged in vehicle maintenance at a vehicle dealership (hereinafter also referred to as a "dealer") has performed maintenance in accordance with a procedure defined in a manual. The operator whose operation is monitored by the monitoring system 1 is not particularly limited, and examples thereof include a factory worker, a dealer's mechanic, etc.

As shown in FIG. 1, the monitoring system 1 includes an imaging device 11, a detection device 12, a display device 13, and a monitoring device 14. The devices that make up the monitoring system 1 are connected via known means such as wired or wireless LAN so that they can exchange data with each other. The number of imaging devices 11, detection devices 12, and display devices 13 is not particularly limited, as long as there is at least one of each. Furthermore, the imaging device 11, detection device 12, and display device 13 do not need to be installed together with the monitoring device 14, and may be installed in a location remote from the monitoring device 14. For example, the imaging device 11, detection device 12, and display device 13 may be installed near the assembly line of an assembly plant, and the monitoring device 14 may be installed in a central control room away from the assembly line or on a server in a remote location away from the assembly plant.

The imaging device 11 is a device for acquiring image data of surrounding objects, such as a camera equipped with an imaging element such as a CCD, an ultrasonic camera, or an infrared camera. Objects include not only the operator but also objects present around the operator. Examples of objects include switches and touch panels located around the worker, parts being assembled or tools being used by the worker, and vehicles being serviced by dealership mechanics. The imaging device 11 is attached to the operator's arm, allowing it to reliably acquire images of the operator operating the object. The imaging device 11 has an attachment portion for attachment to the operator's arm, and can be attached to the palm side of the operator's upper arm using, for example, a hook-and-loop fastener or an adhesive pad.

The detection device 12 is a device for detecting the operator's situation and the conditions around the operator, and includes any type of sensor. Examples of sensors include an electromyograph that measures the operator's myoelectric potential, a microphone that captures sounds around the operator as audio data, an accelerometer attached to the operator to detect the operator's physical movements, and a thermometer that measures the temperature around the operator. The electrodes included in the electromyograph may be needle electrodes, surface electrodes, or wire electrodes. The electromyograph is attached to the operator's body, particularly the arm including the upper arm and/or forearm, using, for example, an adhesive pad or hook-and-loop fastener, and measures the myoelectric potential of the part of the body that comes into contact with the attached electromyograph.

Microphones include stand microphones, close-talking microphones, and gun microphones, and may be omnidirectional or directional. Communication methods may be either wired or wireless. The microphone captures the operator's voice and the voices of people around the operator, as well as sounds resulting from the operator's actions. Examples of sounds resulting from user actions include the sound generated from a switch operated by a worker, the sound generated from a speaker when a worker touches a touch panel, the sound of parts engaging when a worker connects multiple parts, and the operating sounds of tools used by dealership mechanics. The microphone is installed in a position where it can detect sounds around the operator, such as the assembly line in an assembly plant, the workbench, and the tools used by the worker.

The accelerometer is not particularly limited, but is preferably large enough to be attached to the operator's body, particularly the arm including the upper arm and/or forearm. For example, a small accelerometer using MEMS (Micro Electro Mechanical Systems) can be used. The thermometer is also not particularly limited, and may be a contact thermometer or a non-contact thermometer using infrared rays. The monitoring device 14 obtains detection results from these sensors at predetermined time intervals.

The display device 13 is a device for notifying the operator and/or supervisor of operations monitored by the monitoring device 14. The supervisor does not need to be near the operator; it may be located in the factory's central control room or a remote location away from the factory. The display device 13 may be, for example, an LCD display or a projector, and may also be equipped with a speaker. The display device 13 is installed, for example, in a location near the operator, such as the worker's work area in an assembly factory, where it can notify the operator of necessary information, or in a location where it can notify a supervisor located away from the operator of necessary information. The display device 13 may also be attached to the operator as a wearable terminal. Furthermore, the imaging device 11, detection device 12, display device 13, and monitoring device 14 may be integrated into a single wearable terminal attached to the operator.

An example of the relative positions of the imaging device 11, detection device 12, and display device 13 is shown in Figure 2. In Figure 2, the imaging device 11 is attached to the operator's forearm A1 using the attachment portion 11a, and the detection device 12 is attached to the operator's upper arm A2 using an adhesive pad. The operator holds an operation target O in their hand H. If the viewing angle of the imaging device 11 is within the range indicated by VA, the imaging device 11 can capture an image including a portion of the operator's hand H and the operation target O. The image data captured by the imaging device 11 and the detection results of the detection device 12 are sent to the monitoring device 14 via wireless communication. In Figure 2, the display device 13 is provided in front of the operator, allowing the operator to check the display on the display device 13.

Returning to Figure 1, the monitoring device 14 is a device for monitoring whether an operator has properly performed a certain operation. The image data and sensor detection results used for this monitoring are acquired at predetermined time intervals from the imaging device 11 and detection device 12, respectively. The monitoring device 14 uses the processor 15 to perform functions such as processing the acquired data, determining whether the operation was properly performed, and outputting the results of this determination. The processor 15 includes a ROM (Read Only Memory) 152 in which a program is stored, a CPU (Central Processing Unit) 151 which is an operating circuit that functions as the monitoring device 14 by executing the program stored in the ROM 152, and a RAM (Random Access Memory) 153 which functions as an accessible storage device.

[Operation monitoring section]
The program used in the monitoring device 14 of this embodiment includes an operation monitoring unit 2, which is a functional block for implementing functions such as processing acquired data, determining whether an operation has been properly performed, and outputting the results of the determination in the monitoring device 14. The operation monitoring unit 2 has the functions of acquiring image data from the imaging device 11, acquiring detection results from the detection device 12, identifying an operation performed by a user based on the acquired data, determining whether the identified operation has been properly performed in a predetermined procedure, and outputting the determination result. As shown in FIG. 1 , the operation monitoring unit 2 includes an image acquisition unit 21, an operation identification unit 22, an operation determination unit 23, an operation procedure determination unit 24, and a determination result output unit 25. FIG. 1 illustrates each unit extracted for convenience.

The monitoring device 14 shown in Figure 1 has all of the above functional blocks, but a single monitoring device 14 does not need to have all of the functional blocks; some of the above functional blocks may be provided in other equipment included in the monitoring system 1 or in another information processing device not shown. For example, in the monitoring system 1 of Figure 1, the judgment result output unit 25 may be provided in the display device 13. In this case, the functions of the judgment result output unit 25 are executed using the CPU, ROM, and RAM of the display device 13.

Furthermore, it is not necessary for all of the processing of each functional block to be performed by a single device; the functions of each functional block may be realized across multiple devices that are connected and capable of sending and receiving data. For example, in the monitoring system 1 of FIG. 1, some of the processing performed by the operation identification unit 22 may be performed by the imaging device 11, and the remaining processing may be performed by the monitoring device 14. In this case, some of the processing required to realize the function of the operation identification unit 22 is performed using the CPU, ROM, and RAM of the imaging device 11. As another example, some of the processing performed by the operation determination unit 23 and/or operation procedure determination unit 24 may be performed by the imaging device 11, and the remaining processing may be performed by the monitoring device 14.

The functions performed by each functional block of the operation monitoring unit 2 are explained below.

The image acquisition unit 21 has the function of acquiring an image including the operator performing the operation and the operation object operated by the operator. In particular, the image acquisition unit 21 of this embodiment acquires image data from the imaging device 11 attached to the operator's arm, and the acquired image data includes part or all of the operator's hand H and the operation object O operated by the operator. The operator's hand H may be either the right hand or the left hand, as long as at least part of the hand H is included. However, the operation can be determined more accurately if part or all of the operator's palm and the operation object O are included in the image.

The operations performed by the operator are not particularly limited, and include any operation in which the operator makes some kind of input to the operation target. Specific examples include an operation in which an assembly plant worker touches a touch panel to change the display of an assembly line device, an operation in which an assembly plant worker connects a coupler connected to a sensor with a coupler connected to an electronic control unit (ECU), an operation in which an assembly plant worker tightens bolts using a tool to attach an exhaust manifold to an engine block, an operation in which a dealership mechanic fits a spark plug into an engine, and an operation in which a dealership mechanic tightens a bolt using a torque wrench.

Furthermore, the object to be operated is not particularly limited, and includes any object onto which the operator performs some kind of input operation. Examples include an ON/OFF switch, a touch panel user interface, parts such as couplers and exhaust manifolds, and tools such as torque wrenches and electric screwdrivers.

Furthermore, the image acquisition unit 21 can change the quality of the images acquired from the imaging device 11, and may lower the frame rate of the acquired time-series images (video) while the operation determination unit 23 (described later) is not processing image data, such as when the operator is not performing an operation. For example, if the operation determination unit 23 (described later) determines that the operator's hand movement does not correspond to a predetermined movement, the frame rate of the acquired time-series images may be lowered below a predetermined frame rate (e.g., 60 fps). The predetermined frame rate can be set to an appropriate value within the range in which the monitoring device 14 can appropriately monitor the operation. On the other hand, while the operator is actually performing an operation, the frame rate of the acquired time-series images may be increased above the predetermined frame rate.

The operation identification unit 22 has the function of identifying the operation performed by the operator from the image data acquired by the image acquisition unit 21. The operation identification unit 22 performs analysis such as pattern matching on the image data acquired from the imaging device 11 and classifies the objects contained in the image data. Next, it selects the operator from the classified objects and acquires data related to the operator's hand extracted from the image data. Similarly, it selects an operation target from the classified objects and acquires data related to the operation target extracted from the image data. Then, from this data, it recognizes the positional relationship between the operator's hand and the operation target and identifies the user's operation from the recognized positional relationship.

The correspondence between the positional relationship between the operator's hand H and the operation target O, and the operation the operator is attempting to perform is determined in advance for each operation and stored in a database such as database 16. The operation identification unit 22 can acquire the correspondence between the positional relationship and the operation from database 16 as necessary. Based on the correspondence acquired from database 16, the operation identification unit 22 identifies the operation the operator is attempting to perform from the image data acquired by the image acquisition unit 21.

As an example, let us consider a case where the image data acquired by the image acquisition unit 21 includes a state in which a worker working on a factory assembly line is holding a coupler in his left hand and reaching out his right hand to another coupler. In this case, the operation identification unit 22 searches for an operation corresponding to the positional relationship between the right hand extended toward the coupler and the coupler itself, based on the correspondence between positional relationships and operations acquired from the database 16. If the operations corresponding to this action include the operation of removing one coupler from an already mated pair of couplers and the operation of mating the coupler connected to the sensor with the coupler connected to the electronic control unit, the operation identification unit 22 determines that, because the worker is holding the coupler in his left hand, this positional relationship corresponds to the operation of mating the coupler connected to the sensor with the coupler connected to the electronic control unit.

As another example, let us consider a case where the image data acquired by the image acquisition unit 21 includes a state in which a dealer's mechanic is holding engine oil in his left hand and extending his right hand into the engine compartment. In this case, the operation identification unit 22 searches the correspondence between positional relationships and operations acquired from the database 16 for an operation corresponding to the positional relationship between the right hand extended into the engine compartment and the right hand inside the engine compartment. If the operations corresponding to this action are the operation of refilling engine coolant and the operation of refilling engine oil, the operation identification unit 22 determines that the positional relationship corresponds to the operation of refilling engine oil because the mechanic is holding engine oil in his left hand.

The operation identification unit 22 may also identify an operation using time-series images, such as a video, acquired by the image acquisition unit 21. The operation identification unit 22 acquires image data from the image acquisition unit 21 at predetermined time intervals and analyzes the image data. In this way, using time-series images with predetermined time intervals allows for more accurate operation identification. The time intervals can be set appropriately depending on the calculation capacity of the CPU 151. Alternatively or in addition to this, the operation identification unit 22 may identify an operation using the detection results of the detection device 12. For example, the operation of the operator may be detected from the value of the operator's myoelectric potential detected by an electromyograph attached to the operator's upper arm and the value of acceleration detected by an accelerometer attached to the operator's forearm, and the operation may be identified using the detected operation.

Furthermore, the operation identification unit 22 may identify an operation using a first trained model that has been trained to identify an operation from image data acquired by the image acquisition unit 21. The trained model is a model that has been trained in advance by machine learning so that appropriate output data can be obtained for certain input data, and includes at least a program that performs calculations to obtain output data from input data, and weighting coefficients (parameters) used in the calculations. When image data acquired by the image acquisition unit 21 is input as input data, the first trained model of this embodiment causes the computer (particularly the CPU 151 of the processor 15) to function so that, based on the input data, the operation that the operator is attempting to perform on the image is identified and output data is output. By using such a first trained model, operations can be identified even for image data that indicate positional relationships for which no correspondence has been previously established.

The first trained model of this embodiment is not particularly limited, but may be, for example, a first neural network 3 as shown in FIG. 3. The first neural network 3 includes a first input layer 31, a first intermediate layer 32, and a first output layer 33, each of which includes at least one neuron. The first input layer 31 receives first input data 34, which includes at least one of image data acquired by the image acquisition unit 21 and the detection result of the detection device 12, and outputs the input data to the first intermediate layer 32. The first intermediate layer 32 extracts the positional relationship between the operator's hand and the operation target from the data input from the first input layer 31. It then identifies the operation corresponding to the extracted positional relationship. The first output layer 33 outputs the data input from the first intermediate layer 32 as first output data 35, which includes the identified operation.

In the first intermediate layer 32, a correspondence between the positional relationship between the operator's hand and the operation object and the operation the operator is attempting to perform is established through machine learning to identify the operation the operator is attempting to perform. This makes it possible to identify the operation the operator is attempting to perform from the positional relationship between the operator's hand and the operation object, such as the positional relationship between the worker's hand and the switch, the positional relationship between the worker's hand and the touch panel user interface, the positional relationship between the worker's hand and the part, and the positional relationship between the mechanic's hand and the tool. In addition to the positional relationship between the operator's hand and the operation object, multiple factors, such as the operator's surroundings detected by the detection device 12, are associated with this correspondence, and parameters are set so that the appropriate operation is output for the input data. For example, the fact that the worker is holding a coupler in each hand and the mechanic is holding engine oil in his left hand are taken into account as parameters when identifying the operation.

The correspondences in the first intermediate layer 32 described above may be learned in advance through machine learning, or may be newly learned using first teacher data 36 including first input data 34 previously input to the first neural network 3 and first output data 35 previously output from the first neural network 3, or a trained model previously learned through machine learning may be further trained. This learning is performed by the first machine learning unit 22a included in the operation identification unit 22. The first teacher data 36 is stored in a database such as the database 16 shown in FIG. 1, for example, and can be acquired as needed. By learning using a combination of past first input data 34 and first output data 35, the operation identification unit 22 can more accurately identify an operation based on the positional relationship between the operator's hand and the operation target.

Returning to FIG. 1, the operation determination unit 23 has the function of using image data acquired by the image acquisition unit 21 to determine whether the operation identified by the operation determination unit 23 is a predetermined operation. A predetermined operation is an operation that is predetermined as an operation to be performed by an operator. For example, for a factory worker, operations such as mating a coupler connected to a sensor with a coupler connected to an electronic control unit (ECU), and tightening bolts using a tool to attach an exhaust manifold to an engine block are pre-set as predetermined operations. The operation determination unit 23 uses, for example, a series of images acquired by the image acquisition unit 21 to determine whether the operation identified by the operation determination unit 23 is a predetermined operation, and outputs the determination result to the operation procedure determination unit 24 and the determination result output unit 25.

The operation determination unit 23 can also acquire the movement of the operator's hand H from the image acquired by the image acquisition unit 21 and determine whether the acquired movement corresponds to a predetermined movement. If it is determined that the movement of the hand H corresponds to a predetermined movement, it determines whether the identified operation is a predetermined operation only from a predetermined time before the movement of the hand H was determined to be a predetermined movement to a predetermined time after the movement of the hand was determined to be a predetermined movement. In this way, it is possible to make a determination based only on the time before and after the movement of the hand H was determined to be a predetermined movement, which is when it is necessary to determine the operation. The movement of the operator's hand H can be acquired from an analysis of the image data acquired by the image acquisition unit 21, the value of the myoelectric potential detected by an electromyograph attached to the operator's upper arm A2, and the value of acceleration detected by an accelerometer attached to the operator's forearm A1. The predetermined movement is preset as the initial hand H movement of a predetermined operation and stored in the database 16, etc. The operation determination unit 23 acquires data of the predetermined movement from the database 16 as necessary.

When the operation determination unit 23 determines that the operation identified by the operation identification unit 22 corresponds to a predetermined operation, the operation procedure determination unit 24 has the function of determining whether the predetermined operation was performed in accordance with the predetermined procedure. In particular, the operation procedure determination unit 24 uses the time-series image data acquired by the image acquisition unit 21 to determine whether the predetermined operation was performed in accordance with the predetermined procedure. The predetermined procedure is set in advance for each predetermined operation and is stored in a database such as database 16.

The operation procedure determination unit 24 acquires the operations identified from the operation identification unit 22 at predetermined time intervals and grasps the positional relationship between the operator's hand and the operation target contained in the image data. It then grasps the operator's operations in chronological order and, each time, determines whether the identified operation matches the predetermined procedure described above. If there is no difference from the predetermined procedure until the identified operation is completed, the operation procedure determination unit 24 determines that the predetermined operation was performed according to the predetermined procedure. On the other hand, if the identified operation differs from the predetermined procedure, or if the identified operation deviates from the predetermined procedure, it determines that the predetermined operation was not performed according to the predetermined procedure. These determination results are output to the determination result output unit 25.

As an example, let us use Figure 4 to explain the case where the predetermined operation is the operation of mating two couplers. In Figure 4, coupler C1 held in the worker's left hand is mated with coupler C2 held in the worker's left hand. When the mating portions Z of couplers C1 and C2 are properly mated, an operating sound S is generated. The predetermined procedure in this case consists of four steps, as shown in Figure 4: Step 1) the worker holds couplers C1 and C2 in their left and right hands, respectively; Step 2) combines coupler C1 with coupler C2; Step 3) pushes couplers C1 and C2 together until the mating portions Z of the couplers engage and generate an operating sound S, and Step 4) releases couplers C1 and C2 upon generation of operating sound S. The operating procedure determination unit 24 acquires the identified operation from the operation identification unit 22 at predetermined time intervals and determines the positional relationship between the operator's hand mating the couplers and the couplers. It then chronologically tracks how the operator who engages the coupler operates the coupler, and each time determines whether the four steps described above match the identified operation.

If the coupler is successfully mated according to the steps described above, it is determined that the coupler has been properly mated according to the specified procedure. On the other hand, if the coupler was not pushed in enough in step 3 and the coupler was not properly mated, it is determined that the coupler was not properly mated. Here, the operation procedure determination unit 24 may use the detection results of the detection device 12 when determining whether the specified operation was performed according to the specified procedure. In this case, for example, the operator's actions in mating the coupler may be detected from the myoelectric potential value acquired by an electromyograph attached to the operator's upper arm and the acceleration value acquired by an accelerometer attached to the operator's forearm accelerometer, and the detected values may be used for the determination by the operation procedure determination unit 24. Furthermore, a microphone installed near the operator mating the coupler may detect the clicking sound S that occurs when the coupler mating portion Z engages, and the detected values may be used for the determination by the operation procedure determination unit 24.

As another example, let us consider a case where the predetermined operation is the operation of refilling engine oil. In this case, the predetermined procedure consists of four steps: 1) removing the oil filler cap, 2) pouring engine oil into the oil inlet, 3) closing the oil filler cap, and 4) checking the engine oil level with the oil level gauge. The operation procedure determination unit 24 acquires the operations identified from the operation identification unit 22 at predetermined time intervals and determines the positional relationship between the operator's hands when refilling oil and the engine oil and oil level gauge. It then chronologically determines the operations performed by the operator when refilling oil and determines each time whether the four steps described above match the identified operations.

If oil refilling is completed in accordance with the above steps, it is determined that the engine oil has been properly refilled in accordance with the specified procedure. In contrast, if the engine oil level is not confirmed in step 4), it is determined that the engine oil has not been properly refilled. Furthermore, as in the above example, the operation procedure determination unit 24 may use the detection results of the detection device 12 when determining whether the specified operation has been performed in accordance with the specified procedure. In this case, for example, the action of grasping the oil level gauge may be detected from the myoelectric potential value acquired by an electromyograph attached to the operator's upper arm and the acceleration value acquired by an accelerometer attached to the operator's forearm accelerometer, and these may be used in the determination by the operation procedure determination unit 24.

Furthermore, the operation procedure determination unit 24 may determine whether a predetermined operation has been performed in a predetermined procedure using a second trained model trained to determine whether a predetermined operation has been performed in a predetermined procedure using time-series image data acquired by the image acquisition unit 21 and/or the detection results of the detection device 12 that detects the conditions around the operator. In this embodiment, when time-series image data acquired by the image acquisition unit 21 and/or the detection results of the detection device 12 that detects the conditions around the operator are input as input data, the second trained model causes the computer (particularly the CPU 151 of the processor 15) to function so as to identify the operation the operator is attempting to perform in the image based on the input data and output data including a determination result as to whether the operator performed the identified operation in the predetermined procedure. By using such a second trained model, it is possible to determine whether an operation has been performed in a predetermined procedure even for image data showing a positional relationship for which no correspondence has been established in advance.

The second trained model is not particularly limited, but may be, for example, a second neural network 4 as shown in FIG. 5. The second neural network 4 includes a second input layer 41, a second intermediate layer 42, and a second output layer 43, each of which includes at least one neuron. The second input layer 41 receives second input data 44, which includes at least one of the time-series image data acquired by the image acquisition unit 21 and the detection results of the detection device 12, and outputs the input data to the second intermediate layer 42. The second intermediate layer 42 extracts the positional relationship between the operator's hand and the operation target from the data input from the second input layer 41. Next, it identifies an operation corresponding to the extracted positional relationship. It then determines whether the identified operation corresponds to a predetermined operation, and if the identified operation corresponds to the predetermined operation, it determines whether the predetermined operation was performed according to a predetermined procedure. The second output layer 43 outputs the data input from the second intermediate layer 42 as second output data 45, which includes information on whether the predetermined operation was performed according to the predetermined procedure.

In the second intermediate layer 42, similar to the first intermediate layer 32 described above, a correspondence between the positional relationship between the operator's hand and the operation target and the operation the operator is attempting to perform is established through machine learning in order to identify the operation the operator is attempting to perform. This correspondence is associated with multiple factors, such as the positional relationship between the operator's hand and the operation target, as well as the state of the operator's surroundings detected by the detection device 12, and parameters are set so that an appropriate operation is output for the input data. In addition, in the second intermediate layer 42, a correspondence between the identified operation and a predetermined procedure is established in order to determine whether the specified operation was performed in a predetermined procedure when the identified operation corresponds to a predetermined operation. Specifically, the correspondence between the data arranging the identified operations in chronological order and each step of the predetermined procedure is learned through machine learning, and the necessary parameters are set so that the identified operation can be appropriately compared with the predetermined procedure.

The correspondence relationships in the second intermediate layer 42 described above may be learned in advance by machine learning, or may be newly learned using second teacher data 46 including second input data 44 previously input to the second neural network 4 and second output data 45 previously output from the second neural network 4, or a trained model previously learned by machine learning may be further trained. This learning is performed by the second machine learning unit 24a provided in the operation procedure determination unit 24. The second teacher data 46 is stored in a database such as the database 16 shown in FIG. 1, for example, and can be acquired as needed. Learning using combinations of past second input data 44 and second output data 45 enables the operation procedure determination unit 24 to more accurately determine whether a predetermined operation has been performed according to the predetermined procedure.

Returning to FIG. 1, the judgment result output unit 25 has the function of acquiring the judgment results of the operation judgment unit 23 and the operation procedure judgment unit 24 and outputting them to an external device. In particular, if the operation judgment unit 23 judges that the operation performed by the operator is not a predetermined operation, it outputs to the display device 13 a message that the operation is inappropriate. Also, if the operation procedure judgment unit 24 judges that the predetermined operation was not performed according to the predetermined procedure, it outputs to the display device 13 a message that the operation is inappropriate. By outputting these judgment results to the display device 13 and displaying them on the display device 13, it is possible to notify the operator and/or supervisor that the predetermined operation was not performed appropriately.

The power supply of the monitoring device 14 may be turned on after it is confirmed from the detection result of the detection device 12 that the operator has started using the monitoring system 1. Furthermore, the determination results of the operation determination unit 23 and the operation procedure determination unit 24 may be output to a server (not shown).

[Processing in operation detection system]
The procedure for processing information by the monitoring device 14 will be described with reference to Fig. 6. Fig. 6 is an example of a flowchart showing information processing in the monitoring system 1 of this embodiment. The processing described below is executed by the processor 15 of the monitoring device 14 at predetermined time intervals.

First, in step S1, the image acquisition unit 21 uses its function to acquire from the imaging device 11 an image that includes part or all of the operator's hand performing the operation and the object being operated by the operator. For example, if a worker engaged in assembly work in a factory tightens a bolt using a tool, an image that includes part of the worker's hand, the bolt, and the tool is acquired.

In the following step S2, the operation identification unit 22 functions to detect the conditions around the operator using the detection device 12. For example, when a worker engaged in assembly work in a factory tightens a bolt using a tool, the worker's myoelectric potential is obtained using an electromyograph attached to the worker's arm.

In the following step S3, the operation identification unit 22 uses its function to identify the operation the operator is attempting to perform from the image data acquired by the image acquisition unit 21 and the detection results of the detection device 12. For example, if a worker engaged in assembly work at a factory is using a tool to tighten a bolt, it identifies that the worker is attempting to tighten a bolt and which bolt the worker is attempting to tighten.

In the following step S4, the operation determination unit 23 determines whether the identified operation corresponds to a predetermined operation. If the identified operation corresponds to a predetermined operation, the process proceeds to step S5. On the other hand, if the identified operation does not correspond to a predetermined operation, the process proceeds to step S6. For example, if a worker engaged in assembly work at a factory uses a tool to tighten a bolt, if tightening the bolt by the worker is a predetermined operation, the process proceeds to step S5. On the other hand, if the work of the worker does not include tightening a bolt, the process proceeds to step S6.

In step S5, the operation procedure determination unit 24 determines whether the specified operation was performed according to the specified procedure. If it is determined that the specified operation was performed according to the specified procedure, execution of the routine is stopped and the monitoring process is terminated. On the other hand, if it is determined that the specified operation was not performed according to the specified procedure, the process proceeds to step S6. For example, if a worker engaged in assembly work at a factory uses a tool to tighten a bolt, if the worker tightened the bolt according to the specified procedure, execution of the routine is stopped and the monitoring process is terminated. On the other hand, if the worker left the bolt loose, the process proceeds to step S6.

In step S6, the function of the judgment result output unit 25 outputs to the display device 13 that the identified operation was not a predetermined operation and that the predetermined operation was not performed properly according to the predetermined procedure. The display device 13 then receives the output of the judgment result output unit 25 and notifies the operator and/or supervisor that the operation was not performed properly. For example, if a worker engaged in assembly work in a factory uses a tool to tighten a bolt, the supervisor is notified that the worker did not tighten the bolt properly. After this display, the routine stops execution and the monitoring process ends.

As another example, when a worker engaged in assembly work at a factory is mating two couplers, for example, in step S1, image data is acquired by the imaging device 11 attached to the worker as a wearable terminal, and in step S2, sounds around the worker are detected by the operation identification unit 22. In the following step S3, based on the image data acquired by the image acquisition unit 21 and the detection result of the detection device 12, it is identified that the worker's operation is mating a coupler. In the following step S4, the operation determination unit 23 determines whether the identified operation corresponds to a predetermined operation. If the identified operation, which is the mating operation of the couplers, is a predetermined operation, the process proceeds to step S5. On the other hand, if the work of the worker does not include mating a coupler, the process proceeds to step S6. In step S5, the operation procedure determination unit 24 determines whether the predetermined operation was performed in the predetermined procedure. If it is determined that the coupler mating operation was performed in the predetermined procedure, the process stops execution of the routine and ends the monitoring process. On the other hand, if it is determined that the coupler fitting operation was not performed according to the predetermined procedure, the process proceeds to step S6. Then, in step S6, the function of the determination result output unit 25 outputs to the display device 13 that the specified coupler fitting operation was not the predetermined operation, or that the coupler fitting operation was not performed appropriately according to the predetermined procedure.

[Embodiments of the present invention]
As described above, the monitoring device 14 of this embodiment provides a monitoring device including: an image acquisition unit 21 that acquires an image including part or all of the operator's hand H and an operation object O operated by the operator using an imaging device 11 attached to the arm of the operator performing the operation; an operation determination unit 23 that determines whether the operation is a predetermined operation using the image acquired by the image acquisition unit 21; and a determination result output unit 25 that outputs the determination result of the operation determination unit 23. This makes it possible to reliably acquire an image of the operator operating the operation object O. Furthermore, it is possible to acquire an image suitable for determining the operator's action, thereby improving the accuracy of determining the operator's action.

Furthermore, according to the monitoring device 14 of this embodiment, the image acquisition unit 21 acquires an image including part or all of the operator's palm and the operation object. This makes it possible to more reliably acquire an image of the operator operating the operation object.

Furthermore, according to the monitoring device 14 of this embodiment, the imaging device 11 has an attachment portion for attachment to the palm side of the operator's upper arm A2. This makes it possible to more reliably acquire images of the operator operating an operation object.

Furthermore, the monitoring device 14 of this embodiment is equipped with an operation identification unit 22 that identifies the operation from the images acquired by the image acquisition unit 21, and the operation determination unit 23 determines whether the operation identified by the operation identification unit 22 is the predetermined operation from the series of images acquired by the image acquisition unit 21. This allows for more accurate operation identification.

Furthermore, according to the monitoring device 14 of this embodiment, the operation identification unit 22 identifies the operation using the time-series images acquired by the image acquisition unit 21 and the detection results of the detection device 12, which detects the conditions around the operator. This allows for more accurate operation identification.

Furthermore, according to the monitoring device 14 of this embodiment, when the operation determination unit 23 determines that the operation is not the predetermined operation, the determination result output unit 25 outputs to the display device 13 that the operation is inappropriate. This makes it possible to notify the operator and/or monitor that the operation is inappropriate.

Furthermore, according to the monitoring device 14 of this embodiment, the operation identification unit 22 identifies the operation using a first trained model that has been trained to identify the operation from the image acquired by the image acquisition unit 21. This makes it possible to identify an operation for a positional relationship that has not been set in advance.

Furthermore, according to the monitoring device 14 of this embodiment, the first trained model is a first neural network 3 that, when first input data 34 including at least one of the image acquired by the image acquisition unit 21 and the detection result of the detection device 12 that detects the state around the operator is input to the first input layer 31, outputs first output data 35 including the identified operation from the first output layer 33. The operation identification unit 22 includes a first machine learning unit 22a that trains the first neural network 3 using the first input data 34 previously input to the first neural network 3 and the first output data 35 previously output from the first neural network 3 as first teacher data 36, and the first machine learning unit 22a trains the first neural network 3. This makes it possible to more accurately identify operations for positional relationships that were not previously set.

Furthermore, the monitoring device 14 of this embodiment is provided with an operation procedure determination unit 24 that, when the operation identification unit 22 determines that the operation is the predetermined operation, determines whether the predetermined operation was performed according to a predetermined procedure, and the determination result output unit 25 outputs the determination result of the operation procedure determination unit 24. This makes it possible to determine the appropriateness of the procedure for performing the operation.

Furthermore, according to the monitoring device 14 of this embodiment, the operation procedure determination unit 24 determines whether the predetermined operation was performed according to the predetermined procedure using the time-series images acquired by the image acquisition unit 21. This makes it possible to more accurately determine whether the predetermined operation was performed according to the predetermined procedure.

Furthermore, according to the monitoring device 14 of this embodiment, when the operation procedure determination unit 24 determines that the specified operation was not performed according to the specified procedure, the determination result output unit 25 outputs to the display device 13 that the operation was inappropriate. This makes it possible to notify the operator and/or monitor that the operation was inappropriate.

Furthermore, according to the monitoring device 14 of this embodiment, the operation procedure determination unit 24 uses the time-series images acquired by the image acquisition unit 21 and the detection results of the detection device 12 that detects the state around the operator to determine whether the predetermined operation was performed in the predetermined procedure, using a second trained model that has been trained to determine whether the predetermined operation was performed in the predetermined procedure. This makes it possible to determine whether an operation was performed in a positional relationship that was not previously set, and the appropriateness of that procedure.

Furthermore, according to the monitoring device 14 of this embodiment, the second trained model is a second neural network 4 that, when second input data 44 including the image acquired by the image acquisition unit 21 and the detection results of the detection device 12 that detects the state around the operator is input to the second input layer 41, outputs second output data 45 including a determination result as to whether the predetermined operation was performed according to the predetermined procedure from the second output layer 43. The operation procedure determination unit 24 includes a second machine learning unit 24a that trains the second neural network 4 using the second input data 44 previously input to the second neural network 4 and the second output data 45 previously output from the second neural network 4 as second teacher data 46, and the second machine learning unit 24a trains the second neural network 4. This makes it possible to more accurately determine operations for positional relationships that have not been previously set and the appropriateness of the procedures.

Furthermore, according to the monitoring device 14 of this embodiment, the operation determination unit 23 acquires the movement of the operator's hand H from the image acquired by the image acquisition unit 21, determines whether the acquired movement corresponds to a predetermined movement, and if it is determined that the movement corresponds to the predetermined movement, determines whether the operation corresponds to the predetermined operation only from a time period from a predetermined time before the movement of the hand H was determined to be the predetermined movement to a time period from a predetermined time after the movement of the hand H was determined to be the predetermined movement. This makes it possible to reduce the power consumption of the monitoring device 14.

Furthermore, according to the monitoring device 14 of this embodiment, if it is determined that the movement does not correspond to a predetermined movement, the image acquisition unit 21 reduces the frame rate of the acquired time-series images. This makes it possible to reduce the amount of image data stored.

Furthermore, according to the monitoring method of this embodiment, in a monitoring method for monitoring an operator performing an operation using a processor, the processor acquires an image including all or part of the operator's hand H and the operation object O operated by the operator from an imaging device 11 attached to the operator's arm, determines whether the operation is a predetermined operation using the acquired image, and outputs a determination result as to whether the operation is the predetermined operation. This makes it possible to reliably acquire an image of the operator operating the operation object O. Furthermore, it is possible to acquire an image suitable for determining the operator's behavior, thereby improving the accuracy of determining the operator's behavior.

REFERENCE SIGNS LIST 1 monitoring system 11 imaging device 11a mounting portion 12 detection device 13 display device 14 monitoring device 15 processor 151 CPU
152...ROM
153...RAM
DESCRIPTION OF SYMBOLS 16...Database 2...Operation monitoring unit 21...Image acquisition unit 22...Operation identification unit 22a...First machine learning unit 23...Operation determination unit 24...Operation procedure determination unit 24a...Second machine learning unit 25...Determination result output unit 3...First neural network 31...First input layer 32...First intermediate layer 33...First output layer 34...First input data 35...First output data 36...First teacher data 4...Second neural network 41...Second input layer 42...Second intermediate layer 43...Second output layer 44...Second input data 45...Second output data 46...Second teacher data A1...Forearm A2...Upper arm C1, C2...Coupler H...Hand O...Operation object S...Operation sound VA...Visual angle Z...Fitting portion

Claims (17)

an image acquisition unit that acquires an image including an operation target, which is an object on which the operator performs an input operation, and a part or all of the operator's hand that performs the input operation on the object, using an imaging device attached to the arm of the operator performing the operation;
an operation determination unit that determines whether the operation identified from the positional relationship between the hand and the operation target in the image acquired by the image acquisition unit is a predetermined operation;
a determination result output unit that outputs a determination result of the operation determination unit ,
The imaging device is a monitoring device having an attachment portion for attachment to the palm side of the operator's forearm . an image acquisition unit that acquires an image including an operation target, which is an object on which the operator performs an input operation, and a part or all of the operator's hand that is in contact with the object, using an imaging device attached to the arm of the operator performing the operation;
an operation determination unit that determines whether the operation is a predetermined operation by using the image acquired by the image acquisition unit;
a determination result output unit that outputs a determination result of the operation determination unit ,
The imaging device is a monitoring device having an attachment portion for attachment to the palm side of the operator's forearm . The monitoring device described in claim 1 or 2, wherein the image acquisition unit acquires an image including part or all of the operator's palm and the operation target. an operation identification unit that identifies the operation based on a positional relationship between the hand and the operation target in the image acquired by the image acquisition unit;
The monitoring device according to any one of claims 1 to 3 , wherein the operation determination unit determines whether the operation identified by the operation identification unit is the predetermined operation from the series of images acquired by the image acquisition unit. The monitoring device according to claim 4 , wherein the operation identification unit identifies the operation using the time-series images acquired by the image acquisition unit and a detection result of a detection device that detects a state around the operator. The monitoring device according to claim 4 , wherein the determination result output unit outputs, to a display device, a message indicating that the operation is inappropriate when the operation determination unit determines that the operation is not the predetermined operation. The monitoring device according to any one of claims 4 to 6 , wherein the operation identification unit identifies the operation using a first trained model that is trained to identify the operation from the image acquired by the image acquisition unit. the first trained model is a first neural network in which, when first input data including at least one of the image acquired by the image acquisition unit and a detection result of a detection device that detects a state around the operator is input to a first input layer, first output data including the identified operation is output from a first output layer;
8. The monitoring device of claim 7, wherein the operation identification unit includes a first machine learning unit that trains the first neural network using first input data that was previously input to the first neural network and first output data that was previously output from the first neural network as first teacher data, and trains the first neural network using the first machine learning unit. an operation procedure determination unit that determines, when the operation identification unit determines that the operation is the predetermined operation, whether the predetermined operation has been performed in a predetermined procedure;
The monitoring device according to claim 4 , wherein the determination result output unit outputs the determination result of the operating procedure determination unit. The monitoring device according to claim 9 , wherein the operation procedure determination unit determines whether the predetermined operation has been performed in accordance with the predetermined procedure by using the time-series images acquired by the image acquisition unit. The monitoring device according to claim 9 or 10, wherein the determination result output unit outputs, to a display device, a message indicating that the operation is inappropriate when the operation procedure determination unit determines that the specified operation has not been performed in accordance with the specified procedure. The monitoring device according to any one of claims 9 to 11, wherein the operation procedure determination unit determines whether the predetermined operation has been performed according to the predetermined procedure by using a second trained model that has been trained to determine whether the predetermined operation has been performed according to the predetermined procedure, using the time- series images acquired by the image acquisition unit and a detection result from a detection device that detects the state around the operator. the second trained model is a second neural network in which, when second input data including the image acquired by the image acquisition unit and a detection result of a detection device that detects a state around the operator is input to a second input layer, second output data including a determination result as to whether or not the predetermined operation has been performed in the predetermined procedure is output from a second output layer;
The monitoring device described in claim 12, wherein the operation procedure determination unit includes a second machine learning unit that trains the second neural network using second input data that was previously input to the second neural network and second output data that was previously output from the second neural network as second teacher data, and trains the second neural network using the second machine learning unit. The operation determination unit
acquiring the hand movement of the operator from the image acquired by the image acquisition unit;
determining whether the acquired movement corresponds to a predetermined movement;
The monitoring device according to any one of claims 1 to 13, wherein, when it is determined that the movement corresponds to a predetermined movement, it determines whether the operation is the predetermined operation only during the period from a time before a predetermined time from when the movement of the hand was determined to be the predetermined movement to a time after a predetermined time from when the movement of the hand was determined to be the predetermined movement. The monitoring device according to claim 14 , wherein when it is determined that the movement does not correspond to a predetermined movement, the image acquisition unit reduces a frame rate of the acquired time-series images. A monitoring method for monitoring an operator performing an operation using a processor, comprising:
The processor:
acquiring, from an imaging device having an attachment portion to be attached to the palm side of the forearm of the operator and attached to the arm of the operator, an image including an operation target that is an object on which the operator performs an input operation, and a part or all of the hand of the operator performing the input operation on the object;
Identifying the operation based on a positional relationship between the hand and the operation target in the acquired image;
determining whether the operation is a predetermined operation;
and outputting a determination result as to whether the operation is the predetermined operation. A monitoring method for monitoring an operator performing an operation using a processor, comprising:
The processor:
acquiring, from an imaging device having an attachment portion to be attached to the palm side of the forearm of the operator and attached to the arm of the operator, an image including an operation target that is an object on which the operator performs an input operation and a part or all of the operator's hand that is in contact with the object;
Using the acquired image, it is determined whether the operation is a predetermined operation;
and outputting a determination result as to whether the operation is the predetermined operation.