JP7849466B2 - machine operating device - Google Patents

Description

This invention relates to a machine operating device.

Currently, various industrial machines such as robots, conveyors, machine tools, and mechanical equipment are in operation in industrial sectors such as manufacturing. Many of these industrial machines are equipped with control panels, and the equipment that controls each industrial machine, such as PLCs (Programmable Logic Controllers), NCs (Numerical Controllers), and control panels, also have control panels.

Industrial machinery controls often have numerous buttons and screens, making them complex to operate and requiring considerable time to master. Voice input interfaces allow users to perform desired operations simply by uttering voice commands. Therefore, efforts are being made to improve usability using voice input interfaces. Voice input interfaces allow for commands to be issued without touching the industrial machinery in a variety of situations, such as opening and closing doors, starting and pausing operation, turning lights on and off inside the machine, stopping the use of cutting fluid, and cleaning chips. Furthermore, because there is no need to directly operate buttons or screens, it allows for operations such as monitoring the machining progress during prototyping, operating a robot from outside a safety fence, or quickly stopping the machine by voice without having to approach a button if danger is detected.

Voice commands used to operate industrial machinery can be anticipated based on the type of industrial machinery, the installation site, and the nature of the operation. Therefore, anticipated voice commands can be created using grammar (syntax and vocabulary). See, for example, Patent Document 1.

Japanese Patent Application Publication No. 9-325787

In manufacturing sites such as factories where industrial machinery is installed, multiple machines are typically in operation. Furthermore, noise is constantly generated in these manufacturing environments. Therefore, when operating industrial machinery using only voice commands, processing noises and other sounds are simultaneously input into the microphone. This can reduce the accuracy of voice command recognition. In noisy environments, it's possible to consider methods that use gestures to capture images and recognize commands from the captured video, rather than relying solely on voice. This would eliminate the impact of noise. However, using only gestures limits the expressive power of commands compared to language, making it impractical in certain situations.
Therefore, there is a need for machine operation methods that can achieve both productivity and precision in manufacturing sites.

The machine operation device according to the present invention solves the above problem by integrating voice recognition based on voice detected by a microphone or the like, and face (mouth shape) or gesture recognition based on images detected by a camera or the like, for the operation of industrial machines such as machine tools and robots, thereby enabling machine operation that is more productive than manual operation.

Furthermore, one aspect of the present disclosure is a machine operating device that uses voice recognition of voice signals input at a manufacturing site as operating commands for operating industrial machinery, comprising: a voice recognition unit that recognizes first data indicating a predetermined operation based on a voice signal converted from voice uttered by a worker detected by a voice sensor; a video analysis unit that analyzes second data indicating a predetermined operation based on a video signal obtained by capturing the worker's movements with an imaging sensor; a command determination unit that determines an operating command for operating the industrial machinery based on the first data and the second data; and a command processing unit that processes the operating command determined by the command determination unit , wherein the machine operating device does not process an operating command if it recognizes that it is dangerous to execute according to at least one of the worker's work movements and standing position .

One aspect of this disclosure improves productivity by simplifying and streamlining operations, and ensures worker safety by enabling machine operation without contact with the machine itself.

This is a schematic hardware diagram of a machine operating device according to one embodiment of the present invention. A block diagram illustrating the schematic function of a machine operating device according to one embodiment of the present invention. This block diagram shows the schematic function of a machine operating device according to a modified version of the present invention. This block diagram shows a schematic function of a mechanical operating device according to another embodiment of the present invention. This figure shows an example of using multiple industrial machines for a task.

Embodiments of the present invention will be described below with reference to the drawings.
Figure 1 is a schematic hardware configuration diagram showing the main components of a machine operating device according to one embodiment of the present invention. The machine operating device 1 according to this embodiment can be mounted on a control device that controls industrial machinery 2 installed in a manufacturing site such as a factory. The machine operating device 1 can also be mounted on a personal computer attached to the control device, or on a computer such as a fog computer 6 or cloud server 7 connected to the control device via a wired or wireless network. The following description will be based on an example in which the machine operating device 1 according to this embodiment is mounted on a control device that controls industrial machinery 2.

The CPU 11 in the machine operating device 1 according to this embodiment is a processor that controls the machine operating device 1 as a whole. The CPU 11 reads the system program stored in the ROM 12 via the bus 22 and controls the entire machine operating device 1 according to the system program. The RAM 13 temporarily stores temporary calculation data, display data, and various data input from external sources.

The non-volatile memory 14 is composed of, for example, a memory backed up by a battery (not shown) or an SSD (Solid State Drive), and its stored state is maintained even when the power to the machine operating device 1 is turned off. The non-volatile memory 14 stores data acquired from the industrial machine 2, control programs and data read from external devices 72 via the interface 15, control programs and data input via the input device 71, and control programs and data acquired from other devices via the network 5. The control programs and data stored in the non-volatile memory 14 may be expanded into the RAM 13 when executed/used. In addition, various system programs, such as known analysis programs, are pre-written in the ROM 12.

Interface 15 is an interface for connecting the CPU 11 of the machine operation device 1 to an external device 72 such as a USB device. From the external device 72, control programs and setting data used to control the industrial machine 2 are read, for example. Control programs and setting data edited within the machine operation device 1 can also be stored in an external storage means via the external device 72. The PLC 16 executes a ladder program and outputs signals to the industrial machine 2 and its peripheral devices (for example, tool changers, actuators such as robots, and multiple sensors 3 such as temperature sensors and humidity sensors attached to the industrial machine 2) via the I/O unit 19 to control them. It also receives signals from various switches on the control panel located on the main body of the industrial machine 2 and peripheral devices, performs the necessary signal processing, and then passes them to the CPU 11.

Interface 20 is an interface for connecting the CPU of the machine operating device 1 to a wired or wireless network 5. Other industrial machines 4, such as machine tools and electrical discharge machines, as well as a fog computer 6 and a cloud server 7, are connected to the network 5, and they exchange data with the machine operating device 1.

The display device 70 displays data obtained as a result of the execution of various data, programs, etc., loaded into memory, which are output via the interface 17. The display device 70 may also be equipped with LED indicators and warning lamps to show the status of the machine. In addition, the input device 71, which consists of a keyboard, pointing device, etc., transmits commands, data, etc., based on the operator's operations to the CPU 11 via the interface 18.

Interface 21 is an interface for connecting the CPU 11 of the machine operating device 1 with the voice sensor 73. The voice sensor 73 may be a sound-collecting device such as a microphone. The voice sensor 73 may be attached to, for example, the input device 71, a machine operating panel (not shown), a pendant (portable machine operating panel), etc. The voice emitted by the worker detected by the voice sensor 73 is passed to the CPU 11 as an audio signal.

Interface 23 is an interface for connecting the CPU 11 of the machine operating device 1 to the image sensor 74. The image sensor 74 may be, for example, a camera. The image sensor 74 may be attached to, for example, a display device 70, a machine operating panel (not shown), a pendant (portable machine operating panel), etc. It may also be attached above the position where the machine is installed so that it can capture images of the area around the machine. The movements of the worker's face and body detected by the image sensor 74 are passed to the CPU 11 as video signals.

The axis control circuit 30, which controls the axes of the industrial machine 2, receives the axis movement command amount from the CPU 11 and outputs the axis command to the servo amplifier 40. The servo amplifier 40 receives this command and drives the servo motor 50 that moves the axis of the machine tool. The axis servo motor 50 has a built-in position and speed sensor, and the position and speed feedback signal from this position and speed sensor is fed back to the axis control circuit 30 to perform position and speed feedback control. Although only one axis control circuit 30, servo amplifier 40, and servo motor 50 are shown in the hardware configuration diagram of Figure 1, in reality, there are as many as the number of axes of the industrial machine 2 to be controlled.

Figure 2 is a schematic block diagram showing the functions of a machine operating device 1 according to one embodiment of the present invention. Each function of the machine operating device 1 according to this embodiment is realized by the CPU 11 of the machine operating device 1 shown in Figure 1 executing a system program and controlling the operation of each part of the machine operating device 1.

The machine operating device 1 of this embodiment includes an audio signal acquisition unit 100, an audio recognition unit 110, an audio signal acquisition unit 120, an audio analysis unit 130, a command determination unit 140, a command processing unit 150, and an output unit 160.

The audio signal acquisition unit 100 acquires the audio signal detected by the audio sensor 73. It then extracts the audio signal that is recognized as a single utterance from the acquired audio signal. The audio signal detected by the audio sensor 73 is mainly based on the voice uttered by the worker. The audio signal acquisition unit 100 may extract the audio signal corresponding to a single utterance by the worker from among these. For example, this can be done by defining a point where the audio signal is below a predetermined level Lv _th for a predetermined period of Ts _th or more as the boundary of the audio, and extracting the audio signal that is between the boundary and remains at a predetermined level Tn _th or higher for a predetermined period as the audio signal corresponding to a single utterance. Alternatively, other known audio signal analysis techniques may be used for extracting the audio. The audio signal extracted by the audio signal acquisition unit 100 is output to the speech recognition unit 110.

The speech recognition unit 110 performs speech recognition processing on the speech signal corresponding to a single utterance by the operator, which is input from the speech signal acquisition unit 100. The recognized result is then output to the command determination unit 140. For the speech recognition processing, any known speech recognition algorithm, such as dynamic time stretching, hidden Markov models, or neural network methods, may be used. The recognition result of the speech signal output by the speech recognition unit 110 may be, for example, a string representing the words spoken in the speech signal, and a confidence level of the recognition result. In this case, multiple recognition results and the confidence levels of each recognition result may be output together. Furthermore, when using a neural network, the recognition result of the speech signal may be a feature quantity indicating the characteristics of the utterance in the speech signal.

The video signal acquisition unit 120 acquires the audio signal detected by the imaging sensor 74. Then, it extracts the video signal corresponding to the worker's speech from the acquired video signal. The video signal detected by the imaging sensor 74 may be an image of the worker's face, especially the mouth. Alternatively, the video signal detected by the imaging sensor 74 may be an image of the worker's upper body, including both hands. The video signal acquisition unit 120 may, when the audio signal acquisition unit 100 extracts the audio signal corresponding to a single utterance by the worker, extract the video signal acquired at the same timing as the extracted audio signal as the video signal corresponding to a single utterance by the worker. The video signal extracted by the video signal acquisition unit 120 is output to the video analysis unit 130.

The video analysis unit 130 analyzes the video signal corresponding to a single utterance by the worker, which is input from the video signal acquisition unit 120. The analysis results are then output to the command determination unit 140. If the video signal captures the worker's face, particularly their mouth, known algorithms such as AAM (Active Appearance Model) or LipNet can be used for video analysis processing. If the video signal captures the worker's upper body, including both hands, known gesture recognition algorithms using hidden Markov models or multilayer neural networks can be used. The video analysis results output by the video analysis unit 130 may include, for example, a string representing the operation of the industrial machine 2 obtained by analyzing the mouth movements recognized from the video signal, and a confidence level of the analysis result. Alternatively, it may include a string representing the operation of the industrial machine 2 corresponding to the worker's gesture, and a confidence level of the analysis result. In either case, multiple analysis results and the confidence levels of each recognition result may be output together. Furthermore, the analysis results of the video signal may also include feature quantities that indicate characteristics related to mouth movements recognized from the video signal, or feature quantities that indicate gesture characteristics.

The command determination unit 140 determines the operation command for the industrial machine 2 to be performed by the operator based on the recognition result of the audio signal input from the speech recognition unit 110 and the analysis result of the video signal input from the video analysis unit 130. It then outputs the determined command to the command processing unit 150 and the output unit 160. The command determination unit 140 may, for example, compare the confidence level included in the audio signal recognition result with the confidence level included in the video signal analysis result and determine the operation command based on the result with the higher confidence level. In this case, the confidence level included in the audio signal recognition result and the confidence level included in the video signal analysis result may be multiplied by a predetermined weight before comparison. Furthermore, if multiple recognition results and analysis results are input, the system may determine the operation command from among those operation commands where the confidence levels of the recognition result and analysis result indicating the same operation command are above a predetermined threshold, and the operation command with the highest confidence level among the recognition result and analysis result may be selected.

For more precise determination, the command determination unit 140 may use a multilayer neural network model to determine the operation command. In this case, for example, a model is created that has been trained using training data that takes as input a string representing the words spoken in the audio signal, the confidence level of the recognition result, a string representing the operation of the industrial machine 2 corresponding to the operator's gesture, and the confidence level of the analysis result, and outputs an operation command. The command determination unit 140 then inputs the recognition result input from the speech recognition unit 110 and the analysis result input from the video analysis unit 130 into the trained neural network and determines the output operation command as the operation command for the industrial machine 2 by the operator.

Another example of using a multilayer neural network is to create a model that has been trained using training data that takes the features of the audio signal and the features of mouth movements and gestures as inputs and the operation commands as outputs. Then, the command determination unit 140 inputs the recognition results input from the speech recognition unit 110 and the analysis results input from the video analysis unit 130 into the trained neural network and determines the output operation command as the operation command for the industrial machine 2 by the operator.

The command processing unit 150 executes a predetermined function corresponding to an operation command output from the command determination unit 140. The predetermined function may be a general function of the control device. Examples include a function to call a predetermined screen of the machine operation device 1, a function to set predetermined parameters, and a function related to the control of the industrial machine 2. Furthermore, by sending signals to peripheral devices of the industrial machine 2 via the PLC 16, functions such as opening and closing safety doors arranged around the industrial machine 2, turning lights on and off inside the building, stopping the use of cutting fluid, and cleaning chips can also be executed.

The output unit 160 displays the operation command output from the command determination unit 140 to the display device 70. The output unit 160 may display the operation command in a position that does not interfere with the display of a predetermined function being executed on the screen of the display device 70 (for example, in the status display area at the bottom of the screen). Alternatively, it may display the command in the form of a dialog box. Furthermore, it may output the operation command as an audible message from a speaker (not shown) to repeat it. The output unit 160 may not only output the determined operation command, but also output each stage of the processing, such as the start of the operation command recognition process, processing in progress, and processing completed. Additionally, the display device 70 may be used in conjunction with the illumination of LEDs or warning lamps. The output unit 160 may transmit the operation command to a higher-level computer such as other industrial machines 4, fog computers 6, or cloud servers 7 via the network 5. Alternatively, it may output the command to a log recording area pre-provided on the non-volatile memory 14.

The machine operating device 1, equipped with the above configuration, determines the operation command to the industrial machine 2 from the operator based on the acquired audio and video signals. Compared to determining the operation command based solely on the recognition result of the audio signal, the video signal can compensate for the decrease in recognition accuracy caused by noise in the manufacturing site, thus improving the accuracy of the operation command recognition. Furthermore, compared to determining the operation command based solely on the analysis result of the video signal, the audio signal is expected to compensate for the limitations of the expressiveness of the commands.

As one modification of the machine operating device 1 according to this embodiment, the machine operating device 1 may be equipped with a function for authenticating the operator. Figure 3 shows a schematic block diagram of the functions provided by the machine operating device 1 according to this modification. The machine operating device 1 according to this modification further includes an operator authentication unit 170.

The worker authentication unit 170 authenticates a worker based on the voice signal detected by the voice recognition unit 110 and the video signal of the worker's face analyzed by the video analysis unit 130. The RAM 13 to non-volatile memory 14 of the machine operation device 1 is pre-configured with an authentication information storage unit 210 that stores information related to worker authentication, associating identification information that uniquely identifies a worker with feature quantities indicating the characteristics of the worker's voice and feature quantities indicating the characteristics of the worker's face. When a worker speaks, the worker authentication unit 170 analyzes the feature quantities indicating the characteristics of the worker's voice and feature quantities indicating the characteristics of the worker's face from the voice spoken by the worker and the video signal of the worker's face. It then compares this analysis with the voice and face feature quantities of multiple workers stored in the authentication information storage unit 210 to authenticate which worker is speaking. The worker authentication unit 170 outputs the identification information of the authenticated worker to the command processing unit 150. The worker's voice and/or face may be used for authentication, either individually or in combination.

The RAM 13 or non-volatile memory 14 of the machine operating device 1 is further equipped with an authorization information storage unit 220 that stores authorization information associating each operator's executionable operation commands with the operator's identification information. When an operation command determined by the command processing unit 150 is input, the command processing unit 150 retrieves the executionable operation commands associated with the operator's identification information input from the operator authentication unit 170 from the authorization information storage unit 220. If the determined operation command is included among the retrieved executionable operation commands, the operation command is processed as usual. On the other hand, if it is not included, the command is output to the output unit 160 indicating that the operation command is not authorized for the operator.

The machine operating device 1 in this modified version can restrict the execution of operation commands that cannot be executed depending on the operator's job title, authority, etc. Since operator authentication is performed based on the operator's voice and facial recognition, it is expected to prevent the execution of operation commands through impersonation. For example, it can prevent the misuse of recordings of other operators' voices.

Although embodiments of the present invention have been described above, the present invention is not limited to the examples of embodiments described above, and can be implemented in various forms by making appropriate modifications.
For example, in the modified example described above, the authentication information storage unit 210 may also store information related to the language used by the worker, in association with the worker's identification information. Furthermore, the language model used by the speech recognition unit 110 for speech recognition may be switched in advance according to the language used by the worker. By configuring it in this way, it becomes possible to improve the recognition accuracy of operation commands for various workers who speak different languages.

Furthermore, the authentication information storage unit 210 may be configured to automatically authenticate the worker when the worker enters the imaging area of the imaging sensor 74. If the worker is not authenticated, the command determination unit 140 may be configured to stop the process related to determining the operation command. With this configuration, it is possible to prevent the input of operation commands unless the worker is within the operating range of the industrial machine 2, thereby preventing malfunctions of the industrial machine 2 due to the detection of other sounds. Also, if one worker is operating multiple industrial machines 2, the system can be configured to switch which industrial machine 2 the worker is operating on depending on which industrial machine 2 the worker is within the operating range of.

Furthermore, in addition to the worker's mouth movements and gestures, the system may also detect the worker's work actions and standing position, and use this information to recognize operation commands. In this case, the video analysis unit 130 analyzes what kind of work actions the worker is performing (for example, whether they are carrying a workpiece, putting their arm inside the machine, etc.) and where the worker is standing near the industrial machine 2. The analysis results are then output to the command determination unit 140. The command determination unit 140 determines the operation command by narrowing down the operation commands the worker is trying to perform based on the work actions the worker is performing and their standing position. For example, it is dangerous for a worker to input an operation command to close the safety door or an operation command to restart operation while their hand is inside the machine's safety door. In such a situation, even if the voice recognition unit 110 recognizes such an audio signal, the command determination unit 140 can choose not to process dangerous operation commands. Also, in manufacturing sites such as factories, multiple workers may be working together. In such cases, there is a possibility that conversations between workers may be mistakenly detected as voice commands. Therefore, after authenticating the worker, if the system analyzes the worker's actions to indicate a conversation with another worker, it can prevent the input of operation commands. This is expected to reduce errors in operating industrial machinery.

Furthermore, in addition to the above, it is also possible to acquire the operating status of the industrial machine 2 and narrow down the operation command that the operator is trying to perform based on that operating status. For example, it is not usually possible for an operation command to open the safety door to be entered while the workpiece is being processed. Therefore, if an operation command to open the safety door is recognized or analyzed while the workpiece is being processed, the command determination unit 140 may exclude such an operation command from the determination target.

When multiple industrial machines installed nearby are being used for parallel operation, the industrial machine 2 to be operated may be determined by the worker's standing position or gestures. In this configuration, as shown in Figure 4, the machine operation device 1 is implemented on a computer connected via a network to the control devices that control each industrial machine 2. The voice spoken by the worker can be collected, for example, using a voice sensor 73 connected to the machine operation device 1 via a wired/wireless network 5. In addition, the worker's video can be collected using an imaging sensor 74 installed on the ceiling of the manufacturing site or near the worker and connected to the machine operation device 1 via a wired/wireless network 5, or an imaging sensor 474 attached to the industrial machine 4 that the worker is managing.

Furthermore, as illustrated in Figure 5, when performing work using multiple industrial machines 4, the system can be configured to identify the target industrial machine 4 from among the multiple industrial machines 4 based on the worker's standing position, orientation (body direction, gaze, etc.), and gestures (which industrial machine 4 they are pointing to, etc.) captured by the image sensors 74 and 474, and to recognize and analyze the operation commands for that industrial machine 4. At this time, terms that identify the industrial machine 4 (operation commands executed only on a specific industrial machine 4, the name of the industrial machine 4, etc.) from the worker's spoken voice may also be detected and used.

1. Machine operating device 2. Industrial machinery 4. Industrial machinery 5. Network 6. Fog computer 7. Cloud server 11. CPU
12 ROM
13 RAM
14 Non-volatile memory 15, 17, 18, 20, 21 Interface 16 PLC
19 I/O unit 22 Bus 30 Axis control circuit 40 Servo amplifier 50 Servo motor 70 Display device 71 Input device 72 External device 73 Voice sensor 74 Image sensor 100 Voice signal acquisition unit 110 Voice recognition unit 120 Video signal acquisition unit 130 Video analysis unit 140 Command determination unit 150 Command processing unit 160 Output unit 170 Worker authentication unit 210 Authentication information storage unit 220 Authorization information storage unit

Claims (6)

A machine operating device that uses voice recognition to process voice signals input in a manufacturing site and utilizes them as operating commands for operating industrial machinery,
A voice recognition unit recognizes first data indicating a predetermined operation based on an audio signal obtained by converting the voice spoken by the worker detected by a voice sensor.
A video analysis unit analyzes second data indicating a predetermined operation based on video signals obtained by capturing the worker's movements using an imaging sensor.
A command determination unit that determines an operation command for operating the industrial machine based on the first data and the second data,
A command processing unit that processes the operation command determined by the command determination unit,
Equipped with ,
If the system recognizes that an operation command is dangerous to execute depending on at least one of the worker's work movements and/or standing position, it will not process that operation command.
Machine operating equipment. A machine operating device that uses voice recognition to process voice signals input in a manufacturing site and utilizes them as operating commands for operating industrial machinery,
A voice recognition unit recognizes first data indicating a predetermined operation based on an audio signal obtained by converting the voice spoken by the worker detected by a voice sensor.
A video analysis unit analyzes second data indicating a predetermined operation based on video signals obtained by capturing the worker's movements using an imaging sensor.
A command determination unit that determines an operation command for operating the industrial machine based on the first data and the second data,
A command processing unit that processes the operation command determined by the command determination unit,
Equipped with ,
The system narrows down the available operating commands based on the operating status of the industrial machine being controlled by the operator.
Machine operating equipment. The first data further includes a worker authentication unit that authenticates the worker based on the voice spoken by the worker and the worker's face,
The command processing unit prohibits the execution of operation commands that are not authorized by the operator authenticated by the operator authentication unit.
The machine operating device according to claim 1 or 2. The speech recognition unit switches the language model it uses depending on the language used by the authenticated worker.
The machine operating device according to claim 3. The command determination unit stops determining the operation command if the video analysis unit does not detect an operator.
The machine operating device according to claim 1 or 2. Based on at least one of the recognition results from the voice recognition unit and the analysis results from the video analysis unit, the industrial machine to be operated is identified from among multiple industrial machines.
The machine operating device according to claim 1 or 2.