JP7674114B2 - Information processing device, learning device, information processing system, robot system, and evaluation method - Google Patents

Description

This disclosure relates to an information processing device, a learning device, an information processing system, and a robot system.

There are technologies for evaluating robot operators. For example, the management system disclosed in Patent Document 1 provides an environment in which the operator can operate a robot located near a remote object while making the operator feel as if the remote object is nearby. A server provided in the management system acquires status information indicating the state of the robot or the state of the operator when the operator operates the robot, and determines the operator's suitability for operating the robot by comparing the state indicated by the status information with a reference level.

JP 2020-135362 A

In Patent Document 1, the skill level of robot operation is determined as the degree of suitability. For example, the operator can only know the overall skill level of operation for a task through the determined level. The operator cannot find a specific measure to improve the robot operation skill, and may not be able to increase motivation to improve skills and willingness to operate.

The present disclosure aims to provide an information processing device, a learning device, an information processing system, and a robot system that increase a user's motivation to operate a robot.

An information processing device according to one aspect of the present disclosure is an information processing device that evaluates the operation of a robot to cause the robot to perform a task, and includes a detection processing unit that performs, during the execution of the task, a detection process that detects action-related information, which is information related to the action of the robot that operates according to an operation command input by a user to an operation device; an evaluation processing unit that performs, during the execution of the task, an evaluation process that compares and evaluates the action-related information detected by the detection processing unit with an action-related criterion, which is a criterion related to the action of the robot; and a presentation processing unit that performs, during the execution of the task, a presentation device that presents the evaluation result of the evaluation processing unit to the user while the user is operating the operation device and receives presentation of information. The evaluation processing unit executes the evaluation process at least at one of the timings of each evaluation target action, which is the action of the robot to be evaluated, and each evaluation target stage, which is the task stage of the task to be evaluated, and the presentation processing unit, when the evaluation process is executed at the timing of each evaluation target action, causes the presentation device to present the evaluation results for the evaluation target action in the same order as the execution order of the evaluation target action, and when the evaluation process is executed at the timing of each evaluation target stage, causes the presentation processing unit to present the evaluation results for the evaluation target stage in the same order as the execution order of the evaluation target stage.

The technology disclosed herein can increase users' motivation to operate robots.

FIG. 1 is a diagram showing an example of a configuration of a robot system according to an embodiment; FIG. 1 is a perspective view showing an example of a configuration of a robot according to an embodiment; FIG. 1 is a block diagram showing an example of a hardware configuration of a control device according to an embodiment. FIG. 1 is a block diagram showing an example of a functional configuration of a control device according to an embodiment; 1 is a flowchart showing an example of an operation of a robot system according to an embodiment. FIG. 1 is a plan view showing an example of an operation of the robot system according to the embodiment; FIG. 11 is a block diagram showing an example of a functional configuration of an information processing device and a learning device according to a modified example.

(Embodiment)
Hereinafter, the embodiments of the present disclosure will be described with reference to the drawings. Note that the embodiments described below are all comprehensive or specific examples. In addition, among the components in the following embodiments, components that are not described in the independent claims showing the highest concept are described as optional components. In addition, each figure in the attached drawings is a schematic diagram and is not necessarily illustrated precisely. Furthermore, in each figure, substantially the same components are given the same reference numerals, and duplicated descriptions may be omitted or simplified. In addition, in this specification and claims, "apparatus" may mean not only one apparatus but also a system consisting of multiple apparatuses.

[Robot system configuration]
A configuration of a robot system 1 according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of a configuration of the robot system 1 according to an embodiment. The robot system 1 includes at least one robot 100, at least one operation terminal 200, and a server 300. Although not limited thereto, in this embodiment, the robot system 1 is configured to provide a service to a user P using the robot 100 operated from a remote location. The robot system 1 can be used in various service industries such as nursing care, medical care, cleaning, security, guidance, rescue, cooking, sales, rental, and provision of goods.

The robot 100 is a robot that can perform tasks such as providing services. In this specification and claims, "task" may refer to a series of actions performed by the robot 100 to accomplish, build, and/or create something productive that is expected to produce some useful result. For example, "task" may not refer to a series of actions performed by the robot 100 in the course of teaching the robot 100. For example, "task" may not refer to a series of actions performed by the robot 100 for play, games, and/or entertainment.

In this embodiment, although not limited to this, multiple robots 100 are placed in one service provision area AS, which is a place where services are provided to user P. Furthermore, at least one operation terminal 200 is placed in each of multiple operation areas AO located away from the service provision area AS.

Each robot 100 is configured to be connectable to the communication network N via wireless communication so as to enable data communication. The communication connecting the robot 100 and the communication network N may be wired communication, or a combination of wired communication and wireless communication, etc. Each operation terminal 200 is configured to be connectable to the communication network N so as to enable data communication via wired communication, wireless communication, or a combination of these. One robot 100 and one operation terminal 200 may be connected to each other via the communication network N so as to enable data communication. Any wired communication or wireless communication may be used.

The server 300 manages communication via the communication network N. The server 300 includes a computer device. The server 300 manages authentication, connection, and disconnection of communication between the robot 100 and the operation terminal 200. For example, the server 300 stores identification information and security information of the robot 100 and the operation terminal 200 registered in the robot system 1, and uses the information to authenticate the qualification of each operation terminal 200 to connect to the robot system 1. The server 300 manages the transmission and reception of data between the robot 100 and the operation terminal 200, and the data may pass through the server 300. The server 300 may be configured to convert data transmitted from a transmission source into a data format that can be used by the transmission destination. The server 300 may also be configured to store and accumulate information, commands, data, and the like transmitted and received between the robot 100 and the operation terminal 200 during the operation of the robot 100.

The communication network N is not particularly limited, and may include, for example, a local area network (LAN), a wide area network (WAN), the Internet, or a combination of two or more of these. The communication network N may be configured to use short-range wireless communication such as Bluetooth (registered trademark) and ZigBee (registered trademark), a network dedicated line, a dedicated line of a communication carrier, a public switched telephone network (PSTN), a mobile communication network, the Internet network, satellite communication, or a combination of two or more of these. The mobile communication network may use a fourth generation mobile communication system and a fifth generation mobile communication system, etc. The communication network N may include one or more networks. In this embodiment, the communication network N is the Internet.

[Configuration of operation terminal]
The configuration of an operation terminal 200 according to an embodiment will be described with reference to Fig. 1. The operation terminal 200 is configured to be able to accept input of commands, information, data, etc. by an operator PO and to be able to output the accepted commands, information, data, etc. to another device. The operation terminal 200 includes an operation input device 201, a terminal computer 202, a presentation device 203, and a communication device 204. The operation input device 201, the terminal computer 202, the presentation device 203, and the communication device 204 may have an integrated configuration to form one device, each may be configured to form a device independently and be connected to each other, or at least two of them may be configured to form one device and be connected to another device.

The operation terminal 200 is an example of an operation device. The configuration of the operation terminal 200 is not particularly limited, and for example, the operation terminal 200 may be a computer such as a personal computer, a smart device such as a smartphone or a tablet, a personal information terminal, a game terminal, a known teaching device used for teaching a robot, a known operation device for a robot, other operation devices, other terminal devices, devices that use these, and devices that have been improved upon. The operation terminal 200 may be a dedicated device designed for the robot system 1, or a general-purpose device available on the general market. In this embodiment, a known general-purpose device is used for the operation terminal 200. The device may be configured to realize the functions of the operation terminal 200 of the present disclosure by installing dedicated software.

The operation input device 201 is configured to receive input from the operator PO and to output signals indicating the input commands, information, data, etc. to the terminal computer 202. The configuration of the operation input device 201 is not particularly limited, and for example, the operation input device 201 may include a device to which input is given through the operation of the operator PO, such as a button, a lever, a dial, a joystick, a mouse, a key, a touch panel, and a motion capture. The operation input device 201 may include an imaging device that captures an image of the operator PO, etc., and an audio input device such as a microphone that receives audio input from the operator PO, etc. The operation input device 201 may be configured to output the captured image data and a signal indicating the input audio to the terminal computer 202.

The terminal computer 202 is configured to process commands, information, data, etc. received via the operation input device 201 and output them to other devices, and to receive input of commands, information, data, etc. from other devices and process those commands, information, data, etc.

The presentation device 203 is configured to be able to present information to the operator PO. In this embodiment, the presentation device 203 includes a display capable of displaying an image to the operator PO, although this is not limited thereto. The presentation device 203 displays an image of the image data received from the terminal computer 202. The presentation device 203 may include an audio output device such as a speaker capable of emitting audio to the operator PO. The presentation device 203 outputs the audio of the audio data received from the terminal computer 202.

The communication device 204 has a communication interface that can be connected to the communication network N. The communication device 204 is connected to the terminal computer 202, and connects the terminal computer 202 and the communication network N so that data communication is possible. The communication device 204 may include communication devices such as a modem, an ONU (Optical Network Unit), a router, and a mobile data communication device. The communication device 204 may include a computer device having a calculation function, etc.

[Robot configuration]
The configuration of the robot 100 according to the embodiment will be described with reference to FIG. 2. FIG. 2 is a perspective view showing an example of the configuration of the robot 100 according to the embodiment. Each robot 100 includes one transport vehicle 110, at least one robot arm 120, at least one end effector 130, a secondary battery module 141, a power supply circuit 142, a communication device 143, imaging devices 144, 145, and 146, a sound collection device 147, a display device 148, a sound output device 149, and a control device 170. The control device 170 includes an information processing device 180 and a robot controller 190. The information processing device 180 and the robot controller 190 may be configured as separate devices or may be configured as an integrated device. Although not limited thereto, in this embodiment, a robot arm that can also function for industrial use is used as the robot arm 120. The quantity of each of the above components is not limited to the above-mentioned quantity and can be changed as appropriate.

The transport vehicle 110 is configured to be self-propelled. In this embodiment, the transport vehicle 110 includes, but is not limited to, two drive wheels 111, four auxiliary wheels 112, and a transport drive device 113 that drives the two drive wheels 111. The transport drive device 113 includes a servo motor (not shown) as an electric actuator that is a drive source. The servo motor is controlled by the robot controller 190. The transport drive device 113 can rotate the two drive wheels 111 in various rotation directions and rotation speeds, thereby causing the transport vehicle 110 to travel at various speeds in the forward direction D1A, the backward direction D1B, and the turning direction.

In this embodiment, although not limited thereto, two robot arms 120A and 120B are mounted on the transport vehicle 110 via the base 120C as the robot arm 120. Both robot arms 120A and 120B are configured to be rotatable about an axis S1 in a direction from the transport vehicle 110 toward the base 120C, and constitute a coaxial double-arm robot arm. The robot arm 120A includes links 121A to 124A and arm drive devices M1A to M4A (see FIG. 3) that drive the joints that connect the links 121A to 124A to each other. The robot arm 120B includes links 121B to 124B and arm drive devices M1B to M4B (see FIG. 3) that drive the joints that connect the links 121B to 124B to each other. The arm drive devices M1A to M4A and M1B to M4B include servo motors as electric actuators that are drive sources. Each servo motor is controlled by a robot controller 190. Links 121A and 121B are connected to base 120C via joints (not shown). Links 124A and 124B include a mechanical interface that can be connected to end effector 130.

The robot arms 120A and 120B as described above have the configuration of a horizontal multi-joint arm, but may have any configuration. For example, the robot arms 120A and 120B may be vertical multi-joint type, polar coordinate type, cylindrical coordinate type, Cartesian coordinate type, or other types of robot arms. The number of robot arms 120 arranged on the transport vehicle 110 may be one or more.

As the end effector 130, two end effectors 130A and 130B are detachably attached to the links 124A and 124B, respectively. The end effectors 130A and 130B are configured to be able to act on an object handled by the robot 100. The end effectors 130A and 130B are configured to be operable and may be equipped with a drive unit (not shown). The drive unit may be powered by electricity, air pressure, hydraulic pressure, or the like. The drive unit powered by electricity may be equipped with a servo motor as an electric actuator. The drive unit may be controlled by the robot controller 190.

The robot 100 further includes an equipment housing 140 on the transport vehicle 110. The secondary battery module 141, the power supply circuit 142, the communication device 143, the information processing device 180, and the robot controller 190 are disposed within the equipment housing 140, but may be disposed at any position on the transport vehicle 110. A table 140a on which an item can be placed is disposed on the equipment housing 140.

The secondary battery module 141 functions as a power source for the robot 100. The secondary battery module 141 includes at least one secondary battery. A secondary battery is a battery that can charge and discharge power. Examples of secondary batteries include lead-acid batteries, lithium-ion secondary batteries, solid-state batteries, nickel-metal hydride batteries, and nickel-cadmium batteries.

The power supply circuit 142 is a circuit that controls the supply and demand of power to the secondary battery module 141. The power supply circuit 142 is configured to control the supply and demand of power in accordance with commands from the information processing device 180 and the robot controller 190. For example, the power supply circuit 142 may include devices such as a converter, inverter, transformer, and amplifier. The power supply circuit 142 is connected to an external power source such as a commercial power source, and is configured to supply and store power supplied from the external power source to the secondary battery module 141, and to supply the power stored in the secondary battery module 141 to components within the robot 100 that consume power.

The communication device 143 is a device for wireless communication, and is configured to be able to connect to the communication network N via wireless communication. The wireless communication used by the communication device 143 is not particularly limited, but may be, for example, wireless communication using mobile data communication, wireless LAN such as wireless Wi-Fi (Wireless Fidelity), short-range wireless communication such as Bluetooth (registered trademark) and ZigBee (registered trademark), or a combination of two or more of these. The communication device 143 has equipment compatible with the wireless communication used.

The display device 148 includes a display capable of displaying an image. The display may be configured to allow input by the user P, and may be, for example, a touch panel. The display device 148 can display an image of image data sent from the information processing device 180 or the like. The display device 148 can display an image to the user P facing the robot 100.

The sound collection device 147 includes a microphone that can acquire sound from the surroundings and output an audio signal of the sound. The sound collection device 147 is configured to output the audio signal to the information processing device 180 or the like. The information processing device 180 may be configured to convert the audio signal into audio data and transmit it to the operation terminal 200, or may be configured to perform voice recognition from the audio signal. The sound collection device 147 can constitute an external sensor of the robot 100 that can sense the outside of the robot 100. The internal sensor of the robot 100 can sense the inside of the robot 100, and an example of an internal sensor is a rotation sensor provided on the servo motors of the arm driving devices M1A to M4A and M1B to M4B, etc.

The audio output device 149 includes a speaker that can convert an audio signal into sound waves and emit them as sound. The audio output device 149 can output sound corresponding to an audio signal sent from the information processing device 180 or the like. The audio output device 149 can output sound to the user P facing the robot 100.

Each of the imaging devices 144, 145, and 146 includes a camera that captures a digital image, and is configured to send the captured image data to the information processing device 180. The information processing device 180 may be configured to process the image data captured by the imaging devices 144, 145, and 146 into data that can be transmitted over a network, and to send the data to the operation terminal 200 via the communication network N. The information processing device 180 may be configured to perform image processing on the image data and use the image data for other processing. The camera may be a camera that can capture an image for detecting the three-dimensional position of the subject relative to the camera, such as the distance to the subject. For example, the camera may have a configuration such as a stereo camera, a monocular camera, a TOF camera (Time-of-Flight-Camera), a pattern light projection camera such as a stripe projection camera, or a camera using a light cutting method. The imaging devices 144, 145, and 146 can constitute an external sensor.

The imaging device 144 is disposed at the tip of at least one of the robot arms 120A and 120B. In this embodiment, the imaging device 144 is disposed at the end effector 130A of the robot arm 120A, although this is not limited thereto. The imaging device 145 is disposed at the display device 148. The imaging device 145 can capture an image of the user P to whom the service is to be provided, who faces the robot 100. The imaging device 145 may be configured to be able to change the imaging direction and/or have a wide-angle imaging field of view so that it can capture images of the object on which the end effectors 130A and 130B act, as well as the end effectors 130A and 130B. The imaging device 146 is disposed on the transport vehicle 110 facing the forward direction D1A.

The robot controller 190 is configured to control the operation of the robot arms 120A and 120B, the end effectors 130A and 130B, and the transport vehicle 110. For example, the robot controller 190 is configured to control the operation of the arm drive devices M1A-M4A and M1B-M4B of the robot arms 120A and 120B, the drive devices (not shown) of the end effectors 130A and 130B, and the transport drive device 113 of the transport vehicle 110. The information processing device 180 is connected to the robot controller 190 via wired communication, wireless communication, or a combination thereof, and is configured to perform arithmetic processing related to the control of the robot arms 120A and 120B, the end effectors 130A and 130B, and the transport vehicle 110.

The information processing device 180 and the robot controller 190 include computer devices. The configuration of the information processing device 180 is not particularly limited, but for example, the information processing device 180 may be an electronic circuit board, an electronic control unit, a microcomputer, a personal computer, a workstation, a smart device such as a smartphone and a tablet, and other electronic devices. The information processing device 180 is connected to the terminal computer 202 of the operation terminal 200 so as to be able to perform data communication via the communication device 143, the communication network N, and the communication device 204. The configuration of the robot controller 190 is not particularly limited, but for example, the robot controller 190 may be an electronic circuit board, an electronic control unit, a microcomputer, and other electronic devices. The robot controller 190 may include a drive circuit for controlling the power supplied to the arm drive devices M1A to M4A and M1B to M4B, the drive devices (not shown) of the end effectors 130A and 130B, and the transport drive device 113.

In this embodiment, when the robot 100 is caused to perform a task, the information processing device 180 and the robot controller 190 are configured to control the operation of the control target elements such as the robot arms 120A and 120B, the end effectors 130A and 130B, and the transport vehicle 110 by manual driving control, but are not limited to this. The information processing device 180 and the robot controller 190 may be configured to control the operation of at least one of the above control target elements by automatic driving control or a combination of automatic driving control and manual driving control. For example, the combination of automatic driving control and manual driving control may be configured so that automatic driving control and manual driving control are assigned to each control target element, and part of the task of the robot 100 may be performed by automatic driving control and other parts of the task may be performed by manual driving control.

For example, manual driving control may be control in which the controlled element is operated in accordance with the operation content input by the operator PO to the operation input device 201 of the operation terminal 200. For example, in manual driving control, the controlled element may perform an operation in accordance with the operation of the operator PO who operates the operation input device 201. Automatic driving control may be control in which the controlled element is operated autonomously in accordance with a control program.

[Hardware configuration of the control device]
With reference to FIG. 3, the hardware configuration of the control device 170 according to the embodiment will be described. FIG. 3 is a block diagram showing an example of the hardware configuration of the control device 170 according to the embodiment. The information processing device 180 includes a processor 1801, a memory 1802, a storage 1803, and input/output I/Fs (interfaces) 1804a to 1804f as components. The components of the information processing device 180 are connected to each other by a bus 1805, but may be connected by any other wired or wireless communication. The robot controller 190 includes a processor 1901, a memory 1902, an input/output I/F 1903, and a drive I/F 1904 as components. The robot controller 190 may include a storage. The components of the robot controller 190 are connected to each other by a bus 1905, but may be connected by any other wired or wireless communication. Not all of the components included in the information processing device 180 and the robot controller 190 are essential.

The pair of processor 1801 and memory 1802, and the pair of processor 1901 and memory 1902 each constitute a computing unit. The computing unit transmits and receives commands, information, data, etc. to and from other devices. The computing unit inputs signals from various devices and outputs control signals to each controlled element.

Memories 1802 and 1902 store programs executed by processors 1801 and 1901, and various fixed data, etc. Memories 1802 and 1902 may be configured with storage devices such as semiconductor memories, including volatile and non-volatile memories. Although not limited thereto, in this embodiment, memories 1802 and 1902 include RAM (Random Access Memory), which is a volatile memory, and ROM (Read-Only Memory), which is a non-volatile memory.

Storage 1803 stores various data. Storage 1803 may be configured with a storage device such as a semiconductor memory, a hard disk drive (HDD: Hard Disk Drive), or a solid state drive (SSD: Solid State Drive).

Both the processors 1801 and 1901 form a computer system together with the RAM and ROM. The computer system of the information processing device 180 may realize the functions of the information processing device 180 by the processor 1801 using the RAM as a work area to execute a program recorded in the ROM. The computer system of the robot controller 190 may realize the functions of the robot controller 190 by the processor 1901 using the RAM as a work area to execute a program recorded in the ROM.

Some or all of the functions of the information processing device 180 and the robot controller 190 may be realized by the above-mentioned computer system, by a dedicated hardware circuit such as an electronic circuit or integrated circuit, or by a combination of the above-mentioned computer system and hardware circuit. The information processing device 180 and the robot controller 190 may each be configured to execute each process by centralized control by a single device, or may be configured to execute each process by distributed control through the cooperation of multiple devices. The information processing device 180 and the robot controller 190 may be configured to include at least some of each other's functions, or may be integrated.

For example, but not limited to, processors 1801 and 1901 may include a CPU (Central Processing Unit), MPU (Micro Processing Unit), GPU (Graphics Processing Unit), microprocessor, processor core, multiprocessor, ASIC (Application-Specific Integrated Circuit), FPGA (Field Programmable Gate Array), etc., and each process may be realized by a logic circuit or a dedicated circuit formed in an IC (Integrated Circuit) chip, LSI (Large Scale Integration), etc. Multiple processes may be realized by one or multiple integrated circuits, or may be realized by a single integrated circuit.

The first input/output I/F 1804a of the information processing device 180 connects the information processing device 180 and the robot controller 190, enabling input/output of information, commands, data, etc. between them. The second input/output I/F 1804b connects the information processing device 180 and the communication device 143, enabling input/output of information, commands, data, etc. between them. The third input/output I/F 1804c connects the information processing device 180 and the imaging devices 144 to 146, enabling input/output of information, commands, data, etc. between them. The fourth input/output I/F 1804d connects the information processing device 180 and the sound collection device 147, enabling input/output of information, commands, data, etc. between them. The fifth input/output I/F 1804e connects the information processing device 180 and the display device 148, enabling input/output of information, commands, data, etc. between them. The sixth input/output I/F 1804f connects the information processing device 180 and the audio output device 149, enabling the input and output of information, commands, data, etc. between them.

The input/output I/F 1903 of the robot controller 190 connects the robot controller 190 and the first input/output I/F 1804a of the information processing device 180, enabling input and output of information, commands, data, and the like between them. The drive I/F 1904 connects the robot controller 190 and the drive circuit 142a, enabling transmission and reception of signals, and the like between them. The drive circuit 142a is configured to control the power supplied to the arm drive devices M1A to M4A and M1B to M4B, the drive devices (not shown) of the end effectors 130A and 130B, and the conveyance drive device 113 according to a command value included in a signal received from the robot controller 190. For example, the drive circuit 142a can drive each drive device in cooperation with each other. Although not limited to this, in this embodiment, the drive circuit 142a is formed as a part of the power supply circuit 142.

The robot controller 190 may be configured to servo-control the servo motors of each drive device. The robot controller 190 receives the detection value of the rotation sensor equipped in each servo motor and the command value of the current from the drive circuit 142a to the servo motor as feedback information from the drive circuit 142a. The robot controller 190 uses the feedback information to determine the command value for driving the servo motor and transmits it to the drive circuit 142a.

In addition, since a general-purpose device can be used for the operation terminal 200, a detailed description of the hardware configuration of the operation terminal 200 will be omitted. The terminal computer 202 of the operation terminal 200 includes a processor and memory (both not shown) similar to the information processing device 180, etc. The terminal computer 202 may also include input/output I/Fs (not shown) for establishing a connection between the terminal computer 202 and the operation input device 201, a connection between the terminal computer 202 and the communication device 204, and a connection between the terminal computer 202 and the presentation device 203.

[Functional configuration of the control device]
With reference to FIG. 4, the functional configuration of the control device 170 according to the embodiment will be described. FIG. 4 is a block diagram showing an example of the functional configuration of the control device 170 according to the embodiment. The information processing device 180 includes an operation command unit 180a, a detection processing unit 180b, an evaluation processing unit 180c, a consideration determination unit 180d, a presentation processing unit 180e, an accumulation processing unit 180f, a criterion determination unit 180g, a timekeeping unit 180h, and memory units 181a and 181b as functional components. The functions of the functional components other than the memory units 181a and 181b are realized by a processor 1801 or the like, and the functions of the memory units 181a and 181b are realized by a storage device such as a memory 1802 and/or a storage 1803. The functions of the storage units 181a and 181b may be realized by separate storage devices or may be realized by one storage device. All of the above functional components are not essential. The robot controller 190 includes drive command units 190a to 190e as functional components. The functions of the drive command units 190a to 190e are realized by a processor 1901, etc. Not all of the above functional components are essential.

The first memory unit 181a stores motion-related reference data (represented as "reference data" in FIG. 4) for various tasks. The motion-related reference data includes information on motion-related references, which are references related to the motion of the robot 100. The motion-related references include references for motion-related information, which is information related to the motion of the robot 100. The motion-related information may include information on motion-related elements such as the motion of the robot 100, the state of the environment surrounding the robot 100, the state of an object on which the robot 100 acts, and the state of the environment surrounding the object.

For example, the action-related criteria may be criteria that represent a minimum required level of the action-related element, a standard level of the action-related element, a good level of the action-related element, an ideal level of the action-related element, etc. The action-related criteria may include criteria for an evaluation target feature, which is a feature for evaluating the action-related element.

For example, one task is composed of a series of multiple motions of the robot 100. The motion-related criteria for the motions of the robot 100 may include criteria for the series of multiple motions that form the task of each of the robot arms 120A and 120B, the end effectors 130A and 130B, and the transport vehicle 110. For example, the evaluation features of the motion-related criteria for the motions of the robot 100 may include features such as the time required for the motion, the order of the motion, the position of the robot 100 and each part thereof in the motion, the posture of the robot 100 and each part thereof in the motion, the trajectory of the robot 100 and each part thereof in the motion, the speed of the above position in the motion, the speed of the above posture in the motion, the acceleration of the above position in the motion, the acceleration of the above posture in the motion, and the work performance of the robot 100 in the motion.

The state of the surrounding environment of the robot 100 may include the state of the robot peripheral elements, which are elements such as devices, facilities, and equipment that form the surrounding environment for the robot 100 to perform work. Examples of the robot peripheral elements are devices, facilities, and equipment arranged in the work area (service provision area AS, etc.) of the robot 100, devices, facilities, and equipment that cooperate with the robot 100, and sensors such as imaging devices arranged in the work area (service provision area AS, etc.). The evaluation target features of the operation-related criteria of the robot peripheral elements may include the force and impact that the robot 100 applies to the robot peripheral elements, the changes in position, posture, shape, and state that the robot 100 applies to the robot peripheral elements, and the position, posture, position speed, posture speed, position acceleration, and posture acceleration of the robot 100 and each part thereof relative to the robot peripheral elements. For example, in FIG. 1, the robot peripheral elements of the robot 100 are other robots 100, a storage shelf SR in which the target item W is stored, and users other than the user P to whom the robot 100 provides a service.

The state of the object may include the state of the object handled by the robot 100 using the end effectors 130A and 103B, etc. The evaluation target features of the object's motion-related criteria may include features such as the object's position, posture, position velocity, posture velocity, position acceleration, and posture acceleration, changes in the object's shape, changes in the object's state, and the workmanship of the object. For example, in FIG. 1, the object is an item W.

The state of the surrounding environment of the object may include the state of the object peripheral elements, which are elements of devices, facilities, and equipment that handle objects other than the robot 100. Examples of the object peripheral elements are devices, facilities, and equipment that support, hold, store, load, mount, and transport the object, devices, facilities, and equipment to which the object is attached, and devices, facilities, and equipment that process the object. The evaluation target features of the operation-related criteria of the object peripheral elements may include the force and impact that the robot 100 applies to the object peripheral elements, the changes in position, posture, shape, and state that the robot 100 applies to the object peripheral elements, and the position, posture, position speed, posture speed, position acceleration, and posture acceleration of the robot 100 and each part thereof relative to the object peripheral elements. For example, in FIG. 1, the object peripheral elements are the shelves (not shown) of the storage shelf SR, other items in the storage shelf SR, and the user P.

The second memory unit 181b stores accumulated data, which is data accumulated by the accumulation processing unit 180f. The accumulated data includes information on the evaluation result determined by the evaluation processing unit 180c and information on the action-related criteria used to determine the evaluation result, in association with each other. In other words, the accumulated data associates action-related information, information on the action-related criteria for the action-related information, and information on the evaluation result based on the action-related criteria. The second memory unit 181b is an example of an accumulation unit.

During manual operation control, the operation command unit 180a generates operation commands for operating the robot arms 120A and 120B, the end effectors 130A and 130B, and the transport vehicle 110 in accordance with operation commands received from the operation terminal 200, and outputs the operation commands to the robot controller 190. During automatic operation control, the operation command unit 180a generates operation commands for operating the robot arms 120A and 120B, the end effectors 130A and 130B, and the transport vehicle 110 in accordance with a control program, and outputs the operation commands to the robot controller 190.

The operation command is a command indicating the content of the operation input to the operation input device 201 by the operator PO. The operation command includes a command for causing the robot 100 to operate under manual driving control in accordance with the content of the operation. The terminal computer 202 receives a signal indicating the content of the operation from the operation input device 201, converts the signal into an operation command, and transmits the operation command to the information processing device 180.

The operation command unit 180a generates a first operation command for the robot arm 120A, a second operation command for the robot arm 120B, a third operation command for the end effector 130A, a fourth operation command for the end effector 130B, and a fifth operation command for the transport vehicle 110. For example, the third operation command and the fourth operation command can be generated when the end effectors 130A and 130B are equipped with driving devices. The first operation command and the second operation command can include commands for the target positions of each part of the robot arms 120A and 120B and commands for the target forces generated by each part. The third operation command and the fourth operation command can include commands for the target states of the end effectors 130A and 130B. The fifth operation command can include a command for the target position of the transport vehicle 110.

The target position command may include commands for the target position, the target moving speed of the position, the target attitude, and the target moving speed of the attitude. The target force command may include commands for the magnitude and direction of the target force. The target force command may include a command for the target acceleration.

The timing unit 180h measures the elapsed time and outputs the timing result to the detection processing unit 180b, the evaluation processing unit 180c, the compensation determination unit 180d, the presentation processing unit 180e, etc. The timing result can be used for processing in the detection processing unit 180b, the evaluation processing unit 180c, the compensation determination unit 180d, the presentation processing unit 180e, etc. For example, the function of the timing unit 180h may be realized by a clock or the like installed in the computer device of the information processing device 180.

The detection processing unit 180b executes a detection process to detect the motion-related information of the robot 100, which operates according to an operation command, while the robot 100 is performing a task. The detection processing unit 180b outputs the detection result to the evaluation processing unit 180c. In this embodiment, the detection processing unit 180b is configured to output the detection result to the evaluation processing unit 180c without a delay time after detecting the motion-related information, but is not limited to this and a delay time may be provided.

The detection processing unit 180b detects information on motion-related elements as motion-related information while a task is being performed. The detection processing unit 180b may be configured to detect information on motion-related elements using any method.

For example, the detection processing unit 180b detects the motion of the robot 100 as the motion-related element, specifically, the motion of the robot arms 120A and 120B, the end effectors 130A and 130B, and the transport vehicle 110. For example, the detection processing unit 180b may be configured to process an operation command and detect the target motion of the robot 100 indicated by the operation command as the motion-related element, or to process a motion command and detect the target motion of the robot 100 indicated by the motion command as the motion-related element. The detection processing unit 180b may be configured to acquire and process a drive command from the drive command units 190a to 190e of the robot controller 190, and detect the target motion of the robot 100 indicated by the drive command as the motion-related element. The detection processing unit 180b may be configured to acquire and process feedback information from the drive command units 190a to 190e of the robot controller 190, and detect the motion of the robot 100 indicated by the feedback information as the motion-related element. The detection processing unit 180b may be configured to acquire captured image data from the imaging devices 144, 145, and 146 and from an imaging device (not shown) external to the robot 100, process the images, and detect the state of the robot 100 indicated by the processed image data as a movement-related element.

For example, the detection processing unit 180b detects the state of the robot peripheral elements constituting the surrounding environment of the robot 100 as the operation-related elements. For example, the detection processing unit 180b detects the state of the robot peripheral elements using the detection results received from the external sensors of the robot 100 and the information received from the robot peripheral elements. Examples of the external sensors include force sensors (not shown) arranged between the end effectors 130A and 103B and the robot arms 120A and 120B, the imaging devices 144 to 146, the sound collection device 147, and sensors (not shown) such as an imaging device arranged in the service provision area AS. Examples of information received from the robot peripheral elements include the state, vibration, impact, position, attitude, speed of position, and speed of attitude of the robot peripheral elements.

For example, the detection processing unit 180b detects the state of an action target, which is an object on which the robot 100 applies an action, as a motion-related element. For example, the detection processing unit 180b detects the state of the action target using detection results received from the external sensors of the robot 100 and information received from the action target. Examples of the external sensors are the force sensors of the robot arms 120A and 120B, the image capture devices 144-146, the sound collection device 147, and sensors placed in the service provision area AS. Examples of information received from the action target are the detection results of the sensors equipped in the action target, as well as the state, vibration, impact, position, attitude, position velocity, attitude velocity, position acceleration, and attitude acceleration of the action target.

For example, the detection processing unit 180b detects the state of the object peripheral elements that constitute the surrounding environment of the acting object as the action-related elements. For example, the detection processing unit 180b detects the state of the object peripheral elements using the detection results received from the external sensors of the robot 100 and information received from the object peripheral elements. Examples of the external sensors are the force sensors of the robot arms 120A and 120B, the imaging devices 144 to 146, the sound collection device 147, and sensors placed in the service provision area AS. Examples of information received from the object peripheral elements are the state, vibration, impact, position, attitude, speed of position, and speed of attitude of the object peripheral elements.

The detection processing unit 180b may be configured to detect the motion-related element at any timing. For example, the detection processing unit 180b may be configured to detect the motion-related element at a predetermined timing, such as at a predetermined time interval, at the execution of a predetermined motion of the robot 100, and at a predetermined work stage. Although not limited to this, in this embodiment, the detection processing unit 180b executes the detection process at a predetermined time interval.

In the following, the motion of the robot 100 that is the target of processing by the functional components of the information processing device 180 other than the detection processing unit 180b is referred to as the "motion to be processed." For example, a predetermined specific motion of the robot 100 is an example of a motion to be processed. The unit of the motion to be processed may be set arbitrarily. For example, the division of one unit of the motion to be processed may be set by a time division such as the execution period of the motion, a division of a motion change such as the timing at which the motion changes, a specified division, other divisions, or a combination of two or more of these.

For example, when the robot 100 executes a series of operations to grasp an object placed at point A with the end effector 130A and place it on the table 140a, one unit of the operation to be processed may be set to include all of the series of operations to be executed, or may be set to include a part of the series of operations. For example, one unit of the operation to be processed may be set to include at least one of four operations: a movement operation of the robot arm 120A to move the end effector 130A to the object, a gripping operation of the end effector 130A to grip the object, a movement operation of the robot arm 120A to move the end effector 130A onto the table 140a and place the object, and a grip release operation of the end effector 130A to release the grip of the object. For example, one operation to be processed may be set for the four operations, or two or more operations to be processed may be set.

A work stage may be set by dividing a series of actions included in a work. The unit of a work stage may be set arbitrarily. For example, the division of a work stage unit may be set by a time division such as the period during which the work is performed, a division of an action change such as the timing at which the actions included in the work change, a specified division, other divisions, or a combination of two or more of these. For example, when the robot 100 receives a command from a user P requesting an object, and performs a task of transporting the object placed at point A and handing it over to the user P, a work stage unit may be set to include a portion of all actions performed during the work.

For example, one unit of work stage may be set to include at least one of four steps: a step of the robot 100 moving from a position near the user P to a position near the point A, a step of the robot 100 grasping an object at the point A with the end effector 130A and placing it on the robot 100, a step of the robot 100 moving from a position near the point A to a position near the user P, and a step of the robot 100 handing over the object on the robot 100 to the user P with the end effector 130A. For example, one work stage may be set for the four steps, or two or more work stages may be set.

The evaluation processing unit 180c executes an evaluation process for comparing and evaluating the motion-related information of the robot 100 detected by the detection processing unit 180b with the motion-related criteria of the motion-related information while the robot 100 is performing a task. Specifically, the evaluation processing unit 180c compares the information of the motion-related element included in the motion-related information with the motion-related criteria of the motion-related element to determine an evaluation result of the motion-related element, and determines an evaluation result of the motion-related information based on the evaluation result of the motion-related element.

The motion-related information of the evaluation processing unit 180c is motion-related information corresponding to the motion to be evaluated among the motions to be processed included in the work. Hereinafter, the motion to be evaluated is referred to as the "motion to be evaluated". The motion to be evaluated is at least one motion to be processed among all the motions to be processed included in the work. All of the motions to be processed may be set as the motion to be evaluated, or a part of all of the motions to be processed may be set as the motion to be evaluated. The multiple motions to be evaluated may be consecutive or discontinuous.

For example, the evaluation processing unit 180c may determine the evaluation result of the action-related element and/or the evaluation target feature of the action-related information corresponding to the action to be evaluated based on the amount of deviation, direction of deviation, frequency of deviation, etc. of the action-related element and/or the evaluation target feature of the action-related element from the action-related criterion. Furthermore, the evaluation processing unit 180c may determine the evaluation result of the action-related information based on the evaluation result of the action-related element and/or the evaluation target feature.

The evaluation processing unit 180c executes the evaluation process at least at one of the timings of each evaluation target action and each evaluation target stage, which is a work stage to be evaluated in a task. All work stages included in a task may be the evaluation target stages, or a portion of all the work stages may be the evaluation target stages. Furthermore, action-related information corresponding to all processed action included in the evaluation target stage may be the evaluation target for the evaluation of the evaluation target stage, or action-related information corresponding to a portion of the processed action of all the processed action may be the evaluation target for the evaluation of the evaluation target stage.

In the evaluation process at each timing of the evaluation target action, the evaluation processing unit 180c reads out the action-related reference data of the evaluation target action from the first storage unit 181a, compares the action-related reference data with the action-related information corresponding to the evaluation target action, and determines the evaluation result of the action-related information. For example, the evaluation processing unit 180c may determine the evaluation level of the action-related information. For example, the evaluation processing unit 180c may compare the criteria of the evaluation target feature included in the action-related reference data with the evaluation target feature of the action-related element of the action-related information, and determine the evaluation level of each evaluation target feature. The evaluation processing unit 180c may determine the evaluation level of the action-related element and/or the evaluation level of the action-related information based on the evaluation level of each evaluation target feature.

Furthermore, the evaluation processing unit 180c, for example, executes evaluation processing of the action-related information corresponding to the first action to be evaluated by the timing of the second action to be evaluated, which is the action to be evaluated next to the first action to be evaluated. The evaluation processing unit 180c executes and completes evaluation processing of the action-related information corresponding to the first action to be evaluated by the timing of the second action to be evaluated. For example, the timing of the second action to be evaluated may be the reference timing of the second action to be evaluated included in the action-related reference data, the detection timing of the action-related information corresponding to the second action to be evaluated, or a combination of these. The reference timing and the detection timing may be the start timing of the action, the completion timing of the action, or a predetermined timing between these.

The evaluation processing unit 180c may be configured to perform the evaluation process of the action-related information corresponding to the first action to be evaluated by the timing of the next action to be processed after the first action to be evaluated. For example, the timing of the next action to be processed may be the completion timing of the actual action of the robot 100 corresponding to the action to be processed, or a predetermined timing between the start timing and the completion timing. Although not limited to this, in this embodiment, the evaluation processing unit 180c starts the evaluation process without a delay time after receiving the detection result of the action-related information corresponding to the first action to be evaluated from the detection processing unit 180b.

In the evaluation process at each timing of the evaluation target stage, the evaluation processing unit 180c reads out the action-related reference data of the processed action of the evaluation target among the processed actions included in the evaluation target stage from the first storage unit 181a, compares the action-related reference data with the action-related information corresponding to the processed action of the evaluation target, and determines the evaluation result of the action-related information. The evaluation processing unit 180c integrates the evaluation results of the action-related information corresponding to the processed actions of all the evaluation targets, and determines the evaluation result of the evaluation target stage based on the integration result. For example, the evaluation processing unit 180c may determine the evaluation level of the evaluation target stage. The integration method may be any method. For example, the integration method may be a method of adding the evaluation results of multiple pieces of action-related information, a method of adding the evaluation results of multiple pieces of action-related information with weighting, or a method of integrating the evaluation results of multiple pieces of action-related information by other statistical processing.

Furthermore, the evaluation processing unit 180c, for example, executes the evaluation processing of the first work stage by the timing of the second evaluation stage, which is the evaluation stage next to the first evaluation stage. The evaluation processing unit 180c executes and completes the evaluation processing of the action-related information corresponding to the processed actions of all evaluation targets included in the first evaluation stage by the timing of the second evaluation stage. For example, the timing of the second work stage may be the reference timing of the second evaluation stage included in the action-related reference data, the detection timing of the second evaluation stage, or a combination of these. The reference timing and the detection timing may be the start timing of the first processed action in the second evaluation stage, the completion timing of the last processed action, or a predetermined timing between these.

The evaluation processing unit 180c may be configured to execute the evaluation process of the first evaluation target stage by the timing of the next work stage of the first evaluation target stage. For example, the timing of the next work stage may be the start timing of the first process target action of the work stage, the completion timing of the last process target action, or a predetermined timing between these. Although not limited to this, in this embodiment, the evaluation processing unit 180c starts the evaluation process without a delay time after receiving from the detection processing unit 180b the detection result of the action-related information corresponding to the process target action of the last evaluation target among all the process target actions of the evaluation targets included in the first evaluation target stage.

The compensation determination unit 180d determines compensation to be given to the operator PO for operating the robot 100 based on the evaluation result of the evaluation processing unit 180c. Each time the compensation determination unit 180d performs evaluation processing on the action-related information corresponding to the action to be evaluated, the compensation determination unit 180d determines compensation corresponding to the evaluation result of the action-related information. Each time the compensation determination unit 180d performs evaluation processing on the stage to be evaluated, the compensation determination unit 180d determines compensation corresponding to the evaluation result of the stage to be evaluated. For example, the compensation may be a remuneration amount, points, points, score, etc. given to the operator PO. The remuneration amount may be a remuneration amount in real currency, or may be a remuneration amount in virtual currency.

Furthermore, the compensation determination unit 180d sequentially accumulates each determined compensation while the work is being performed, and calculates an accumulated result that is the result of accumulating the compensation. Each time the compensation determination unit 180d determines a compensation, it outputs information on the determined compensation and information on the accumulated result including the determined compensation to the presentation processing unit 180e. The compensation determination unit 180d may be configured to separately accumulate the compensation for the action-related information corresponding to the evaluated action and the compensation for the evaluated stage to calculate two accumulated results, or may be configured to integrate and accumulate the two compensations to calculate one accumulated result.

The compensation determination unit 180d may be configured to determine compensation for the action-related information by adding a weight to the evaluation level of the action-related information corresponding to the action to be evaluated. The compensation determination unit 180d may be configured to determine compensation for the action-related information by adding a weight to the evaluation level of the evaluation target feature of the action-related information and integrating the evaluation levels after the weighting, or may be configured to further add a weight to the determined compensation for the action-related information. For example, the difficulty of the operation differs depending on the action to be evaluated. The compensation determination unit 180d may be configured to change the weight depending on the action to be evaluated. The compensation determination unit 180d may be configured to change the weight depending on the evaluation target feature.

The compensation determination unit 180d may be configured to determine the compensation by adding a weight to the evaluation level of the evaluation target stage. The compensation determination unit 180d may determine the compensation for the evaluation target stage by adding a weight to the evaluation level of the action-related information corresponding to the processing target action of the evaluation target included in the evaluation target stage and integrating the evaluation levels after the weighting, or may be configured to further add a weight to the determined compensation for the evaluation target stage. The compensation determination unit 180d may be configured to change the weight depending on the evaluation target stage and the processing target action. The integration method used by the compensation determination unit 180d in the above may be any method.

The presentation processing unit 180e executes a presentation process to have the presentation device 203, which receives information presented while the operator PO is operating the operation input device 201, present the evaluation result of the evaluation processing unit 180c while the work is being performed. Although not limited to this, in this embodiment, the presentation processing unit 180e causes the presentation device 203 to present the consideration determined by the consideration determination unit 180d and the accumulated result of the consideration while the work is being performed. The presentation processing unit 180e generates data including the consideration received from the consideration determination unit 180d and the accumulated result, and transmits the data and a presentation command for the data to the operation terminal 200. For example, the presentation processing unit 180e may generate image data, audio data, character string data, or data including at least two of these, indicating the consideration and the accumulated result, and transmit it to the operation terminal 200.

When the evaluation processing unit 180c executes the evaluation process at the timing of each evaluated action, the presentation processing unit 180e transmits the evaluation results of the action-related information corresponding to the evaluated action to the operation terminal 200 in the same order as the execution order of the evaluated action, and causes the presentation device 203 to present them. The evaluation results may include an evaluation level and/or a compensation. For example, the presentation processing unit 180e executes the presentation process of the evaluation results of the action-related information corresponding to the first evaluated action by the timing of the next second evaluated action. The timing of the second evaluated action may be any of the timings exemplified for the second evaluated action of the evaluation processing unit 180c, and may be the same as or different from the timing of the second evaluated action of the evaluation processing unit 180c.

When the evaluation processing unit 180c executes the evaluation process at the timing of each evaluation target stage, the presentation processing unit 180e transmits the evaluation results of the evaluation target stage to the operation terminal 200 in the same order as the execution order of the evaluation target stages, and causes the presentation device 203 to present them. The evaluation results may include an evaluation level and/or a compensation. For example, the presentation processing unit 180e executes the presentation process of the evaluation results of the first evaluation target stage by the timing of the next second evaluation target stage. The timing of the second evaluation target stage may be any of the timings exemplified for the timing of the second evaluation work stage of the evaluation processing unit 180c, and may be the same as or different from the timing of the second evaluation work stage of the evaluation processing unit 180c.

The accumulation processing unit 180f associates the action-related reference data, information on the evaluation result of the action-related information based on the action-related reference data, and the date and time of the performance of the action of the robot 100 corresponding to the action-related information, and stores them as accumulated data in the second storage unit 181b. The accumulation processing unit 180f stores the accumulated data related to the action to be evaluated and the accumulated data related to the stage to be evaluated in the second storage unit 181b.

For example, the accumulation processing unit 180f may store, as accumulated data related to the evaluated action, information on the evaluated action, information on the evaluation result of the action-related information corresponding to the evaluated action, and the action-related reference data used in the evaluation process of the evaluation result in the second storage unit 181b in association with each other. The evaluation result information may include an evaluation level and/or compensation.

For example, the accumulation processing unit 180f may store in the second storage unit 181b, as accumulated data relating to the evaluation target stage, information on the evaluation target action included in the evaluation target stage, information on the evaluation result of the action-related information corresponding to the processing target action, information on the evaluation result of the evaluation target stage, and at least some of the action-related reference data used in the evaluation process of the evaluation result, in association with each other. The evaluation result information may include an evaluation level and/or compensation.

The criterion determination unit 180g determines new action-related reference data using the accumulated data in the second storage unit 181b, and stores the action-related reference data in the first storage unit 181a. The criterion determination unit 180g determines new second action-related reference data using the first action-related reference data included in the accumulated data in the second storage unit 181b and an evaluation result based on the first action-related reference data. The evaluation result based on the first action-related reference data is an evaluation result determined using the first action-related reference data.

For example, the criterion determination unit 180g can accept a specified condition for determining the second action-related reference data. For example, the criterion determination unit 180g accepts the specified condition from the operation terminal 200. For example, the specified condition may be a condition for limiting the first action-related reference data and the evaluation result used in determining the second action-related reference data. The criterion determination unit 180g may extract, from the accumulated data in the second storage unit 181b, information on the evaluation result that satisfies the specified condition and the first action-related reference data that is the action-related reference data used in the evaluation result, and determine the second action-related reference data using the information on the evaluation result and the first action-related reference data.

For example, the specified condition may be a condition that limits an attribute of the evaluation result for which the first action-related reference data is used. The attribute of the evaluation result may be the user that the evaluation result targets, the robot that the evaluation result targets, the task that the evaluation result targets, the work location that the evaluation result targets, the work environment that the evaluation result targets, the task execution date that the evaluation result targets, and the task execution period that the evaluation result targets, etc. The specified condition may be a condition that limits at least one attribute of the evaluation result.

For example, the criterion determination unit 180g can use the evaluation result of a specific user and the first motion-related reference data to determine second motion-related reference data corresponding to the user. For example, if the evaluation level of the user is low, the criterion determination unit 180g may generate second motion-related reference data with a lower standard than the first motion-related reference data. When the user operates the robot 100, the user can issue a command to the information processing device 180 via the operation terminal 200 to request an evaluation using the second motion-related reference data. This allows the user to receive a higher evaluation from the information processing device 180, which can increase the user's motivation to operate the robot.

The first drive command unit 190a generates drive commands for the arm drive devices M1A to M4A of the robot arm 120A to drive the joints (not shown) until the target position command and target force command state are reached, and outputs the drive commands to the drive circuit 142a. The drive commands include command values for the servo motor current. The first drive command unit 190a generates drive commands using feedback information from the servo motor.

The second drive command unit 190b generates drive commands for the arm drive devices M1B to M4B of the robot arm 120B to drive the joints (not shown) until the target position command and target force command state are reached, and outputs the drive commands to the drive circuit 142a. The drive commands include command values for the servo motor current. The second drive command unit 190b generates drive commands using feedback information from the servo motor.

The third drive command unit 190c generates a drive command for driving the drive device (not shown) of the end effector 130A to the target state included in the third operation command, and outputs the drive command to the drive circuit 142a.

The fourth drive command unit 190d generates a drive command for driving the drive device (not shown) of the end effector 130B to the target state included in the fourth operation command, and outputs the drive command to the drive circuit 142a.

The fifth drive command unit 190e generates a drive command for the transport drive device 113 of the transport vehicle 110 to drive the drive wheel 111 to the state of the target position command included in the fifth operation command, and outputs the drive command to the drive circuit 142a. The drive command includes a command value for the servo motor current. The fifth drive command unit 190e generates the drive command using feedback information of the servo motor.

[Operation of the robot system]
An example of the operation of the robot system 1 according to the embodiment will be described with reference to Fig. 5 and Fig. 6. Fig. 5 is a flowchart showing an example of the operation of the robot system 1 according to the embodiment. Fig. 6 is a plan view showing an example of the operation of the robot system 1 according to the embodiment. An example of executing a service in which the robot 100 searches for an item W requested by a user P from the storage shelf SR and hands it over to the user P will be described below. In the examples of Figs. 5 and 6, the information processing device 180 executes an evaluation process and a presentation process for each evaluation target action and each evaluation target stage, respectively.

First, the operator PO inputs a request to operate the robot into the operation terminal 200, and the operation terminal 200 transmits the request to the server 300 (step S101). The server 300 searches for a robot 100 that the operator PO can operate, and connects the information processing device 180 of the searched robot 100 to the operation terminal 200 via the communication network N (step S102).

When the operator PO receives notification of connection completion from the server 300, he/she inputs a command to start executing the robot operation into the operation terminal 200. The operation terminal 200 transmits the command to the information processing device 180. The information processing device 180 starts the robot 100 so that it can operate according to the operation command received from the operation terminal 200. The information processing device 180 sets the execution flag for the work for servicing the robot 100 to the ON state, that is, starts the work (step S103).

Next, the operator PO inputs the designation conditions for the motion-related reference data to be used in the evaluation of the robot operation into the operation terminal 200 (step S104). The operation terminal 200 transmits a command for the designation conditions to the information processing device 180. For example, the designation conditions may be conditions that designate standard motion-related reference data, or conditions that designate attributes of the evaluation results to be used. In this example, the operator PO inputs designation conditions that designate the use of the evaluation results for the operator PO.

Next, the information processing device 180 generates new action-related reference data using the specified conditions received from the operation terminal 200 and the accumulated data accumulated in the information processing device 180, and decides to use the action-related reference data (step S105). The action-related reference data is action-related reference data adjusted according to the evaluation result for the operator PO.

Next, the operator PO searches for a user P requesting a service within the service provision area AS while checking the screen of the presentation device 203 that displays the captured images, etc., of the imaging device 145. When the operator PO finds the user P, the operator PO operates the operation terminal 200 while visually checking the screen of the presentation device 203 that displays the captured images, etc., of the imaging devices 145 and/or 146, to make the robot 100 travel to a position near the user P (step S106).

The information processing device 180 detects the motion-related information of the robot 100 at a predetermined time interval while the robot 100 is running, and executes evaluation processing and presentation processing of the motion-related information each time the motion-related information is detected (step S107). In this example, each of the motion-related information at the predetermined time intervals corresponds to a motion to be evaluated. The information processing device 180 displays the compensation for the motion-related information and the accumulated compensation result on the presentation device 203. For example, the information processing device 180 evaluates the evaluation target characteristics such as the running speed, running acceleration, running trajectory, and interference with the robot peripheral elements of the robot for each motion-related information. In this example, the information processing device 180 executes the detection processing, evaluation processing, and presentation processing without any delay between them, so that the operator PO can check the compensation for the motion to be evaluated and the accumulated compensation result without any delay from the time when the operation for the motion to be evaluated is input to the operation terminal 200, that is, in real time (real time).

When the robot 100 arrives at a position near the user P, the information processing device 180 executes an evaluation process and a presentation process for the first work stage, which is the travel process from the starting position to the nearby position, and causes the presentation device 203 to display the price and estimated cost of the first work stage (step S108).

When the operator PO confirms the arrival of the robot 100 near the user P via the screen of the presentation device 203, he/she inquires of the user P about the desired item W via input to the operation terminal 200 and output from the display device 148 and/or audio output device 149 of the robot 100 (step S109).

Next, when the user P inputs information about the requested item W1 to the touch panel of the sound collection device 147 and/or the display device 148, the operator PO requests information about the position of the requested item W1 in the storage shelf SR from the information processing device 180 via input to the operation terminal 200. The information processing device 180 may store information about the service provision area AS in which the robot 100 is located and information about the provided services, in addition to information about the robot 100 on which the information processing device 180 is mounted. The operation terminal 200 receives information about the position PW1 of the requested item W1 from the information processing device 180 and presents it to the operator PO (step S110). Note that the server 300 may hold information about the service provision area AS and information about the provided services, and the operation terminal 200 may request information about the position of the requested item W1 from the server 300.

Next, the operator PO operates the operation terminal 200 while visually checking the screen of the presentation device 203, which displays information about the position PW1, images captured by the imaging devices 145 and/or 146, etc., to move the robot 100 to a position near the position PW1 (step S111).

Similar to the process of step S107, the information processing device 180 detects the motion-related information of the robot 100 at a predetermined time interval while the robot 100 is running, and performs evaluation processing and presentation processing of the motion-related information each time the information is detected (step S112).

Similar to the processing of step S108, when the robot 100 arrives at a position near the position PW1, the information processing device 180 executes an evaluation process and a presentation process for the second work stage, which is the traveling process from a position near the user P to a position near the position PW1 (step S113).

When the operator PO confirms via the screen of the presentation device 203 that the robot 100 has arrived at a position near the position PW1, he/she inputs an operation to the operation terminal 200 to operate the robot arms 120A and 120B and the end effectors 130A and 130B (step S114). This operation is to take the requested item W1 stored in the storage shelf SR from the storage shelf SR using the end effectors 130A and 130B, and place the requested item W1 on the table 140a of the robot 100. The operator PO inputs an operation to the operation terminal 200 while visually checking the screen of the presentation device 203, which displays the captured images of the imaging devices 144 and 145, etc.

The information processing device 180 detects motion-related information of the robot 100 while the robot 100 is in motion, and executes evaluation processing and presentation processing of the motion-related information each time motion-related information corresponding to the motion to be evaluated is detected (step S115). For example, the motion to be evaluated is at least one of the motions to be processed, such as the movement motion of the robot arms 120A and 120B that moves the end effectors 130A and 130B to the requested item W1, the gripping motion of the end effectors 130A and 130B that grips the requested item W1, the movement and placement motion of the robot arms 120A and 120B that moves and places the requested item W1 on the table 140a, and the grip release motion of the end effectors 130A and 103B that releases the grip of the requested item W1. In this example, all the motions to be processed are motions to be evaluated.

The information processing device 180 evaluates the characteristics of the operation related information corresponding to each operation to be evaluated, such as the time required for the operation of the robot 100, the position, posture, speed of movement of the position, speed of movement of the posture, acceleration of the position, acceleration of the posture and movement trajectory of the posture, the position, posture and state of the requested item W1, the force and impact that the robot 100 applies to the requested item W1, the contact, force and impact that the robot 100 applies to the surrounding elements of the target object (such as the shelves of the storage shelf SR and other items), and interference between the robot 100 and the surrounding elements of the robot. For example, the acceleration of the end effectors 130A and 103B may indicate the impact that the end effectors 130A and 103B apply to the requested item W1, etc. For example, the information processing device 180 may be configured to process an image of the requested item W1 captured by the imaging device 144 and/or 145, etc., to detect the holding state of the requested item W1 held by the end effectors 130A and 103B, and to evaluate the holding state as an evaluation target feature. The information processing device 180 may be configured to process an image of the requested item W1 on the table 140a captured by the imaging device 144, etc., to detect the placement state of the requested item W1, and to evaluate the placement state as an evaluation target feature.

When the information processing device 180 detects the release of the grip of the requested item W1 by the end effectors 130A and 103B, it executes the evaluation process and presentation process of the third work stage, which is the retrieval operation process from the arrival of the robot 100 at a position near the position PW1 to the completion of the release of the grip, and displays the price and the accumulated result on the presentation device 203 (step S116).

Next, the operator PO confirms via the screen of the presentation device 203 that the requested item W1 has been placed on the table 140a, and then operates the operation terminal 200 while visually checking the screen of the presentation device 203, which displays the captured images of the imaging devices 145 and/or 146, to move the robot 100 to a position close to the user P (step S117).

Similar to the process of step S107, the information processing device 180 detects the motion-related information of the robot 100 at a predetermined time interval while the robot 100 is running, and performs evaluation processing and presentation processing of the motion-related information each time the information is detected (step S118).

Similar to the processing of step S108, when the robot 100 arrives at a position near the user P, the information processing device 180 executes evaluation processing and presentation processing of the fourth work stage, which is the traveling process from a position near the position PW1 to a position near the user P (step S119).

Next, when the operator PO confirms via the screen of the presentation device 203 that the robot 100 has arrived at a position near the user P, he/she inputs an operation to the operation terminal 200 to operate the robot arms 120A and 120B and the end effectors 130A and 130B (step S120). This operation is an operation to hand over the requested item W1 placed on the table 140a to the user P using the end effectors 130A and 130B. The operator PO inputs an operation to the operation terminal 200 while visually checking the screen of the presentation device 203 that displays the captured images of the imaging devices 144 and 145, etc.

The information processing device 180 detects motion-related information of the robot 100 during the motion of the robot 100, and executes evaluation processing and presentation processing of the motion-related information each time motion-related information corresponding to the motion to be evaluated is detected (step S121). For example, the motion to be evaluated is at least one of the motions to be processed, such as the movement motion of the robot arms 120A and 120B that moves the end effectors 130A and 130B to the requested item W1, the gripping motion of the end effectors 130A and 130B that grips the requested item W1, the movement motion of the robot arms 120A and 120B that moves the requested item W1 in front of the user P and presents it to him, and the grip release motion of the end effectors 130A and 103B that releases the grip of the requested item W1 depending on the holding state of the requested item W1 by the user P. In this example, all the motions to be processed are motions to be evaluated.

The information processing device 180 evaluates the evaluation features of the operation-related information corresponding to each evaluation target operation, such as the time required for the operation of the robot 100, the position, posture, position movement speed, posture movement speed, position acceleration, posture acceleration and movement trajectory of the end effectors 130A and 103B, the position, posture and state of the requested item W1, the force and impact that the robot 100 applies to the requested item W1, and the contact, force and impact that the robot 100 applies to the surrounding elements of the object (user P). For example, the information processing device 180 may be configured to process the image of the requested item W1 captured by the imaging device 144 and/or 145, etc., to detect the holding state of the requested item W1 grasped by the end effectors 130A and 103B, and evaluate the holding state as the evaluation target feature.

When the information processing device 180 detects the end effectors 130A and 103B releasing the grip of the requested item W1, it executes the evaluation process and presentation process of the fifth work stage, which is the handover operation process from the arrival of the robot 100 at a position near the user P to the completion of releasing the grip, and displays the price and the accumulated result on the presentation device 203 (step S122).

When the operator PO ends the robot operation, he/she inputs a command to end the operation into the operation terminal 200. When the information processing device 180 receives the command to end the operation from the operation terminal 200 (Yes in step S123), it ends the series of processes. The information processing device 180 sets the task execution flag to OFF and stops the robot 100. The server 300 cuts off the connection between the operation terminal 200 and the information processing device 180. When the information processing device 180 does not receive the command to end the operation (No in step S123), it returns to step S106.

In the process of steps S106 to S123, the information processing device 180 stores and accumulates the action-related information corresponding to the action to be evaluated, the evaluation results of the work stage, the compensation, and the accumulated compensation results, in association with the action-related reference data used in the evaluation.

By processing steps S101 to S123 as described above, the operator PO can operate the robot 100 while checking the evaluation of the operator PO's operation through the compensation and the accumulated compensation results for each evaluated action and task stage. This allows the operator PO to find specific measures for improving his/her robot operation skills. The operator PO can then be motivated to improve his/her robot operation skills in order to receive more compensation.

(Modification)
This modification example differs from the embodiment in that the robot system includes a learning device 400. Below, this modification example will be described focusing on the differences from the embodiment, and descriptions of the same points as in the embodiment will be omitted as appropriate.

FIG. 7 is a block diagram showing an example of the functional configuration of the information processing device 180A and the learning device 400 according to the modified example. As shown in FIG. 7, the information processing device 180A further includes a data input/output unit 180i as a functional component. The data input/output unit 180i inputs and outputs data between the information processing device 180A and the learning device 400. For example, the data input/output unit 180i outputs accumulated data stored in the second storage unit 181b to the learning device 400. In response to a request from the criterion determination unit 180g, the data input/output unit 180i requests action-related reference data from the learning device 400. The data input/output unit 180i receives the designated condition and information on the evaluation result that satisfies the designated condition from the criterion determination unit 180g and sends it to the learning device 400. The learning device 400 sends the action-related reference data corresponding to the designated condition and the evaluation result to the data input/output unit 180i.

The learning device 400 includes a computer device, similar to the information processing device 180A. Although not limited thereto, in this modified example, the learning device 400 is a device separate from the information processing device 180A and the robot controller 190, and is connected to the information processing device 180A so as to be capable of data communication via wired communication, wireless communication, or a combination thereof. Any wired communication or wireless communication may be used. The learning device 400 is mounted on the robot 100 together with the information processing device 180A. The learning device 400 may be incorporated into the information processing device 180A or the robot controller 190. The learning device 400 and the information processing device 180A form an information processing system 500.

The learning device 400 may also be configured to input and output data to and from the information processing device 180A via a storage medium. The storage medium may include a semiconductor-based or other integrated circuit (IC), a hard disk drive (HDD), a hybrid hard disk drive (HHD), an optical disk, an optical disk drive (ODD), a magneto-optical disk, a magneto-optical drive, a floppy disk drive (FDD), a magnetic tape, a solid-state drive (SSD), a RAM drive, a secure digital card or drive, any other suitable storage medium, or a combination of two or more of these.

In this modified example, the learning device 400 is mounted on the robot 100, but may be configured to be arranged outside the robot 100 and connected to the information processing device 180A via the communication network N. The learning device 400 may be configured to be connectable to the information processing devices 180A of multiple robots 100. For example, the learning device 400 may be arranged in the service provision area AS, the operation area AO, or other locations. For example, the learning device 400 may be arranged at the location of the server 300 and configured to perform data communication with the information processing device 180A via the server 300. The learning device 400 may be incorporated into the server 300.

The learning device 400 includes, as functional components, a data input/output unit 401, a learning data storage unit 402, a machine learning unit 403, an input data processing unit 404, and an output data processing unit 405. The function of the learning data storage unit 402 is realized by a memory and/or storage that the learning device 400 may include, and the functions of the functional components other than the learning data storage unit 402 are realized by a processor, etc., included in the learning device 400.

The data input/output unit 401 receives accumulated data from the data input/output unit 180i of the information processing device 180A, and stores the accumulated data in the learning data storage unit 402 as data for machine learning. The data input/output unit 401 receives a request command for action-related reference data, information on specified conditions, and information on evaluation results that satisfy the specified conditions from the data input/output unit 180i, and outputs them to the input data processing unit 404. The data input/output unit 401 receives action-related reference data from the output data processing unit 405, and sends the action-related reference data to the data input/output unit 180i.

The input data processing unit 404 converts the information on the specified conditions and the evaluation results received from the data input/output unit 401 into data that can be input to the machine learning model of the machine learning unit 403, and outputs it to the machine learning unit 403. The output data processing unit 405 converts the output data of the machine learning unit 403 into action-related reference data, and outputs it to the data input/output unit 401.

The machine learning unit 403 includes a machine learning model. The machine learning unit 403 uses data for machine learning to train the machine learning model, thereby improving the accuracy of output data for input data to the machine learning model. The machine learning model may be configured as a neural network, random forest, genetic programming, a regression model, a tree model, a Bayesian model, a time series model, a clustering model, an ensemble learning model, or the like, but in this modified example, it is configured as a neural network. The neural network is configured with a plurality of node layers including an input layer and an output layer. The node layer includes one or more nodes. When the neural network is configured with an input layer, an intermediate layer, and an output layer, the neural network sequentially performs output processing from the input layer to the intermediate layer and output processing from the intermediate layer to the output layer for information input to a node in the input layer, and outputs an output result that matches the input information. Each node in one layer is connected to each node in the next layer, and the connections between the nodes are weighted. The information of the nodes in one layer is output to the nodes in the next layer with the weighting of the connections between the nodes added.

The machine learning model receives as input data condition-related information related to the specified conditions and evaluation-related information related to the evaluation results, and receives as output data information related to the condition-related information and the action-related criteria corresponding to the evaluation-related information. The condition-related information may be information indicating attributes of the specified conditions and/or the evaluation results, and the evaluation-related information may be information indicating the evaluation results.

Furthermore, the machine learning model uses the condition-related information and evaluation-related information included in the machine learning data as machine learning input data, and uses criterion-related information related to the condition-related information and the action-related criterion data corresponding to the evaluation-related information as teacher data. The criterion-related information may be information indicating the action-related criterion data. For example, in machine learning, the machine learning unit 403 adjusts the weighting of the connections between nodes in the neural network so that the output data of the machine learning model when machine learning input data is input matches or minimizes the error between the output data and the teacher data corresponding to the machine learning input data. When condition-related information and evaluation-related information are input, the machine learning model after such weighting adjustment can output criterion-related information appropriate to the information.

The learning device 400 as described above can output action-related reference data that is more appropriate for the specified conditions and evaluation results than the action-related reference data determined by the reference determination unit 180g of the embodiment. In this modified example, the machine learning model of the machine learning unit 403 may include the functions of the output data processing unit 405 and may be configured to output output data similar to that of the output data processing unit 405.

Other Embodiments
Although the embodiment and the modified examples of the present disclosure have been described above, the present disclosure is not limited to the above embodiment and modified examples. In other words, various modifications and improvements are possible within the scope of the present disclosure. For example, the embodiment and modified examples to which various modifications have been applied, and forms constructed by combining components in different embodiments and modified examples are also included within the scope of the present disclosure.

For example, in the embodiment and the modified example, the work of the robot 100 targeted by the robot system 1 is work related to the service industry, but is not limited to this. The robot system 1 may target any work. For example, the robot system 1 may target work related to industry, agriculture, and other industries, etc.

In the embodiment and the modified example, the robot system 1 includes a robot 100, but the robot included in the robot system 1 may be any robot capable of performing work. For example, the robot may include a moving device having a different configuration from the transport vehicle 110, and may include, for example, legs, etc. The robot may not include a moving device. For example, the robot may include components other than a robot arm, and may be, for example, a humanoid robot, an animal-type robot, etc.

In the embodiment and the modified example, the information processing device 180 is configured to use image data of the robot peripheral elements, the target object, and the target peripheral elements, which are elements other than the robot 100, in order to detect the state of these external elements, but is not limited to this. For example, the information processing device 180 may be configured to detect the state of the external elements using the detection result of an external sensor, which is a sensor arranged separately from the external elements, and/or the detection result of an on-board sensor, which is a sensor arranged on the external elements. The external sensor may be configured to detect the position and posture, etc. of the external elements from the outside. For example, the external sensor is configured to perform detection using light waves, lasers, magnetism, radio waves, electromagnetic waves, ultrasonic waves, or a combination thereof, and may be a photoelectric sensor, a laser sensor, a radio wave sensor, an electromagnetic wave sensor, an ultrasonic sensor, various LiDARs, etc. The on-board sensor may be configured to move together with the external elements and detect the position and posture, etc. of the external elements. For example, the on-board sensor may be an acceleration sensor, an angular velocity sensor, a magnetic sensor, a GPS (Global Positioning System) receiver, etc.

The information processing devices 180 and 180A according to the embodiment and the modified example may be configured to use AI (artificial intelligence) for processing. For example, AI can be used for processing to determine an evaluation of an action to be processed, processing to determine an evaluation of a task stage, processing to calculate compensation and an accumulated result, image processing, and the like. For example, AI may include a learning model that performs machine learning. For example, the learning model may include a neural network.

In the embodiment and the modified example, the server 300 may be configured to have at least some of the functions of the information processing devices 180 and 180A and the learning device 400. For example, the server 300 may be configured to have at least one function of the functional components of the information processing devices 180 and 180A shown in FIG. 4 and FIG. 7. The server 300 may be configured to have at least one function of the functional components of the learning device 400 shown in FIG. 7.

Examples of each aspect of the technology of the present disclosure are given below. An information processing device according to one aspect of the present disclosure is an information processing device that evaluates the operation of a robot to have the robot perform a task, and includes a detection processing unit that performs, during the execution of the task, a detection process that detects action-related information, which is information related to the action of the robot that operates in accordance with an operation command input by a user to an operation device, an evaluation processing unit that performs, during the execution of the task, an evaluation process that compares and evaluates the action-related information detected by the detection processing unit with action-related standards, which are standards related to the action of the robot, and a presentation processing unit that performs, during the execution of the task, a presentation device that presents the evaluation results of the evaluation processing unit to a presentation device that receives presentation of information while the user is operating the operation device. The evaluation processing unit executes the evaluation process at least at one of the timings of each evaluation target action, which is the action of the robot to be evaluated, and each evaluation target stage, which is the task stage of the task to be evaluated, and the presentation processing unit, when the evaluation process is executed at the timing of each evaluation target action, causes the presentation device to present the evaluation results for the evaluation target action in the same order as the execution order of the evaluation target action, and when the evaluation process is executed at the timing of each evaluation target stage, causes the presentation processing unit to present the evaluation results for the evaluation target stage in the same order as the execution order of the evaluation target stage.

According to the above aspect, the information processing device executes an evaluation process for the action to be evaluated and/or the stage to be evaluated at least one of the timing for each action to be evaluated and the timing for each stage to be evaluated, and presents the evaluation result to the user via the presentation device. The user can then check the evaluation of the operation at the above timing. Furthermore, the user can check the evaluation of the operation in the same order as the order of the operations. This allows the user to recognize the evaluated operation and the evaluation result for that operation, and therefore may be motivated to improve the accuracy of the operation the next time the operation to be evaluated is performed. Therefore, the information processing device can increase the user's motivation to operate a robot.

In an information processing device according to one aspect of the present disclosure, when the evaluation processing unit executes the evaluation processing at the timing of each of the actions to be evaluated, the evaluation processing unit may execute the evaluation processing for a first action to be evaluated by the timing of a second action to be evaluated, which is the action to be evaluated next to the first action to be evaluated, and the presentation processing unit may execute the presentation processing of the evaluation result for the first action to be evaluated by the timing of the second action to be evaluated.

According to the above aspect, the information processing device executes the evaluation process and the presentation process for the evaluated action by the timing of the next evaluated action. Therefore, the user can accurately recognize the evaluated evaluated action and the evaluation result for the evaluated action.

In an information processing device according to one aspect of the present disclosure, when the evaluation processing unit executes the evaluation processing at the timing of each evaluation target stage, the evaluation processing unit may execute the evaluation processing for the first evaluation target stage by the timing of the second evaluation target stage, which is the evaluation target stage next to the first evaluation target stage, and the presentation processing unit may execute the presentation processing of the evaluation result for the first evaluation target stage by the timing of the second evaluation target stage.

According to the above aspect, the information processing device executes the evaluation process and the presentation process for the evaluation target stage by the timing of the next evaluation target stage. Therefore, the user can accurately recognize the evaluation target stage that has been evaluated and the evaluation result for that evaluation target stage.

The information processing device according to one aspect of the present disclosure may further include a compensation determination unit that determines compensation to be given to the user for operating the robot based on the evaluation result, and the presentation processing unit may cause the presentation device to present the compensation determined by the compensation determination unit and the result of accumulating the compensation during the execution of the task.

According to the above aspect, the information processing device presents the compensation indicating each evaluation result and the accumulated compensation to the user via the presentation device. This may increase the user's motivation to operate the robot in order to obtain more compensation. For example, the compensation may be a reward amount, points, points, score, etc.

In an information processing device according to one aspect of the present disclosure, the detection processing unit may detect, as the operation-related information, at least one of the operation command, an operation command generated for the robot to cause the robot to operate in accordance with the operation command, an actual operation result of the robot, a state of the environment surrounding the robot, a state of an object on which the robot acts, and a state of the environment surrounding the object. According to the above aspect, the information processing device can perform evaluation processing based on a variety of information.

In an information processing device according to one aspect of the present disclosure, the evaluation processing unit may compare and evaluate a plurality of evaluation target features, which are features for evaluating the action-related information, with criteria for the plurality of evaluation target features included in the action-related criteria, and evaluate the action-related information based on the evaluation results of the plurality of evaluation target features.

According to the above aspect, the evaluation method in the information processing device can be clear and simple. Therefore, it is possible to reduce the amount of processing and speed up the evaluation process. For example, the evaluation target features may include features such as the time required for the robot's movement, position, posture, trajectory, position speed, posture speed, position acceleration, and posture acceleration, the force and impact that the robot exerts on the robot's surrounding environment, as well as changes in the position, posture, shape, and state of the surrounding environment, the position, posture, position speed, posture speed, position acceleration, and posture acceleration of the object on which the robot acts, changes in shape, changes in state, and workmanship of the object, the force and impact that the robot exerts on the surrounding environment of the object, as well as changes in the position, posture, shape, and state of the surrounding environment. The information processing device may determine the evaluation result of the movement-related information by weighting the evaluation results of each of the multiple evaluation target features.

The information processing device according to one aspect of the present disclosure further includes a storage unit and a criterion determination unit, the storage unit stores information on the action-related criterion and information on the evaluation result of the action-related information based on the action-related criterion, and the criterion determination unit may determine information on the second action-related criterion using information on the first action-related criterion stored in the storage unit and information on the evaluation result based on the first action-related criterion. According to the above aspect, the information processing device can newly determine the action-related criterion in accordance with the evaluation result stored in the storage unit.

In an information processing device according to one aspect of the present disclosure, the criterion determination unit may extract, from the information stored in the storage unit, information on the evaluation result that satisfies a specified condition and information on the first action-related criterion that is the action-related criterion used in the evaluation result, and determine information on the second action-related criterion using the extracted information on the evaluation result and the information on the first action-related criterion.

According to the above aspect, the information processing device can determine new action-related criteria using evaluation results that satisfy the specified conditions. Thus, the information processing device can determine action-related criteria for various purposes. For example, the specified conditions may be conditions that specify evaluation results that correspond to an object acted upon by the robot, a robot, a task, a work location, a work environment, a date and period during which the task is performed, and the like.

A learning device according to one embodiment of the present disclosure includes a learning data storage unit that stores information on the action-related criterion in an information processing device according to one embodiment of the present disclosure and information on the evaluation result of the action-related information based on the action-related criterion as machine learning data, and a machine learning unit, the machine learning unit performs machine learning using the evaluation result information included in the machine learning data as machine learning input data and the action-related criterion information included in the machine learning data and used in the evaluation result as teacher data, the machine learning unit uses evaluation information corresponding to the evaluation result information as input data and the action-related criterion information corresponding to the evaluation information as output data.

According to the above aspect, when the learning device after machine learning receives input of evaluation information corresponding to the evaluation result, it can output movement-related criteria appropriate to the evaluation information. For example, when evaluation information corresponding to a high evaluation result is input to the learning device, the learning device outputs simple, low-level movement-related criteria that are likely to obtain a high evaluation. For example, when evaluation information corresponding to a low evaluation result is input to the learning device, the learning device outputs advanced-level movement-related criteria that are difficult to obtain a high evaluation. For example, a user can use the learning device to obtain movement-related criteria of the level of difficulty desired by the user and use them in robot operation.

An information processing system according to an aspect of the present disclosure includes an information processing device according to an aspect of the present disclosure and a learning device according to an aspect of the present disclosure, and the learning device is configured to receive input data from the operation device or the information processing device, and output output data corresponding to the input data to the information processing device. According to the above aspect, the information processing system can obtain the same effect as the information processing device and learning device according to an aspect of the present disclosure. The information processing system can execute evaluation processing in the information processing device using the action-related criteria output from the learning device.

A robot system according to one aspect of the present disclosure includes an information processing device according to one aspect of the present disclosure, the robot, and a robot controller that controls the operation of the robot, and the information processing device is communicably connected to the robot controller. According to the above aspect, the robot system can obtain the same effects as the information processing device according to one aspect of the present disclosure.

A robot system according to one aspect of the present disclosure includes an information processing system according to one aspect of the present disclosure, the robot, and a robot controller that controls the operation of the robot, and the information processing device is communicatively connected to the robot controller. According to the above aspect, the robot system can obtain the same effects as the information processing system according to one aspect of the present disclosure.

In addition, all of the ordinal numbers, quantities, and other numbers used above are provided as examples to specifically explain the technology of the present disclosure, and the present disclosure is not limited to the exemplified numbers. In addition, the connection relationships between the components are provided as examples to specifically explain the technology of the present disclosure, and the connection relationships that realize the functions of the present disclosure are not limited to these.

The division of blocks in the functional block diagram is just one example, and multiple blocks may be realized as one block, one block may be divided into multiple blocks, and/or some functions may be transferred to other blocks. Furthermore, the functions of multiple blocks having similar functions may be processed in parallel or in a time-shared manner by a single piece of hardware or software.

1 Robot system 100 Robot 170 Control device 180, 180A Information processing device 180a Operation command unit 180b Detection processing unit 180c Evaluation processing unit 180d Consideration determination unit 180e Presentation processing unit 180f Accumulation processing unit 180g Criteria determination unit 181a First memory unit 181b Second memory unit (accumulation unit)
190 Robot controller 200 Operation terminal (operation device)
203 Presentation device 400 Learning device 402 Learning data storage unit 403 Machine learning unit 500 Information processing system

Claims (13)

An information processing device that evaluates an operation of a robot to cause the robot to execute a task,
a detection processing unit that performs a detection process to detect operation-related information, which is information related to an operation of the robot that operates according to an operation command input by a user to an operation device, during the execution of the work;
an evaluation processing unit that performs an evaluation process, during the execution of the task, of comparing the motion-related information detected by the detection processing unit with a motion-related criterion that is a criterion related to the motion of the robot, and evaluating the motion-related information;
a presentation processing unit that performs a presentation process to present the evaluation result of the evaluation processing unit on a presentation device that receives presentation of information while the user is operating the operation device, during the execution of the work;
the evaluation processing unit executes the evaluation process at least at one of a timing for each evaluation target motion, which is a motion of the robot to be evaluated, and a timing for each evaluation target stage, which is a task stage to be evaluated in the task;
When the evaluation process is executed at a timing for each of the evaluation target actions, when the evaluation process is performed on the evaluation target action by the evaluation processing unit, the presentation processing unit causes the presentation device to present the evaluation result regarding the evaluation target action that has been subjected to the evaluation process;
When the evaluation process is executed at the timing of each of the evaluation target stages, when the evaluation process is performed on the evaluation target stage by the evaluation processing unit, the presentation processing unit causes the presentation device to present the evaluation result regarding the evaluation target stage that has been subjected to the evaluation process ;
One of the tasks includes a plurality of the evaluation target actions and a plurality of the evaluation target steps.
Information processing device. When the evaluation processing unit executes the evaluation process at the timing of each of the evaluated actions, the evaluation processing unit executes the evaluation process for a first evaluated action by the timing of a second evaluated action that is the next evaluated action after the first evaluated action,
The information processing device according to claim 1 , wherein the presentation processing unit executes the presentation process of the evaluation result regarding the first evaluated action by a timing of the second evaluated action. When the evaluation processing unit executes the evaluation process at the timing of each evaluation target stage, the evaluation processing unit executes the evaluation process for the first evaluation target stage by the timing of the second evaluation target stage, which is the evaluation target stage next to the first evaluation target stage;
The information processing device according to claim 1 , wherein the presentation processing unit executes the presentation process of the evaluation result related to the first evaluation target stage by a timing of the second evaluation target stage. and a compensation determination unit that determines a compensation to be given to the user for operating the robot based on the evaluation result.
The information processing device according to claim 1 , wherein the presentation processing unit causes the presentation device to present the compensation determined by the compensation determination unit and a result of accumulating the compensation during the execution of the work. The information processing device according to any one of claims 1 to 4, wherein the detection processing unit detects, as the operation-related information, at least one of the operation command, an operation command generated for the robot to cause the robot to operate in accordance with the operation command, an actual operation result of the robot, a state of the environment surrounding the robot, a state of an object on which the robot acts, and a state of the environment surrounding the object. The information processing device according to any one of claims 1 to 5, wherein the evaluation processing unit compares and evaluates a plurality of evaluation target features, which are features for evaluating the action-related information, with criteria for the plurality of evaluation target features included in the action-related criteria, and evaluates the action-related information based on the evaluation results of the plurality of evaluation target features. The apparatus further includes a storage unit and a criterion determination unit,
The storage unit stores action-related criterion data including the action-related criterion and information on the evaluation result of the action-related information based on the action-related criterion;
The information processing device according to any one of claims 1 to 6, wherein the standard determination unit uses information on the first action-related reference data stored in the storage unit and the evaluation result based on the action-related standard included in the first action-related standard data to generate second action-related standard data by varying the action-related standard included in the first action-related standard data in accordance with the evaluation result . 8. The information processing device according to claim 7, wherein the standard determination unit extracts, from the information stored in the storage unit, information on the evaluation result that satisfies a specified condition and the first action-related reference data which is the action-related reference data used in the evaluation result, and generates the second action-related reference data using the extracted information on the evaluation result and the first action-related reference data . a learning data storage unit that stores information on the action-related criterion and information on the evaluation result of the action-related information based on the action-related criterion as machine learning data in the information processing device according to any one of claims 1 to 8;
A machine learning unit;
the machine learning unit performs machine learning using information of the evaluation result included in the data for machine learning as input data for machine learning and information of the action-related criterion included in the data for machine learning and used in the evaluation result as teacher data;
The machine learning unit receives, as input data, evaluation information corresponding to the evaluation result information, and receives, as output data, information on the action-related criterion corresponding to the evaluation information. An information processing device according to any one of claims 1 to 8;
The learning device according to claim 9,
The learning device is configured to receive input data from the operation device or the information processing device, and to output output data corresponding to the input data to the information processing device. An information processing device according to any one of claims 1 to 8;
The robot;
A robot controller for controlling an operation of the robot,
The information processing device is communicatively connected to the robot controller. An information processing system according to claim 10;
The robot;
A robot controller for controlling an operation of the robot,
The information processing device is communicatively connected to the robot controller. A method for evaluating an operation of a robot to cause the robot to execute a task, comprising the steps of:
executing, during the execution of the work, a detection process for detecting operation-related information, which is information related to an operation of the robot that operates in accordance with an operation command input by a user to an operation device;
executing an evaluation process for comparing and evaluating the motion-related information detected in the detection process with a motion-related criterion that is a criterion related to the motion of the robot, during the execution of the task, at least one of the timing of each evaluation target motion, which is the motion of the robot to be evaluated, and the timing of each evaluation target stage, which is a task stage to be evaluated in the task;
When the evaluation process is executed at a timing for each of the evaluation target actions, when the evaluation process is executed for the evaluation target action, a presentation process is executed during the execution of the work, in which an evaluation result for the evaluation target action that has been subjected to the evaluation process is presented on a presentation device that receives presentation of information while the user is operating the operation device;
When the evaluation process is executed at the timing of each evaluation target stage, when the evaluation process is executed for the evaluation target stage, a presentation process is executed during the execution of the work, the presentation process being for presenting, on the presentation device, an evaluation result for the evaluation target stage that has been subjected to the evaluation process;
One of the tasks includes a plurality of the evaluation target actions and a plurality of the evaluation target steps.
Evaluation method.

Images