JP7595914B2 - Server, its control method, program, and system - Google Patents

Description

This invention relates to a semi-autonomous robot system that involves remote operation by an operator.

Automation using robots is being promoted in production sites, logistics warehouses, etc.

In this type of automated system, it would be ideal if all robots operated autonomously. However, in reality, it is difficult to make all robots perform all operations autonomously due to hardware constraints and the need to respond to unexpected situations.

In recent years, therefore, semi-autonomous systems have been considered, in which most of the robot's movements are automated, but only in situations where the robot cannot handle them are supported by an operator who temporarily intervenes in the robot's movements via remote control.

For example, Literature 1 discloses a robot control method that switches the control of the robot from autonomous control to manual control if task execution fails.

JP 2016-068161 A

However, in previous technologies of this type, when an operator provides support, it was assumed that there was a one-to-one relationship between the operator and the robot, and there was no sufficient consideration given to cases where the relationship between the operator and the robot was 1:N or M:N (N and M are natural numbers of 2 or greater). As a result, in cases where multiple support requests were made simultaneously from multiple robots, there was a risk that it would be difficult for the operator to support the robot.

The present invention was made in consideration of the above-mentioned technical background, and its purpose is to provide a technology that allows an operator to easily respond to support requests even when multiple support requests are made from a robot.

The above technical problems can be solved by a server or the like having the following configuration:

In other words, the server according to the present invention is a server connected to a plurality of robot systems that execute predetermined tasks semi-autonomously and one or more operator terminals, and includes a support request receiving unit that receives support requests from the robot systems, a priority determination unit that determines the priority of the support requests when multiple support requests are received, and an information providing unit that provides information related to the support requests and information related to the priority to the operator terminal.

With this configuration, even if multiple support requests are received from a robot system that executes a task semi-autonomously, the operator can determine the priority order for the support requests, allowing the operator to easily handle the multiple support requests. Note that here, support means performing operations or instructions for the semi-autonomous robot with the aim of assisting it in executing a target action. Also, a support request means information that requests such assistance from the operator.

The priority determination unit may further include a required time calculation unit that calculates, for each robot system that has made the support request, a first total value that is a total value of the required times of subsequent tasks of a task executed by the robot system, and a required time-based priority determination unit that determines the priority order such that the priority order is higher as the first total value is larger.

This configuration allows operators to deal with tasks that have the greatest impact on the entire process first.

The priority determination unit may further include a second required time calculation unit that calculates a second total value for each robot system that has made the support request by adding the remaining required time of the task being executed to the total required time of the tasks subsequent to the task to be executed by the robot system, and a second required time-based priority determination unit that determines the priority order such that the priority order is higher as the second total value is larger.

This configuration allows the operator to deal with tasks that have the greatest impact on the entire process. In addition, the remaining time required for the task being executed is also taken into account, so if there are tasks that take a long time to execute, the priority can be determined more precisely.

The priority order determination unit may further include a third required time calculation unit that calculates a third total value for each robot system that has made the support request by adding the remaining required time of the currently executing task and the required time of any necessary support action to the total required time of the tasks subsequent to the task to be executed by the robot system, and a third required time-based priority order determination unit that determines the priority order such that the priority order is higher as the third total value is larger.

This configuration allows the operator to deal with tasks that have the greatest impact on the entire process. In addition, the remaining time required for the task being executed and the time required for necessary support operations are also taken into account, so when there are tasks that take a long time to execute or tasks that require time for support, the priority can be determined more precisely.

Any of the required times may be determined based on actual measurements obtained by operating the robot system.

This configuration allows the required time to be accurately estimated based on actual measurements.

The server may further include a support authority storage unit that stores, for each operator corresponding to each operator terminal, support authority for the robot system, the task, and/or some of the operations that constitute the task, and the information providing unit may provide information regarding the support request to the operator terminal based on the support authority.

With this configuration, support requests can be efficiently assigned to operator terminals based on whether or not the operator has support authority.

The server may further include a sum calculation unit that calculates the sum of the first sum, the second sum, or the third sum for each combination of the support request and the operator corresponding to the operator terminal, and the information providing unit may further provide information related to the support request to the operator terminal based on the combination that minimizes the sum.

This configuration allows for optimization of the entire system, minimizing downtime throughout the entire process.

The server may further include a support authority storage unit that stores, for each operator corresponding to each of the operator terminals, support authority for the robot system, the task, and/or some of the operations that constitute the task, and a sum calculation unit that calculates the sum of the first sum, the second sum, or the third sum for each combination of the support request and the operator corresponding to the operator terminal that has the support authority for the support request, and the information providing unit may further provide information regarding the support request to the operator terminal based on the combination that minimizes the sum.

With this configuration, even if the number of robot systems or operators increases, it is possible to prevent combination explosion while optimizing the system as a whole, minimizing downtime for the entire process.

The priority may be determined so that the older the support request is received from the robot system, the higher the priority.

This configuration allows appropriate priorities to be determined in a system where chronological handling is appropriate.

The server may include a priority information storage unit that stores predetermined priority information for the robot system, the tasks, and/or operations that constitute part of the tasks, and the priority order may be determined based on the priority information.

With this configuration, you can set in advance the tasks that you want to prioritize or not prioritize during operation.

The operator terminal may include a list display unit that displays a list of the connected robot systems, and a highlighting unit that highlights the robot system that is making the support request based on information related to the support request.

This configuration makes it possible to immediately identify the robot system that is requesting support.

The operator terminal may further include a sorting display unit that sorts and displays the robot systems that have requested support in order of priority based on the information regarding the priority.

With this configuration, the sorted display allows users to intuitively grasp which robot systems they need to support.

The server may further include an account information management unit that manages account information including support authority for the robot system, the task, and/or some of the operations that constitute the task, which is set for each operator of the operator terminal, and the operator terminal may further include a login processing unit that provides a login process that provides a match with the account information for the operator using the operator terminal.

With this configuration, different operators can use the same operator terminal with their own accounts.

The operator terminal may further include an authority-based display unit that displays, among the robot systems for which the support request is being made, the robot system for which the operator who performed the login process has the support authority in a manner that distinguishes it from other robot systems.

With this configuration, the logged-in operator can immediately understand which robot systems they can support based on their support authority.

The robotic system may include a mobile manipulator.

With this configuration, even if multiple support requests are received from a mobile manipulator that executes a task semi-autonomously, the priority order can be determined for the support requests, so the operator who operates the operator terminal can easily handle multiple support requests.

The present invention can also be conceived as a system. That is, the system according to the present invention is a system consisting of a plurality of robot systems that execute predetermined tasks semi-autonomously, one or more operator terminals, and a server connected to the robot systems and the operator terminal, and the server includes a support request receiving unit that receives a support request from the robot systems, a priority determination unit that determines the priority of the support requests when multiple support requests are received, and an information providing unit that provides information about the support requests and information about the priority to the operator terminal.

The present invention can also be conceived as a method. That is, the method according to the present invention is a control method for a plurality of robot systems that execute predetermined tasks semi-autonomously, one or a plurality of operator terminals, and a server connected thereto, the method including a support request receiving step of receiving a support request from the robot systems, a priority order determining step of determining the priority order of the support requests when a plurality of the support requests are received, and an information providing step of providing information on the support requests and information on the priority order to the operator terminal.
It is equipped with:

The present invention can also be thought of as a computer program. That is, the program according to the present invention is a control program for a plurality of robot systems that execute predetermined tasks semi-autonomously, one or more operator terminals, and a connected server, and includes a support request receiving step for receiving a support request from the robot systems, a priority determination step for determining the priority of the support requests when a plurality of the support requests are received, and an information providing step for providing information about the support requests and information about the priority to the operator terminal.

The present invention provides a system that allows an operator to easily respond to multiple support requests even when multiple support requests are received from a robot system that executes tasks semi-autonomously.

FIG. 1 is a diagram showing the overall configuration of the system. FIG. 2 is a functional block diagram of the system. FIG. 3 is a table showing the correspondence between the robot system ID and the task. FIG. 4 is a table showing the correspondence between tasks, actions which are unit operations constituting the tasks, and the required time for each action. FIG. 5 is a table showing the correspondence between operator IDs and the authorities given to the operators. FIG. 6 is a flowchart illustrating the operation of the robot system. FIG. 7 is a flowchart relating to the information gathering operation of the management server. FIG. 8 is a flowchart showing a process for providing information to the operator system. FIG. 9 is a flowchart showing the detailed operation of the priority order determination process. FIG. 10 is an arrow diagram of a series of tasks. FIG. 11 is a display example of the operation status list screen. FIG. 12 is an operational flowchart relating to the support process. FIG. 13 is an example of a display of the support screen.

The preferred embodiment of the present invention will be described in detail below with reference to the attached drawings.

1. First embodiment
As a first embodiment, an example of applying the present invention to a system in a logistics warehouse including a plurality of robot systems each performing a series of tasks, i.e., sorting items, followed by packing them into boxes and taping them up, will be described.

In this embodiment, the robot system is applied to a logistics warehouse system, but is not limited to such an application example, and can be applied to various other situations, such as factories. In this embodiment, the robot system is described as having a robot arm, but various hardware other than the robot arm can be used. The robot system does not need to be fixed, and for example, a mobile manipulator can be used. The tasks of the robot system are not limited to those described above, and can be applied to various other tasks.

(1.1 System Configuration)
1 is a diagram showing the overall configuration of a system 200 according to this embodiment. As is clear from the diagram, the system 200 includes a plurality of robot systems 1, a management server 3, and an operator system 5, which are connected to each other via a LAN (local area network) by wire or wirelessly. Note that, in this embodiment, the devices are described as being connected to each other via a LAN, but they may also be connected via the Internet.

Each robot system 1 is composed of a robot-side information processing device 11 that performs various information processing described below, a sensor 15 connected to the robot-side information processing device 11, and a multi-joint robot arm 17 connected to the robot-side information processing device 11.

In this embodiment, the robot information processing device 11, the sensor 15, and the robot arm 17 are described as constituting a single system while being physically separated from one another, but some or all of them may be configured as an integrated system.

The robot-side information processing device 11 has a control unit consisting of a CPU and other components that execute various programs, and a storage unit consisting of a ROM, RAM, hard disk, etc.

The sensor 15 is a three-dimensional camera fixed in the environment that acquires image information that is the basis of recognition. For example, when performing an item sorting task, the sensor captures an image of the inside of a box containing a large number of target products or other workpieces, and when performing a packing task, the sensor captures an image of the workpieces on the conveyor. The information acquired from the sensor is transmitted to the robot-side information processing device 11.

In this way, the three-dimensional camera serving as the sensor 15 is fixed in the environment, and as described below, by configuring it to be shared for both recognition and remote control, the same viewpoint can be used, resulting in a simple configuration and cost advantages. Note that the method of mounting the sensor 15 is not limited to this example of fixing it in the environment, and it is also possible to use a hand-eye camera mounted on a robot arm 17, etc., or to reuse an existing surveillance camera.

The articulated robot arm 17 has a gripper at its tip and can move freely. For example, when performing an item sorting task, it grips a workpiece and places it on a belt conveyor or the like. Also, when performing a boxing and tape-closing task, it grasps and moves the workpiece placed on the belt conveyor and tape-closes it.

As described below, the management server 3 appropriately collects information about the robot system 1 and information about the operator system 5. In addition, it provides various information to the robot system 1 and the operator system 5 as necessary.

The operator system 5 includes an operator-side information processing device 51, a display device 52, and a haptic input device 53. The operator-side information processing device 51 is an information processing device that performs overall control of the operator system 5, and is, for example, a PC (personal computer). The display device 52 is a display device, and displays a list of robot systems 1 that have issued support requests, as described below.

The haptic input device 53 is provided with an angle sensor for each joint, and can control the six-axis posture of the hand of the robot arm 17 and the opening and closing of the gripper. As described below, the haptic input device 53 is operated by an operator and supports the robot arm 17 in executing a task in response to a support request. The haptic input device 53 may provide force feedback based on sensors provided in the robot arm 17.

The display device and input device are not limited to the examples in this embodiment, and various known devices can be used. For example, the display device may be a head-mounted display, and the input device may be an input device using a stick lever, a keyboard, etc.

In addition, in this embodiment, the operator-side information processing device 51, the display device 52, and the haptic input device 53 are described as constituting a single system while being physically separated from one another, but some or all of them may be configured as an integrated system.

Figure 2 is a functional block diagram of the system 200. As is clear from the figure, the robot system 1 exchanges information with the management server 3 and the operator system 5 via their respective transmitting and receiving units.

The robot system 1 is composed of a robot side information processing device 11, a sensor 15, and a robot arm 17.

The robot-side information processing device 11 includes a sensor information acquisition unit 111 that acquires detection information from the sensor 15, and a recognition processing unit 112 that performs a predetermined recognition process based on the sensor information. The recognition result by the recognition processing unit 112 is provided to a determination unit 114. The determination unit 114 determines whether or not to make a support request (described below) based on the recognition result, and provides the determination result to a motion generation unit 118. The motion generation unit 118 and motion command unit 119 issue motion commands to the robot arm 17.

When the support processing execution unit 121 receives support information via the transmission/reception unit 127, it executes various support processes according to the support information. For example, it controls the operation of the robot arm 17.

The management server 3 includes a support request detection unit 32 and an information acquisition processing unit 33. The support request detection unit 32 detects a support request sent from the robot system 1. Furthermore, the priority determination processing unit 35 determines the priority of the support requests when multiple support requests are received. The information provision processing unit 36 transmits information related to the support request and information related to the priority to the operator system 5 via the transmission/reception unit.

The information acquisition processing unit 33 acquires status information and the like of the robot system 1 and the operator system 5. The acquired information is stored in a storage unit (not shown) by the storage processing unit 38. In addition, the list control unit 37 manages predetermined list information (described later) based on the status information and the like.

Next, we will explain the table information that is stored in advance as configuration information in the management server 3. Note that although the following describes the table information as being stored in advance, the present invention is not limited to such a configuration, and the table information may be set during operation via the operator system 5, etc.

Figure 3 is a table showing the correspondence between robot system IDs and tasks, which are stored in advance in the memory unit of the management server 3. As is clear from the figure, each robot system 1 is given identification information such as "robot01" and "robot02", and the identification information of the task to be executed corresponding to the identification information is given, for example, "TASK_A" (task A) and "TASK_B" (task B). In other words, based on this table, "robot01" is set to execute "TASK_A".

Figure 4 is a table showing the correspondence between tasks, actions that are unit operations that make up the tasks, and the time required for each action ([s]), which are pre-stored in the storage unit of the management server 3. As is clear from the figure, "TASK_A" is composed of seven actions, from "ACTION_01" to "ACTION_07", and each action is associated with the required time shown in the figure.

Figure 5 is a table showing the correspondence between operator IDs, which are identification information of operators and which are stored in advance in the storage unit of the management server 3, and the authorities granted to the operators. As is clear from the figure, operator ID "Sakamoto" has been granted support authority for two actions, "TASK_A" and "TASK_B". Operator ID "Kawanishi" has only been granted support authority for "TASK_B". However, "Sakamoto" has not been granted support authority for "robot03".

In this embodiment, the support authority is defined in units of a task and a robot ID, but it may be defined in other ways. Therefore, for example, the support authority may be defined by other means, such as by action.

The system configuration in this embodiment is merely an example, and the number and configuration of each device connected to the network can be freely changed.

(1.2 System Operation)
Next, the operation of the system 200 will be described.

Figure 6 is a flowchart explaining the operation of the robot system 1. Generally speaking, the robot system 1 executes target actions and tasks semi-autonomously, but sends a support request to the management server 3 if it is unable to properly recognize something or fails to execute the target action.

As is clear from the figure, when processing begins, the sensor information acquisition unit 111 acquires sensor detection information from the sensor 15, which in this embodiment is image information obtained by photographing the inside of a box or on a conveyor (S1). The recognition processing unit 112 performs processing to recognize the workpiece or the like to be grasped by the robot arm 17 based on the sensor information (S2). Note that, although this embodiment is described as performing recognition based on image information, the present invention is not limited to this configuration. Therefore, for example, a configuration may be used in which three-dimensional information acquired by a three-dimensional camera is used instead of or together with an image.

After this recognition process, the determination unit 114 performs a process of determining whether support is necessary (S4). More specifically, if the recognition unit 112 has performed the recognition normally, for example, if the workpiece to be grasped can be recognized with sufficient recognition accuracy, the determination unit 114 determines that support is unnecessary, whereas if the recognition has not been performed normally, the determination unit 114 determines that support is necessary.

In this embodiment, the need for support is determined based on whether or not normal recognition is possible, but the present invention is not limited to this configuration, and various determination methods can be adopted. For example, the determination process may be performed based on a criterion such as whether or not a grasping solution for the object to be grasped can be calculated.

If it is determined as a result of the determination process that the recognition process has been performed normally and that support is not required (S4 NO), the movement generation unit 118 generates a target movement (S6). The target movement is a movement that the robot system 1 is expected to execute, such as a task movement set for the robot system 1. The movement command unit 119 then issues a movement command to the robot arm 17 to execute the generated target movement, and the target movement is executed (S7).

After this action, a process is performed to evaluate whether the target action was successful or not based on the sensor 15 information (S8). If it is determined as a result of this evaluation process that the target action was successful (S8 YES), an end determination is performed (S9), and if a predetermined end signal or the like is detected (S9 YES), the process ends. On the other hand, if an end signal is not detected, the series of processes is repeated again from the sensor information acquisition process (S1) (S9 NO). Note that if it is determined that the target action was unsuccessful (S8 NO), a support request transmission process, described below, is performed (S11).

Note that support means performing operations or instructions for the semi-autonomous robot to assist it in executing a target action. A support request means information requesting such assistance from an operator. Note that in this embodiment, a support request from the robot system 1 may be referred to as an alert.

If it is determined that support is required (S4 YES) or if the target action has failed (S8 NO), a support request is sent to the management server 3 (S11). After that, the robot-side information processing device 11 goes into a standby state until it receives support information (S12 NO).

When support information is received in this state (S12 YES), the support process execution unit 121 executes support execution processing according to the support information (S14). A predetermined end determination is then performed, and if an end signal is not detected, the system enters a reception waiting state again (S15 NO) and the series of processes is repeated. On the other hand, if an end signal is detected (S15 YES), the above-mentioned end determination process (S9) is performed.

Figure 7 is a flowchart of the information collection operation of the management server 3, which acquires various information from the robot system 1 and the operator system 5. As is clear from the figure, when processing begins, the information acquisition processing unit 33 executes processes in parallel to acquire four pieces of information: status information of the robot system 1 (S21), support request (S31), support completion signal (S41), and operator information (S51), and each process goes into a standby state until the corresponding information is acquired (S22 NO, S32 NO, S42 NO, S52 NO).

The status information of the robot system 1 is information that indicates the current operating state of the robot system 1, and includes, for example, information such as the task being executed, the action, the time elapsed since the start of the operation, the joint angle information of the robot arm 17, and the sensor information detected by the sensor 15. The support request is information that is sent from the robot system 1 by the above-mentioned process and requests support from an operator, and includes, for example, information such as the ID of the robot system 1 that made the support request and the time when the support request was generated.

The support completion signal is information sent from the operator system 5 when the support by the operator is completed, and includes, for example, the support completion time. The operator information is information about the operator associated by logging in to the operator side information processing device 51, and includes, for example, information such as the operator ID. The login process is performed by checking against the account information stored by the management server 3.

In this manner, in this embodiment, the operator system 5 is associated with an operator by logging in, so that the same operator system 5 can be shared by different operators.

When the storage processing unit 38 receives status information from the robot system 1 (S22 YES), it performs processing to store the status information (S23) and goes into standby mode again.

When the list control unit 37 receives a support request from a robot system 1 (S32 YES), it performs a process of adding the ID information of the robot system 1 that made the support request to a predetermined support request list (S33). At this time, the storage processing unit 38 also stores the ID information of the robot system 1 that made the support request in the storage unit. Thereafter, the system goes into standby mode again.

When the list control unit 37 receives a support completion signal (S42 YES), it performs a process of deleting the ID information of the robot system 1 for which support has been completed from the predetermined support request list (S43). At this time, the storage processing unit 38 also erases the ID information of the robot system 1 that has made the support request from the storage unit. Thereafter, the system goes into standby mode again.

Furthermore, when the storage processing unit 38 receives operator information from the operator system 5 (S52 YES), it performs a process of storing the operator information (S53) and goes into standby mode again.

In other words, the management server 3 collects, acquires, stores, and manages various information sent from the robot system 1 and the operator system 5.

Figure 8 is a flowchart related to the process of providing information to the operator system 5. The flowchart on the left side of the figure shows the operation of the operator system 5, and the flowchart on the right side of the figure shows the operation of the management server 3.

As is clear from the figure, when processing begins, the display processing unit 57 of the operator-side information processing device 51 makes a display request to the management server 3 (S61). After this display request, the operator-side information processing device 51 goes into a standby state for specific information (S62 NO). When the specific information is received (S62 YES), display processing according to that information is performed on the display unit 52 (S64). After that, the series of processes is repeated again (S61 to S64). In other words, the operator-side information processing device 51 is constantly performing update display processing.

When the support request detection unit 32 of the management server 3, which was in a standby state (S71 NO) until receiving a display request, receives a display request from the operator-side information processing device 51 (S71 YES), it performs a process of detecting from the support request list the robot system 1 for which a support request is being made at the time of reception (S72).

If the result of this detection process is that there is one support request (S73 NO), the information provision processing unit 36 performs a process of providing the operator system 5 with information for presenting the support request to the operator system 5 (S75). Note that this provided information may include any information collected by the management server 3.

On the other hand, if there are multiple support requests (S73 YES), the priority determination processing unit 35 performs a process of determining the priority between the support requests (S76), and after this process, performs a predetermined information provision process together with the priority (S78). Note that this provided information may include any information collected by the management server 3.

Figure 9 is a flowchart showing the detailed operation of the priority determination process. As is clear from the figure, when the process starts, variables are initialized and initialization processing is performed by referencing a specific robot system 1 (S81).

Then, a process is performed to calculate the total required time of the tasks subsequent to the task performed by the referenced robot system 1 (S83). More specifically, the total required time is calculated based on the above-mentioned table showing the correspondence between the robot system ID and the task, and the table showing the correspondence between the task and its required time ([s]).

After calculating the total required time for the subsequent tasks, it is determined whether the calculation process for this total has been completed for all robot systems 1 (S84), and if the calculation process has not yet been completed for all robot systems 1, the robot system 1 to be referenced is changed and the calculation process (S83) is performed again.

On the other hand, when the calculation process has been completed for all robot systems 1 (S84 YES), a process is performed to determine the priority of the robot system 1 that has requested support based on the total value (S86). In this embodiment, the larger the total value of the required time for the subsequent tasks, the higher the priority of the response is determined to be. The priority determination process then ends.

A specific example of priority determination will be described with reference to FIG. 10. FIG. 10 is an arrow diagram of a series of tasks according to this embodiment. As is clear from the diagram, when "robot01" executes the task of sorting items (task A), the sorted objects are placed on different conveyors and transported as three parallel processes. The items transported by the conveyors are packed into boxes by the following "robot02", "robot03" and "robot04", and the boxes are each sealed with tape. Each robot system 1 then places the taped-sealed boxes on the conveyor and transports them (task B).

In this case, it takes 11.6 seconds to execute task A, 30 seconds to execute task B, and 10 seconds for conveyor transport during that time. Note that these times are times measured by actually operating the robot system 1. With this configuration, the required time can be accurately estimated based on the actual measurements.

In this state, for example, when calculating the total required time for subsequent tasks for "robot01" in robot system 1, one series of three subsequent parallel processes takes a total of 50 seconds (=10+30+10), so the total required time is calculated as 150 seconds (=50 x 3). Similarly, when calculating the total required time for subsequent tasks for "robot02," "robot03," and "robot04," each is calculated as 10 seconds.

In other words, in the example shown in the figure, it can be seen that among the robot systems 1, "robot01" is the robot system 1 that will have a greater impact on the entire process if stopped, etc., than "robot02," "robot03," and "robot04." Therefore, the support request from "robot01," which requires a longer total time for subsequent tasks, should be given priority.

Figure 11 shows an example of an operation status list screen displayed on the display unit 52 of the operator system 5 as a result of the display process (S64). As is clear from the figure, an operation status list display unit 80 is provided in the center of the screen, and the ID information of the operator currently using the operator system 5, "Sakamoto", and the current time are displayed in the upper right corner of the screen.

The operation status list screen also displays, from the left, columns labeled "Operation status," "Time of trouble occurrence," "Robot ID," "Task," "Action," "Your assistance authority," "Assistant," and "Assistance status."

"Operation status" indicates whether the robot system 1 is operating normally or in an alert state. An alert state is a state in which the robot system 1 is requesting support from the operator because the target operation could not be executed or is unlikely to be executed.

"Time of trouble occurrence" is information regarding the date and time when the support request occurred. "Robot ID" is ID information that is identification information for the robot system 1 corresponding to the support request.

"Task" is a task assigned to each robot system 1. In the example shown in the figure, the robot is assigned either the "item sorting" task (task A) or the "boxing and taping" task (task B).

"Action" indicates which of the actions that make up the task is being performed. From the example in the figure, it can be seen that "robot01" requested support while performing the "recognize grip position of object in case" action of "ACTION_02". It can also be seen that "robot02" is normally performing the "open and store gripper" action of "ACTION_9".

"Your Assistance Authority" indicates whether the logged-in operator has support authority for each robot system 1 that is making a support request. From the figure, it can be seen that operator "Sakamoto" has support authority for tasks A and B, except for "robot03."

"Assistant" displays the ID of the operator providing support. For example, in the example shown in the figure, it can be seen that the operator "Kawanishi" is providing support for "robot07."

"Assistance status" indicates whether support is being provided in response to a support request. For example, in the example shown in the figure, it is shown that the operator "Kawanishi" is currently responding to a support request from "robot07."

In the example shown in the figure, support requests are displayed in order of priority based on the calculated priority, starting from the highest priority: "robot01," "robot04," "robot07," and "robot03." In other words, the robot systems 1 that are making support requests are sorted and displayed in a manner that makes them distinct from the robot systems 1 that are not making support requests.

The method of distinguishably displaying the robot system 1 making the support request is not limited to this example, and various other display methods can be adopted. For example, only the robot system 1 making the support request may be displayed, or the display related to the robot system 1 making the support request may be highlighted by coloring, etc.

Next, we will explain the support processing performed by the operator via the operator system 5.

Figure 12 is an operational flowchart for the support processing performed by an operator. The flowchart on the left side of the figure is an operational flowchart for the operator system 5, and the flowchart on the right side of the figure is an operational flowchart for the management server 3.

In the figure, when processing starts, the operator-side information processing device 51 performs processing to detect whether an input has been made to select one of the support requests displayed in the operation status list (S101). If no input is detected as a result of this detection processing (S102 NO), the input detection processing is repeated (S101, S102 NO).

If an input is detected as a result of the detection process (S102 YES), a process is performed in which the operating operator sends a support request requesting that the selected support request be dealt with (S104). After sending this support request, the operator-side information processing device 51 goes into a predetermined reception standby state (S105 NO).

Meanwhile, when the management server 3, which was in a standby state (S111 NO) until receiving a support request, receives the support request (S111 YES), it performs a process of setting the operator who sent the support request as the responsible operator in association with the support request (S112). As a result, for example, the corresponding operator ID and the status "In Service" are displayed in the "Assistant" and "Assistance Status" columns in FIG. 11. The management server 3 then sends a notification to the operator system 5 indicating that a support request has been set (S114).

When the operator system 5 receives the notification (S105 YES), it performs a process to display a support screen.

Figure 13 is an example of a support screen that allows an operator to respond to a support request. As is clear from the figure, a status display section 83 for the selected support request being handled is provided at the top center of the screen, and the ID information "Sakamoto" of the operator currently using the operator system 5 and the current time are displayed at the top right of the screen.

At the bottom left of the figure, a camera view display area 83 is provided that displays the camera view captured by the camera 15 of the robot system 1. In the example shown in the figure, the camera view display area 83 displays an image of workpieces 92 stacked in a box 91 that are the subject of an item sorting task. The workpieces 92 are cylindrical, and are each grasped by the robot arm 17 and sorted according to predetermined criteria.

In this embodiment, the camera view display area 83 is described as displaying a camera image, but the present invention is not limited to this configuration. Therefore, for example, a configuration may be used in which three-dimensional information acquired by a three-dimensional camera is presented in place of or together with the image.

At the center of the bottom of the figure, there is a CG image display unit 84 that displays a CG (Computer Graphics) image 841 generated based on posture information obtained from a joint angle sensor (not shown) provided on the robot arm 17 in its current posture. This image display unit allows the operator to instantly grasp the posture of the robot arm 17 currently in.

At the bottom right of the figure, there is a work start button 85 for starting the support process.

Returning to FIG. 12, when the work start button 85 is selected, the operator system 5, management server 3, and robot system 1 communicate appropriately to perform support processing (S108).

In this embodiment, the support process involves remotely controlling the robot arm 17 using the haptic input device 53 while checking the camera view display area 83, and changing the execution conditions of the target action. The action of changing the execution conditions of the target action is, for example, an action of changing the scene or environmental conditions, and more specifically, an action of changing the position and orientation of the workpiece 92 by using the robot arm 17 remotely controlled by the operator to shake the box or grasp and move the workpiece 92 inside the box.

In this embodiment, the content of the support process is an action to change the execution conditions of the target action, but is not limited to this. Therefore, for example, the target action itself may be alternatively executed by remote control.

After that, when the support is completed, a support completion signal is sent to the management server 3, and then the series of processes is repeated again from the input detection process (S101). As described above, the support completion signal is acquired by the management server 3, and the corresponding support request is deleted from the list (S41 to S43).

With this configuration, even if multiple support requests are received from a robot system that executes tasks semi-autonomously, the priority order for the support requests can be determined, so the operator does not need to consider which task should be given priority, and can easily handle multiple support requests.

In addition, with this configuration, operators can deal with tasks that have the greatest impact on the entire process based on the required time, allowing the entire system to be operated efficiently.

(2. Modifications)
In the above embodiment, the total required time of subsequent tasks is calculated to determine the priority order, but the present invention is not limited to such a configuration.

Therefore, for example, the priority order may be determined based on the total required time plus the required time of the remaining actions of the referenced robot system 1. With this configuration, the required time of the remaining actions of the task being executed is also taken into account, so that the priority order can be determined more precisely even when there are actions that take a long time to execute.

In addition, for example, the priority order may be determined based on the total required time plus the time expected to be required to provide support to the robot system 1. With this configuration, even in cases where support requires a certain amount of time, the priority order can be determined more precisely.

In the above embodiment, all robot systems 1 that have requested support are displayed to the operator, but the present invention is not limited to such a configuration. Therefore, for example, the combination of the robot system 1 that has requested support and the operator system 5 may be optimized.

For example, the time required for the entire process may be calculated for all combinations of the robot system 1 that has requested support and the operator system 5, and the combination that requires the shortest time may be recommended or provided to the operator. With this configuration, the entire system can be optimized, and downtime for the entire process can be minimized.

For example, the time required for the entire process may be calculated for all combinations of the robot system 1 that made the support request and the operator system 5 that has the authority to provide support for that support request, and the combination that results in the shortest time may be recommended or provided to the operator. With such a configuration, optimization can be achieved in the entire system, and downtime in the entire process can be minimized. Even if the system 200 becomes larger in scale and the number of robot systems 1 and operator systems 5 increases, an explosion of combinations can be prevented, optimizing the computational load and speeding up processing.

In the above embodiment, the priority order is determined based on the time required for the subsequent tasks, but the support requests may simply be displayed on the display terminal 52 in the chronological order in which they occurred. With this configuration, it is possible to provide optimal support for systems where processing in the order in which the requests occurred is desirable.

Although the embodiments of the present invention have been described above, the above embodiments merely show some of the application examples of the present invention, and the technical scope of the present invention is not intended to be limited to the specific configurations of the above embodiments. Furthermore, the above embodiments can be combined as appropriate to the extent that no contradictions arise.

The present invention can be used at least in industries that manufacture robot systems, etc.

REFERENCE SIGNS LIST 1 Robot system 11 Robot side information processing device 15 Sensor 17 Robot arm 3 Server 5 Operator system 51 Operator side information processing device 52 Display device 53 Haptic input device

Claims (15)

A server connected to a plurality of robot systems that semi-autonomously execute a predetermined task and one or a plurality of operator terminals,
a support request receiving unit that receives a support request from the robot system;
a priority order determination unit that determines a priority order of the support requests when a plurality of the support requests are received;
an information providing unit that provides information regarding the support request and information regarding the priority order to the operator terminal;
The priority order determination unit further
a required time calculation unit that calculates, for each of the robot systems that have made the support request, a first sum value that is a sum value of required times of tasks subsequent to a task to be executed by the robot system;
The server further comprises:
a sum calculation unit that calculates a sum of the first sum values for each combination of the support request and the operator corresponding to the operator terminal,
The information providing unit further provides information regarding the support request to the operator terminal based on the combination that minimizes the sum. A server connected to a plurality of robot systems that semi-autonomously execute a predetermined task and one or a plurality of operator terminals,
a support request receiving unit that receives a support request from the robot system;
a priority order determination unit that determines a priority order of the support requests when a plurality of the support requests are received;
an information providing unit that provides information regarding the support request and information regarding the priority order to the operator terminal;
The priority order determination unit further
a required time calculation unit that calculates, for each of the robot systems that have made the support request, a first sum value that is a sum value of required times of tasks subsequent to a task to be executed by the robot system;
The server further comprises:
a support authority storage unit that stores, for each operator corresponding to each of the operator terminals, support authority for the robot system, the task, and/or a part of operations constituting the task;
a sum calculation unit that calculates a sum of the first sum values for each combination of the support request and an operator corresponding to the operator terminal having the support authority for the support request,
The information providing unit further provides information regarding the support request to the operator terminal based on the combination that minimizes the sum. The server according to claim 1 or 2, wherein each of the required times is determined based on an actual measurement value measured by operating the robot system. The server further comprises:
a support authority storage unit configured to store, for each operator corresponding to each of the operator terminals, support authority for the robot system, the task, and/or a part of operations constituting the task;
The server according to claim 1 , wherein the information providing unit provides information regarding the support request to the operator terminal based on the support authority. The operator terminal includes:
a list display unit that displays a list of the connected robot systems;
The server according to claim 1 , further comprising: a highlighting unit configured to highlight the robot system making the support request based on information regarding the support request. The operator terminal further includes:
The server according to claim 5 , further comprising a sorting and displaying unit that sorts and displays the robot systems making the support requests in order of priority based on information regarding the priority. The server further comprises:
an account information management unit that manages account information including a support authority for the robot system, the task, and/or a part of operations constituting the task, which is set for each operator of the operator terminal;
The operator terminal further includes:
The server according to claim 1 , further comprising: a login processing unit that provides a login process for providing a matching with the account information to an operator using the operator terminal. The operator terminal further includes:
The server according to claim 7, further comprising an authority-based display unit that displays, among the robot systems making the support request, the robot systems for which the operator who performed the login process has the support authority in a manner that makes them distinct from other robot systems. The server according to any one of claims 1 to 8, wherein the robot system includes a mobile manipulator. A system including a plurality of robot systems that execute predetermined tasks semi-autonomously, one or more operator terminals, and a server connected to the robot systems and the operator terminals,
The server,
a support request receiving unit that receives a support request from the robot system;
a priority order determination unit that determines a priority order of the support requests when a plurality of the support requests are received;
an information providing unit that provides information regarding the support request and information regarding the priority order to the operator terminal;
The priority order determination unit further
a required time calculation unit that calculates, for each of the robot systems that have made the support request, a first sum value that is a sum value of required times of tasks subsequent to a task to be executed by the robot system;
The server further comprises:
a sum calculation unit that calculates a sum of the first sum values for each combination of the support request and the operator corresponding to the operator terminal,
The information providing unit further provides information regarding the support request to the operator terminal based on the combination that minimizes the sum. A system including a plurality of robot systems that execute predetermined tasks semi-autonomously, one or more operator terminals, and a server connected to the robot systems and the operator terminals,
a support request receiving unit that receives a support request from the robot system;
a priority order determination unit that determines a priority order of the support requests when a plurality of the support requests are received;
an information providing unit that provides information regarding the support request and information regarding the priority order to the operator terminal;
The priority order determination unit further
a required time calculation unit that calculates, for each of the robot systems that have made the support request, a first sum value that is a sum value of required times of tasks subsequent to a task to be executed by the robot system;
The server further comprises:
a support authority storage unit that stores, for each operator corresponding to each of the operator terminals, support authority for the robot system, the task, and/or a part of operations constituting the task;
a sum calculation unit that calculates a sum of the first sum values for each combination of the support request and an operator corresponding to the operator terminal having the support authority for the support request,
The information providing unit further provides information regarding the support request to the operator terminal based on the combination that minimizes the sum. A method for controlling a plurality of robot systems that semi-autonomously execute a predetermined task, one or more operator terminals, and a server connected thereto, comprising the steps of:
a support request receiving step of receiving a support request from the robot system;
a priority order determination step of determining a priority order of the support requests when a plurality of the support requests are received;
an information providing step of providing information regarding the support request and information regarding the priority to the operator terminal;
The priority determination step further comprises:
a required time calculation step of calculating, for each of the robot systems that have made the support request, a first sum value that is a sum value of required times of tasks subsequent to a task to be executed by the robot system;
The control method further comprises:
a sum calculation step of calculating a sum of the first sum values for each combination of the support request and the operator corresponding to the operator terminal;
The information providing step further comprises providing information regarding the support request to the operator terminal based on the combination that minimizes the sum. A method for controlling a plurality of robot systems that semi-autonomously execute a predetermined task, one or more operator terminals, and a server connected thereto, comprising the steps of:
a support request receiving step of receiving a support request from the robot system;
a priority order determination step of determining a priority order of the support requests when a plurality of the support requests are received;
an information providing step of providing information regarding the support request and information regarding the priority to the operator terminal;
The priority determination step further comprises:
a required time calculation step of calculating, for each of the robot systems that have made the support request, a first sum value that is a sum value of required times of tasks subsequent to a task to be executed by the robot system;
The control method further comprises:
a support authority storing step of storing, for each operator corresponding to each of the operator terminals, support authority for the robot system, the task, and/or a part of operations constituting the task;
a sum calculation step of calculating a sum of the first sum values for each combination of the support request and an operator corresponding to the operator terminal having the support authority for the support request,
The information providing step further comprises providing information regarding the support request to the operator terminal based on the combination that minimizes the sum. A control program for a server connected to a plurality of robot systems that semi-autonomously execute a predetermined task, and one or more operator terminals,
a support request receiving step of receiving a support request from the robot system;
a priority order determination step of determining a priority order of the support requests when a plurality of the support requests are received;
an information providing step of providing information regarding the support request and information regarding the priority to the operator terminal;
The priority determination step further comprises:
a required time calculation step of calculating, for each of the robot systems that have made the support request, a first sum value that is a sum value of required times of tasks subsequent to a task to be executed by the robot system;
The control program further comprises:
a sum calculation step of calculating a sum of the first sum values for each combination of the support request and the operator corresponding to the operator terminal;
a control program for providing information to the operator terminal based on the combination that minimizes the sum in the information providing step; A control program for a server connected to a plurality of robot systems that semi-autonomously execute a predetermined task, and one or more operator terminals,
a support request receiving step of receiving a support request from the robot system;
a priority order determination step of determining a priority order of the support requests when a plurality of the support requests are received;
an information providing step of providing information regarding the support request and information regarding the priority to the operator terminal;
The priority determination step further comprises:
a required time calculation step of calculating, for each of the robot systems that have made the support request, a first sum value that is a sum value of required times of tasks subsequent to a task to be executed by the robot system;
The control program further comprises:
a support authority storing step of storing, for each operator corresponding to each of the operator terminals, support authority for the robot system, the task, and/or a part of operations constituting the task;
a sum calculation step of calculating a sum of the first sum values for each combination of the support request and an operator corresponding to the operator terminal having the support authority for the support request,
The information providing step further includes providing information regarding the support request to the operator terminal based on the combination that minimizes the sum.

Images