JP7324935B2 - ROBOT CONTROL SYSTEM, ROBOT CONTROL METHOD, AND ROBOT CONTROL PROGRAM - Google Patents

Description

One aspect of the present disclosure relates to a robot control system, a robot control method, and a robot control program.

Patent Literature 1 describes a remote control device that transmits an operation command to a slave robot and receives an actual image captured by the slave robot during operation.

JP 2015-47666 A

It is desired to support the operation of robots.

A robot control system according to one aspect of the present disclosure includes a receiving unit that receives a frame image in which an imaging position changes according to the motion of a robot that operates based on a command, a command generation unit that generates a command, and a robot that responds to the command. and an operation image generation unit for generating an operation image by extracting a partial area from the frame image in response to a command so as to represent the delay of the operation of the frame image.

A robot control method according to one aspect of the present disclosure is a display method executed by a robot control system including at least one processor, and is a frame image in which a photographing range changes according to the motion of a robot that operates based on a command. , generating a command, and extracting a partial region from the frame image in response to the command to generate an operation image so as to represent the delay of the robot's motion with respect to the command.

A robot control program according to one aspect of the present disclosure includes steps of receiving a frame image in which an imaging range changes according to the motion of a robot that operates based on a command, generating a command, and controlling the motion of the robot in response to the command. and extracting a partial region from the frame image in response to the command to represent the delay to generate an operational image.

According to one aspect of the present disclosure, it is possible to assist the operation of a robot.

It is a figure which shows an example of application of a robot control system. 1 is a diagram illustrating an example of a hardware configuration used for a robot control system; FIG. It is a figure which shows an example of the functional structure of an operating device. It is a figure which shows an example of coordinate system conversion. 4 is a flow chart showing an example of the operation of the robot control system; FIG. 10 is a diagram showing an example of processing for generating an operation image; FIG. FIG. 10 is a diagram showing an example of changes in a frame image (first frame image); FIG. 8 is a diagram showing an example of changes in an operation image corresponding to FIG. 7;

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the description of the drawings, the same or equivalent elements are denoted by the same reference numerals, and overlapping descriptions are omitted.

[System configuration]
A robot control system 1 according to an embodiment is a computer system that assists a user in operating a robot 2 . The robot 2 operates based on the operation to perform various operations such as processing and assembly. To "support the operation of the robot" means to provide the user with an environment in which the robot 2 can be easily operated. A user is a person who operates a robot, and therefore can also be called an operator.

FIG. 1 is a diagram showing an example of application of the robot control system 1. As shown in FIG. In one example, the robot control system 1 comprises a manipulator 10 . The operating device 10 is connected to the robot computer 3 via a communication network N. As shown in FIG. The robot computer 3 is connected to the robot controller 4 , and the robot controller 4 is connected to the robot 2 .

In FIG. 1, one operating device 10 and one robot 2 are shown. However, the numbers of the operating devices 10 and the robots 2 are not limited, and at least one of the operating devices 10 and the robots 2 may be plural.

In one example, the operating device 10 is located in the operating environment Eu where the user is present, while the robot computer 3, the robot controller 4 and the robot 2 are located in the work environment Ew located remotely from the user. That is, the working environment Ew is a remote environment when viewed from the operating environment Eu. An object being "remotely located" means that the object exists in a position that the user cannot see with his own eyes. The distance between the operating environment Eu and the work environment Ew, that is, the distance between the user and the robot 2 is not limited at all. For example, the working environment Ew may be on the order of kilometers away from the operating environment Eu. Alternatively, the work environment Ew may be separated from the operating environment Eu on the order of several meters, for example, the space next to the room that is the operating environment Eu may be the work environment Ew.

Since the distance between the operating environment Eu and the work environment Ew is not limited, the configuration of the communication network N is also not limited. For example, the communication network N may include at least one of the Internet and an intranet. Alternatively, the communication network N may have a simple configuration realized by one communication cable.

When the user operates the robot 2 while viewing the image of the working environment Ew, a delay may occur between the user's operation and the motion of the robot 2 reflected in the image. That is, there is a possibility that the motion of the robot 2 corresponding to the user's operation will appear in the image slightly later than the user's operation. More specifically, a delay may occur between the command to the robot 2 by the user's operation and the motion of the robot 2 appearing in the image. In the present disclosure, delay means that the motion of the robot 2 reflected in the image is delayed with respect to the command to the robot 2 .

In one example, delays occur in the transmission of commands and responses over communication networks. For example, even if the user inputs a command using the operating device 10 at hand, a certain communication time is required for the command to be transmitted to the robot 2 . Furthermore, it takes a certain amount of communication time for information regarding the response of the robot 2 to the command to be transmitted from the work environment Ew to the operation device 10 . Therefore, the user cannot immediately confirm the response to the command. The robot control system 1 absorbs or reduces this delay and allows the user to intuitively grasp the operation of the robot in real time. Here, a command means a command input by a user's operation. A response is an output for that command. The type of information indicating the response is not limited. For example, the response may be information indicating the motion (position or orientation) of the robot 2 or may be indicated by a command output from the robot controller 4 to the robot 2 .

In one example, the delay effect is expressed as the difference between the integral of the values for the command and the integral of the values for the response. The value related to the command is, for example, a value related to the additional movement vector corresponding to the user's operation, that is, a value representing the movement amount or movement speed and movement direction. The value related to the response is, for example, a value related to an additional movement vector indicating an additional motion of the robot, that is, a value indicating the movement amount or movement speed and movement direction. In the present disclosure, a command-related value is also simply referred to as a "command value," and a response-related value is also simply referred to as a "response value."

The operating device 10 is a computer for operating the robot 2 . The operation device 10 includes an operation interface, which is an input interface for operating the robot 2, and a monitor capable of displaying an image representing the work environment Ew. The user can operate the robot 2 using the operation interface while checking the work environment Ew displayed on the monitor. The operation device 10 transmits to the robot computer 3 command values indicating user operations (commands) input from the operation interface.

The robot computer 3 is a computer that interprets command values input from the operating device 10 and outputs command signals corresponding to the command values to the robot controller 4 .

The robot controller 4 is a device that controls the robot 2 according to command signals input from the robot computer. In other words, the robot controller 4 controls the robot 2 based on commands sent from the operating device 10 in response to user operations. In one example, the command signal includes data for controlling the robot 2, such as a path indicating the trajectory of the robot 2. FIG. A trajectory of the robot 2 refers to the path of movement of the robot 2 or its components. For example, the trajectory of the robot 2 can be a tip trajectory. In one example, the robot controller 4 calculates joint angle target values (angle target values of each joint of the robot 2) for matching the position and orientation of the tip portion with the target values indicated by the command signals, and calculates the joint angle target values. to control the robot 2 according to

A robot 2 is a device or machine that works on behalf of a person. In one example, the robot 2 is a multi-axis serial link type vertical articulated robot. The robot 2 includes a manipulator 2a and an end effector 2b that is a tool attached to the tip of the manipulator 2a. The robot 2 can perform various processes using its end effector 2b. The robot 2 can freely change the position and posture of the end effector 2b within a predetermined range. The robot 2 may be a 6-axis vertical multi-joint robot, or a 7-axis vertical multi-joint robot in which one redundant axis is added to the 6 axes.

The motions of the robot 2 are recorded as responses of the robot 2 by various sensors mounted on the robot 2, and output as sensor data to the robot controller 4, for example. The robot controller 4 calculates a response value indicating the operation (response) of the robot 2 based on the sensor data, and transmits the response value to the operating device 10 via the robot computer 3 . Alternatively, the robot controller 4 may transmit a command signal to the robot 2 as a response value to the operating device 10 via the robot computer 3 . At least part of the sensor data from the sensors of the robot 2 may be output to the robot computer 3 without going through the robot controller 4 . In this case, the robot computer 3 may calculate the response value based on the sensor data.

In one example, a hand camera (first camera) 20 and an overhead camera (second camera) 30 are provided in the work environment Ew. The hand camera 20 is an imaging device arranged on the robot 2 (for example, at the tip of the robot 2). Therefore, at least one of the position and posture of the hand camera 20 changes according to the motion of the robot 2 . The hand camera 20 can photograph the end effector 2b or its surrounding environment. On the other hand, the bird's-eye view camera 30 is an imaging device installed so as to take a position and posture independent of the motion of the robot 2 . The bird's-eye view camera 30 is not attached to the robot 2, but is set at a place (for example, wall surface, ceiling, camera stand, etc.) that is not affected by the motion of the robot 2. FIG. The bird's-eye view camera 30 photographs the work environment Ew, for example, photographs at least a portion of the robot 2 . In one example, the hand camera 20 and the overhead camera 30 each output a frame image generated by photography to the robot computer 3 , and the robot computer 3 transmits the frame image to the operation device 10 . A frame image is an individual still image that constitutes a moving image, and is generated by an imaging device such as a camera. In the present disclosure, the frame image generated by the hand camera 20 is also referred to as a "first frame image", and the frame image generated by the overhead camera is also referred to as a "second frame image".

The computer functioning as the operating device 10 or the robot computer 3 is not limited. In one example, these computers may be composed of personal computers, or may be composed of large computers such as business servers.

FIG. 2 is a diagram showing an example of the hardware configuration of the computer 100 used for the operating device 10 or the robot computer 3. As shown in FIG. In this example, computer 100 comprises main body 110 , monitor 120 and input device 130 .

The main body 110 is composed of at least one computer. The main body 110 has a circuit 160 which has at least one processor 161 , a memory 162 , a storage 163 , an input/output port 164 and a communication port 165 . Storage 163 records programs for configuring each functional module of main body 110 . The storage 163 is a computer-readable recording medium such as a hard disk, nonvolatile semiconductor memory, magnetic disk, or optical disk. The memory 162 temporarily stores programs loaded from the storage 163, calculation results of the processor 161, and the like. The processor 161 configures each functional module by executing a program in cooperation with the memory 162 . The input/output port 164 inputs and outputs electrical signals to/from the monitor 120 or the input device 130 according to instructions from the processor 161 . The input/output port 164 may input/output electrical signals to/from other devices such as the robot controller 4 . Communication port 165 performs data communication with other devices via communication network N according to instructions from processor 161 .

Monitor 120 is a device for displaying information output from main body 110 . Monitor 120 is an example of a display unit in the present disclosure. The monitor 120 may be of any type as long as it can display graphics, and a specific example thereof is a liquid crystal panel. Input device 130 is a device for inputting information into main body 110 . The input device 130 may be of any type as long as it can input desired information, and specific examples thereof include operation interfaces such as a keypad, mouse, and operation controller.

The monitor 120 and the input device 130 may be integrated as a touch panel. For example, the main body 110, the monitor 120, and the input device 130 may be integrated like a tablet computer.

FIG. 3 is a diagram showing an example of the functional configuration of the operating device 10. As shown in FIG. In one example, the operation device 10 includes an image reception unit 11, a command generation unit 12, a data acquisition unit 13, a coordinate conversion unit 14, and an image generation unit 15 as functional modules. The image receiving unit 11 is a functional module that receives frame images from the work environment Ew. The command generator 12 is a functional module that generates commands (command values) to the robot 2 according to inputs from the operation interface. The data acquisition unit 13 is: It is a functional module that acquires data related to the operation or motion of the robot 2 from the operating environment Eu or the working environment Ew. For example, the data acquisition unit 13 acquires command values and response values. The coordinate transformation unit 14 is a functional module that executes coordinate system transformation required for robot control. The image generator 15 is a functional module that generates an operation image to be displayed on the monitor 120 from a frame image. An operation image is an image visually recognized by a user. The image generator 15 may generate an operation image from the first frame image, or may generate an operation image from the second frame image. In this embodiment, the image generator 15 generates an operation image from the first frame image.

In one example, the image generation unit 15 includes a direction information generation unit 16 and a delay information generation unit 17 as functional modules for generating operation images. The direction information generation unit 16 is a functional module that generates direction information indicating the motion direction of the robot 2 in the first frame image. The delay information generator 17 is a functional module that generates delay information indicating a delay in the operation of the robot 2 with respect to the command to the robot 2 .

[Coordinate system conversion]
The coordinate system conversion will be described with reference to FIG. FIG. 4 is a diagram showing an example of coordinate system transformation (coordinate transformation) in the robot control system 1. As shown in FIG. In the robot control system 1, the controller coordinate system Cu corresponds to the direction input from the operation interface _{(input device 130), the robot coordinate system Cr corresponds} to the posture of the robot 2, and the posture of the end effector 2b. A tool coordinate system _Ct , a first camera coordinate system _Cc1 corresponding to the orientation of the hand camera 20, and a second camera coordinate system _Cc2 corresponding to the orientation of the overhead camera 30 can exist.

Coordinate system transformations (in other words, position and orientation transformations between coordinate systems) can be represented by the R matrix. In FIG. 4, the matrix R ^u→c1 represents the transformation from the controller coordinate system C _u to the first camera coordinate system C _{c1 , and} the matrix R ^c1→u represents the transformation from the first camera coordinate system C _c1 to the controller coordinate system C _u . Represents a transform. The matrix Ru ^→r represents the transformation from the controller coordinate system _Cu to the robot coordinate system _{Cr , and} the matrix Rr ^→u represents the transformation from the robot coordinate system _Cr to the controller coordinate system _Cu . The matrix R ^u→c2 represents the transformation from the controller coordinate system C _u to the second camera coordinate system C _{c2 and} the matrix R ^c2→u represents the transformation from the second camera coordinate system C _c2 to the controller coordinate system C _u . The matrix Rr ^→t represents the transformation from the robot coordinate system _Cr to the tool coordinate system _{Ct , and} the matrix Rt ^→r represents the transformation from the tool coordinate system _Ct to the robot coordinate system _Cr . The matrix R ^r→c1 represents the transformation from the robot coordinate system C _r to the first camera coordinate system C _{c1 , and} the matrix R ^c1→r represents the transformation from the first camera coordinate system C _c1 to the robot coordinate system C _r . The matrix Rr ^→c2 represents the transformation from the robot coordinate system _Cr to the second camera coordinate system _{Cc2 , and} the matrix Rc2 ^→r represents the transformation from the second camera coordinate system _Cc2 to the robot coordinate system _Cr .

Since the hand camera 20 is arranged on the robot 2, the first camera coordinate system _Cc1 can be regarded as identical or nearly identical to the robot coordinate system _Cr or the tool coordinate system _Ct . In this case, conversion between these coordinate systems may be omitted.

A matrix R can be replaced by a combination of other matrices R. For example, the matrix R ^u→c1 is replaced by a combination of the matrices R ^u→c2 , R ^c2→r and R ^r→c1 . This replacement is expressed as R ^u→c1 =R ^r→c1 R ^c2→r R ^u→c2 .

User operations may be based on any coordinate system. For example, the user may operate the robot 2 while viewing a moving image composed of operation images or a moving image captured by the overhead camera 30 . The command generator 12 outputs commands based on the coordinate system. In this case, the coordinate transformation unit 14 executes transformation from the first camera coordinate system _Cc1 or the second camera coordinate system _Cc2 to the robot coordinate system _Cr to generate commands in the robot coordinate system _Cr . , and transmits this converted command to the robot computer 3 . In this embodiment, since the operation device 10 includes the coordinate conversion unit 14, coordinate system conversion can be executed without being affected by delay.

[Robot control method]
As an example of the robot control method according to the present disclosure, an example of a series of processing procedures executed by the robot control system 1 will be described with reference to FIG. FIG. 5 is a flow chart showing an example of the operation of the robot control system 1 as a processing flow S1. That is, the robot control system 1 executes the processing flow S1. A processing flow S1 shows processing for processing one first frame image obtained from the hand camera 20 to generate one operation image. In response to the hand camera 20 generating successive first frame images over time, the robot control system 1 sequentially processes the first frame images to generate a series of operation images. Therefore, the processing flow S1 is repeatedly executed. A moving image consisting of a series of operation images is obtained by the iterative process.

In step S11, the image receiving section 11 receives the first frame image. In the work environment Ew, the hand camera 20 outputs the first frame image to the robot computer 3, and the robot computer 3 transmits the first frame image to the operation device 10 via the communication network N. This first frame image may show at least part of the end effector 2b. Since the position or posture of the hand camera 20 changes according to the motion of the robot 2 , the photographing position of the first frame image also changes according to the motion of the robot 2 . The imaging position refers to the position or range of the area captured and recorded by the first frame image (hand camera 20).

In step S12, the data acquisition unit 13 acquires command values and response values. The data acquisition unit 13 acquires command values generated by the command generation unit 12 and receives response values sent from the robot computer. As described above, the response value is not limited. For example, the response value may be information indicating the motion (position or posture) of the robot 2 or a command value output from the robot controller 4 to the robot 2 .

In step S13, the image generator 15 generates an operation image based on these acquired data. An example of the generation process will be described with reference to FIG. FIG. 6 is a flowchart showing an example of processing for generating an operation image.

In step S131, the coordinate conversion unit 14 sets a coordinate system for generating an operation image. In one example, the coordinate transformation unit 14 sets a coordinate system associated with the hand camera 20 that generates the first frame image for generating the operation image. The coordinate transformation unit 14 may set the first camera coordinate system _Cc1 as the coordinate system related to the hand camera 20 . If the first camera coordinate system _Cc1 can be regarded as identical or nearly identical to the robot coordinate system _Cr or the tool coordinate system _Ct , the coordinate transformation unit 14 converts the robot coordinate system _Cr or the tool coordinate system C _t may be set as the coordinate system associated with the hand camera 20 .

The coordinate transformation unit 14 performs transformation from another coordinate system to the coordinate system associated with the hand camera 20 for its setting as necessary. As an example, when a user operation is performed based on a moving image (that is, a series of second frame images) from the bird's-eye view camera 30, the coordinate transformation unit 14 responds to the command generated by the command generation unit 12. The coordinate system (e.g., robot coordinate system C _r or second camera coordinate system C _c2 ) is compared to the coordinate system associated with the commands associated with hand camera 20 (e.g., first camera coordinate system C _c1 , robot coordinate system C r , robot coordinate system C _r , Alternatively, it may be converted into the tool coordinate system C _t ). The coordinate transformation unit 14 sets the transformed coordinate system for generating an operation image.

In step S132, the direction information generator 16 generates direction information. The direction information generation unit 16 identifies the motion direction of the robot 2 in the first frame image according to the coordinate system set by the coordinate transformation unit 14, and generates direction information indicating the motion direction. If the motion direction of the robot 2 in the three-dimensional space is replaced by the motion direction of the robot 2 on the first frame image, the motion directions on the first frame image are horizontal, vertical, depth, and rotational directions. defined by a combination of at least one direction of For example, the direction information generator 16 generates direction information indicating at least one of the horizontal direction and the vertical direction as the motion direction. Alternatively, the direction information generator 16 generates direction information representing at least the depth direction as the motion direction. Alternatively, the direction information generator 16 generates direction information representing at least the rotation direction as the motion direction.

In step S133, the delay information generator 17 generates delay information. The delay information generator 17 generates delay information based on the command generated by the command generator 12 and the response of the robot 2 obtained from the robot computer 3 . More specifically, the delay information generation unit 17 generates a command value output from the command generation unit 12 to the robot computer 3 (in other words, the robot controller 4) and the operation (response) of the robot 2 to the command value. Delay information is generated based on the difference from the indicated response value. For example, the delay information generator 17 may generate delay information indicating the difference. As an example, the delay information generator 17 may use the difference between the integral value of the command value and the integral value of the response value as the difference between the command value and the response value.

In step S<b>134 , the image generator 15 extracts a partial area from the first frame image according to the command generated by the command generator 12 . A partial area refers to a partial area of the first frame image. The image generator 15 sets a partial area corresponding to the command according to the coordinate system converted by the coordinate converter 14 . More specifically, the image generation unit 15 sets at least one of the position and size of the partial area so that the partial area represents the delay in the robot's motion with respect to the command. Then, the image generator 15 trims the first frame image based on the setting to extract a partial area. For example, the image generator 15 trims the entire peripheral area of the first frame image based on at least one of the set position and size, and extracts the remaining area (inner area) as a partial area. The full peripheral area means an area including the entire outer edge. A maximum width of the area to be removed by trimming may be set in advance. The removed regions are not used in the operational image and therefore are not visible to the user.

In one example, the image generator 15 extracts the partial area so that at least part of the end effector 2b is shown in the partial area (in other words, at least part of the end effector 2b is included in the partial area).

In step S135, the image generator 15 superimposes a mark representing the delay on the partial area to generate an operation image. In the present disclosure, a mark refers to a display that allows a person to visually recognize some information. A “delay mark” is a display for allowing the user to visually recognize the delay. In one example, the image generator 15 superimposes the reference mark and the virtual mark on the partial area as marks representing delay.

The reference mark is a mark whose position is fixed in the frame image (more specifically, the first frame image) and which represents the reference point of the robot 2 . "The position is fixed in the frame image (first frame image)" means that the position of the reference mark in the frame image (first frame image) is predetermined and does not change. The “robot reference point” refers to a point grasped as a reference of the robot 2 when the robot 2 is viewed through a frame image (first frame image). That is, the “robot reference point” is the robot reference point in the frame image (first frame image). In one example, the image generator 15 sets the center of the first frame image as the position of the reference mark.

A virtual mark is a mark whose position is fixed in the operational image and which represents a delay. "The position is fixed in the operational image" means that the position of the virtual mark in the operational image is predetermined and does not change. In one example, the image generator 15 sets the center of the partial area as the position of the virtual mark. Alternatively, the image generator 15 may set a virtual mark at the point of action of the end effector 2b or at a position related to the point of action. By displaying a virtual mark at a position related to the point of action, robot operation becomes easier (improved operability).

A specific method of expressing the mark is not limited, and the mark may be designed according to any policy. For example, the mark may be represented by a dot, cross, arrow, or bar, or by a shadow (silhouette) of the end effector reflected in the partial area. The representation method may be the same or different between the fiducial mark and the virtual mark. The shape and dimensions may be the same between the fiducial mark and the virtual mark. By aligning the shape and size between the two types of marks, the presence or absence of delay can be expressed more clearly, which can contribute to the improvement of operability.

To set the position of the reference mark on the partial area, the image generator 15 sets at least one of the position and size of the partial area with respect to the first frame image. This setting is either a process of shifting the position of the partial area extracted from the first frame image from the original position, a process of enlarging or reducing the size of the partial area from the original size, or changing both the position and the size. This is the process of changing. The image generation unit 15 sets at least one of the position and size of the partial area with respect to the first frame image according to at least one of the direction information and the delay information.

When the direction information indicates at least one of the horizontal direction and the vertical direction on the first frame image as the motion direction, the image generation unit 15 aligns the partial area of the first frame image according to the direction information. to set the position of the partial area. When the direction information indicates the depth direction on the first frame image as the movement direction, the image generation unit 15 scales the partial area of the first frame image according to the direction information, and determines the size of the partial area. set. When the direction information indicates the direction of rotation on the first frame image as the movement direction, the image generation unit 15 rotates and moves the partial region with respect to the first frame image in accordance with the direction information, thereby moving the partial region. Set position.

The image generating unit 15 determines at least one of the amount of shifting the position of the partial area with respect to the first frame image (shift amount) and the amount of changing the size of the partial area with respect to the first frame image (scaling amount). , the amount may be set according to the delay information. That is, the image generating section 15 may set at least one of the position and size of the partial area with respect to the first frame image according to the delay information.

In one example, the image generator 15 sets the position or size of the partial area based on the amount of shift according to the delay information. Specifically, the image generator 15 converts the difference indicated by the delay information (that is, the difference between the command value and the response value) into the number of pixels of the first frame image. Assuming that the coefficient for the conversion is called a gain, the image generation unit 15 obtains the product of the difference and a given gain as the shift amount. Then, the image generator 15 sets the position or size of the partial area according to the shift amount.

Alternatively, the image generator 15 may set a fixed value for at least one of the shift amount and the scaling amount. For example, the image generation unit 15 may set the maximum width of the area to be removed when extracting the partial area from the frame image as at least one of the amount of shift and the amount of scaling. The image generation unit 15 sets the shift amount or scaling amount to a given maximum change amount if the shift amount or scaling amount becomes larger than its maximum width, otherwise sets the shift amount or scaling amount to a given maximum change amount. may be set according to the shift amount.

The image generation unit 15 superimposes the reference mark and the virtual mark on the partial area whose position or size is set for the first frame image. That is, the image generator 15 draws the reference mark and the virtual mark on the partial area. Through this processing, the image generator 15 generates an operation image so as to represent the delay. Since the position of the reference mark is fixed in the first frame image as described above, the position of the reference mark on the partial area changes according to at least one of the position and size of the partial area with respect to the first frame image. In one example, the greater the delay, the greater the amount of shift or scaling of the subregion with respect to the first frame image. If there is no delay, the image generator 15 superimposes the virtual mark and the reference mark on the subregion so that these marks overlap. In this case, it is not essential to completely match the virtual mark and the reference mark. The positions may be slightly shifted, or the shape or size of one mark may be different from the shape or size of the other mark. Here, "when there is no delay" means that the difference between the command value and the response value is so small that the user does not perceive the effect of the delay or can ignore the effect. For example, "when there is no delay" means that the difference is 0 or less than or equal to a predetermined threshold.

Returning to FIG. 5, in step S14, the image generator 15 displays the operation image on the monitor 120. FIG. The user can grasp the degree of delay at this point by visually recognizing the marks (for example, reference marks and virtual marks) on the operation image.

An example of generating an operation image will be described with reference to FIGS. 7 and 8. FIG. FIG. 7 is a diagram showing an example of changes in the first frame image. FIG. 8 is a diagram showing an example of changes in the operation image, and corresponds to FIG.

FIG. 7 shows a scene in which the user moves the end effector 2b along the direction Ra in the first frame image in order to bring the end effector 2b closer to the workpiece W. FIG. More specifically, assume that the user moves the end effector 2b along the direction Ra in situation 210a and stops the operation in situation 210b. In response to this operation, the image receiving unit 11 first receives the first frame image 201 showing the situation 210a, and then receives the first frame image 202 and the first frame image 203 both showing the situation 210b in this order. Suppose.

Processing of the first frame image 201 will be described with reference to FIG. The direction information generation unit 16 generates direction information indicating the movement direction of the robot in the first frame image 201, that is, the direction Ra, according to the coordinate system set by the coordinate conversion unit 14. FIG. In this example, direction Ra is assumed to be a mixture of horizontal and vertical directions in first frame image 201 . The delay information generator 17 generates delay information based on the command for moving the end effector 2b along the direction Ra and the response of the robot 2. FIG. The image generator 15 extracts the partial area 301 from the first frame image 201 according to the command. The image generator 15 sets the position of the partial area 301 with respect to the first frame image 201 by translating the partial area 301 with respect to the first frame image 201 . The translation is indicated by vector 221 in FIG. The image generator 15 superimposes the reference mark 310 and the virtual mark 320 on the partial area 301 to generate an operation image 401 showing the situation 210a. In the example of FIG. 8, the image generator 15 superimposes the reference mark 310 on the center of the first frame image, and superimposes the virtual mark 320 on the center of the partial area. The delay is expressed as the difference between the position of fiducial mark 310 and the position of virtual mark 320 . The difference is represented by the direction and magnitude of displacement of the partial area 301 with respect to the first frame image 201 (that is, the vector 221).

Next, processing of the first frame image 202 will be described. As in the case of the first frame image 201, the direction information generator 16 generates direction information indicating the direction Ra, and the delay information generator 17 generates delay information. The image generator 15 sets the position of the partial area 302 with respect to the first frame image 202 according to the direction information and the delay information. In this case also, the image generator 15 sets the position of the partial area 302 with respect to the first frame image 202 by translating the partial area 302 with respect to the first frame image 202 . The image generator 15 superimposes the reference mark 310 and the virtual mark 320 on the partial area 302 to generate an operation image 402 showing the situation 210b. The smaller the delay, the smaller the difference between the position of fiducial mark 310 and the position of virtual mark 320 . In the example of FIG. 8, the reduction in the difference is illustrated by vector 222 representing translation of subregion 302 .

When the delay is eliminated (ie, when the difference between the command value and the response value is 0 or less than or equal to a predetermined threshold), fiducial mark 310 and virtual mark 320 overlap. In the example of FIG. 8, the image generator 15 does not change the position and size of the partial area 303 extracted from the first frame image 203 as a result. Accordingly, the image generation unit 15 superimposes the reference mark 310 and the virtual mark 320 on the partial area 303 so as to overlap these marks, thereby generating an operation image 403 representing the situation 210b.

As shown in FIG. 8, the operating device 10 displays the positional difference between the reference mark 310 and the virtual mark 320 in real time. Therefore, the user can grasp the degree of delay in real time, and as a result, can operate the robot 2 without much stress.

[program]
Each functional module of the operation device 10 is implemented by loading a robot control program into the processor 161 or memory 162 and causing the processor 161 to execute the program. The robot control program includes code for realizing each functional module of the operating device 10 . The processor 161 operates the input/output port 164 or the communication port 165 according to the robot control program to read and write data in the memory 162 or storage 163 . Each functional module of the operation device 10 is realized by such processing.

The robot control program may be provided after being fixedly recorded in a non-temporary recording medium such as a CD-ROM, DVD-ROM, or semiconductor memory. Alternatively, the robot control program may be provided over a communication network as a data signal superimposed on a carrier wave.

[effect]
As described above, the robot control system according to one aspect of the present disclosure includes a receiving unit that receives a frame image in which the shooting position changes according to the motion of the robot that operates based on a command, and a command generation unit that generates a command. and an operation image generation unit that extracts a partial area from the frame image in accordance with the command and generates an operation image so as to express the delay of the robot's motion in response to the command.

A robot control method according to one aspect of the present disclosure is a display method executed by a robot control system including at least one processor, and is a frame image in which a photographing range changes according to the motion of a robot that operates based on a command. , generating a command, and extracting a partial region from the frame image in response to the command to generate an operation image so as to represent the delay of the robot's motion with respect to the command.

A robot control program according to one aspect of the present disclosure includes steps of receiving a frame image in which an imaging range changes according to the motion of a robot that operates based on a command, generating a command, and controlling the motion of the robot in response to the command. and extracting a partial region from the frame image in response to the command to represent the delay to generate an operational image.

In this aspect, since only part of the frame image is used as the operation image instead of the entire frame image, the influence of the delay on the frame image is absorbed or reduced, and the user can intuitively grasp the movement of the robot. can be made This mechanism can support the operation of the robot. Since the user can accurately operate the robot while viewing the operation image, an improvement in work efficiency can be expected.

In the robot control system according to another aspect, the operational image generator may generate the operational image by superimposing a virtual mark for representing the delay fixed on the operational image on the partial area. . This virtual mark can represent the delay in such a way that the user can intuitively grasp the degree of delay.

In the robot control system according to another aspect, the operation image generator generates the operation image by superimposing a reference mark fixed in position on the frame image and representing a reference point of the robot on the partial area. may This reference mark can represent the delay in such a way that the user can intuitively grasp the extent of the delay.

In a robot control system according to another aspect, the operation image generator may superimpose the virtual mark and the reference mark on the partial area such that the virtual mark and the reference mark overlap if there is no delay. By representing the presence or absence of delay by the positional relationship between the virtual mark and the reference mark, the user can intuitively grasp the degree of delay.

A robot control system according to another aspect further includes a direction information generation unit that generates direction information representing the movement direction of the robot in a frame image based on a command, and the operation image generation unit determines the position of the partial region with respect to the frame image. and dimensions may be set according to the direction information. By setting a partial area according to the robot's movement direction, the delay is expressed in a form that associates the delay with the robot's movement, making it possible to provide the user with an environment that makes it easier to operate the robot. Become.

In the robot control system according to another aspect, the direction information generation unit generates direction information indicating at least one of the horizontal direction and the vertical direction as an operation direction, and the operation image generation unit generates The position of the partial area may be set by translating the partial area according to the direction information. In this case, delays are expressed in relation to robot motions that include movement along a horizontal or vertical direction, thus allowing easier robot manipulation along those directions.

In the robot control system according to another aspect, the direction information generation unit generates direction information representing at least the depth direction as the motion direction, and the operation image generation unit scales the partial region of the frame image according to the direction information. The size of the partial area may be set by setting the size of the partial area. In this case, since the delay is represented in association with the motion of the robot including movement along the depth direction, it becomes possible to facilitate the manipulation of the robot along that direction.

In the robot control system according to another aspect, the direction information generation unit generates direction information representing at least the rotation direction as the operation direction, and the operation image generation unit rotates the partial region with respect to the frame image according to the direction information. The position of the partial area may be set by moving it. In this case, the delay is expressed in relation to the robot's motion, including movement along the direction of rotation, making it easier to manipulate the robot along that direction.

A robot control system according to another aspect further includes a delay information generating unit that generates delay information corresponding to the delay based on the command and the response of the robot, and the operation image generating unit determines the position of the partial region with respect to the frame image. and dimensions may be set according to the delay information. By setting a partial area based on the command and the response of the robot, an operation image corresponding to the degree of delay is generated. Robot operation can be made easier by this operation image.

In the robot control system according to another aspect, the delay information generator generates a command value output by the command generator to a robot controller that controls the robot, and an operation of the robot based on the command value, as the command and the response of the robot. The delay information may be generated based on the difference from the indicated response value, and the operation image generator may set the position or size of the partial area based on the shift amount according to the delay information. By setting a partial area according to the shift amount, an operation image reflecting the degree of delay is generated. Robot operation can be made easier by this operation image.

A robot control system according to another aspect includes a robot controller that controls the robot based on a command, a receiving unit, a command generating unit, an operation image generating unit, and an operation image, which are connected to the robot controller via a communication network. An operation device having a display for displaying may be further provided, and the command generation unit may generate a command according to a user's operation. With this configuration, it is possible to absorb or reduce the effects of delays that occur in remote control of the robot, and allow the user to intuitively grasp the movement of the remotely located robot.

In a robot control system according to another aspect, a first camera is arranged on the front end side of the robot so that at least one of its position and orientation changes according to the motion of the robot, and generates a first frame image; a second camera that generates a second frame image that captures at least a part of the robot at a position and orientation independent of the second camera; a coordinate transforming unit that transforms a coordinate system corresponding to the command into a coordinate system corresponding to the command related to the first camera, the operation image generating unit transforming the coordinate system according to the coordinate system transformed by the coordinate transforming unit; A region may be set. With this configuration, even when an operation is performed according to an image from the camera independent of the motion of the robot, the operation image is generated by absorbing the difference in the coordinate system. Therefore, even in this case, the user can intuitively grasp the movement of the robot.

In another aspect of the robot control system, the robot may include an end effector, the frame image may depict the end effector, and the operation image generator may extract a subregion to include the end effector. Since the operation image always shows the end effector, the user can more easily operate the robot.

[Modification]
The above has been described in detail based on the embodiments of the present disclosure. However, the present disclosure is not limited to the above embodiments. Various modifications can be made to the present disclosure without departing from the gist thereof.

Although the operating device 10 includes the coordinate transformation unit 14 in the above embodiment, at least part of the coordinate system transformation may be performed by another device. For example, robot computer 3 or robot controller 4 may perform at least some of the coordinate system transformations.

The robot computer 3 is not an essential component, and in this case, the robot controller 4, the end camera 20, and the overhead camera 30 may be directly connected to the operating device 10 via the communication network N. good.

The hardware configuration of the system is not limited to the mode of realizing each functional module by executing the program. For example, at least part of the functional modules in the above embodiments may be configured by a logic circuit specialized for that function, or may be configured by an ASIC (Application Specific Integrated Circuit) integrated with the logic circuit. .

The processing steps of the method executed by at least one processor are not limited to the examples in the above embodiments. For example, some of the steps (processes) described above may be omitted, or the steps may be performed in a different order. Also, any two or more of the steps described above may be combined, and some of the steps may be modified or deleted. Alternatively, other steps may be performed in addition to the above steps.

When comparing two numerical values within a computer system or within a computer, either of the two criteria "greater than" and "greater than" may be used, and the two criteria "less than" and "less than" may be used. Either of the criteria may be used. Selection of such a criterion does not change the technical significance of the process of comparing two numerical values.

DESCRIPTION OF SYMBOLS 1... Robot control system 2... Robot 2b... End effector 3... Robot computer 4... Robot controller 10... Operating device 11... Image receiving part 12... Command generating part 13... Data acquisition part 14 Coordinate transformation unit 15 Image generation unit 16 Direction information generation unit 17 Delay information generation unit 20 Hand camera 30 Overhead camera 201 to 203 First frame image 301 to 303 Partial area , 310... reference mark, 320... virtual mark, 401 to 403... operation image, N... communication network.

Claims (13)

a receiving unit that receives a frame image in which the photographing position changes according to the motion of the robot that operates based on the command;
a command generator that generates the command;
an operation image generating unit for generating an operation image by extracting a partial area from the frame image according to the command so as to represent the delay in the operation of the robot with respect to the command;
with
The operation image generator generates a virtual mark whose position is fixed in the operation image and which represents the delay, and a reference mark whose position is fixed in the frame image and which represents a reference point of the robot. , superimposing on the partial area to generate the operation image;
robot control system. The operation image generator superimposes the virtual mark and the reference mark on the partial area so that the virtual mark and the reference mark overlap when there is no delay.
The robot control system according to claim 1 . further comprising a direction information generating unit that generates direction information representing an operating direction of the robot in the frame image based on the command;
The operation image generator sets at least one of the position and size of the partial area with respect to the frame image according to the direction information.
The robot control system according to claim 1 or 2 . The direction information generating unit generates the direction information representing at least one of a horizontal direction and a vertical direction as the motion direction,
The operation image generation unit sets the position of the partial area by translating the partial area with respect to the frame image according to the direction information.
The robot control system according to claim 3 . The direction information generating unit generates the direction information representing at least a depth direction as the motion direction,
The operation image generation unit scales the partial area with respect to the frame image according to the direction information, and sets the size of the partial area.
The robot control system according to claim 3 or 4 . The direction information generating unit generates the direction information representing at least a rotation direction as the motion direction,
The operation image generation unit rotates and moves the partial area with respect to the frame image according to the direction information, and sets the position of the partial area.
The robot control system according to any one of claims 3-5 . further comprising a delay information generating unit that generates delay information corresponding to the delay based on the command and the response of the robot;
The operation image generator sets at least one of the position and size of the partial area with respect to the frame image according to the delay information.
The robot control system according to any one of claims 1-6 . The delay information generation unit generates, as the command and the response of the robot, a command value output by the command generation unit to a robot controller that controls the robot, and a response value indicating the operation of the robot based on the command value. generating the delay information based on the difference between
The operation image generator sets the position or size of the partial area based on the shift amount according to the delay information.
The robot control system according to claim 7 . a robot controller that controls the robot based on the command;
an operating device connected to the robot controller via a communication network and having the receiving section, the command generating section, the operation image generating section, and a display section for displaying the operation image;
further comprising
The command generation unit generates the command according to a user operation.
The robot control system according to any one of claims 1-8 . a first camera that is arranged on the tip side of the robot so that at least one of its position and orientation changes according to the motion of the robot, and that generates a first frame image;
a second camera that generates a second frame image that captures at least a portion of the robot in a position and orientation that are independent of the motion of the robot;
Coordinate transformation for converting a coordinate system corresponding to the command generated by the command generation unit into a coordinate system corresponding to the command related to the first camera when a user operation based on the second frame image is performed. Department and
further comprising
The operation image generation unit sets the partial area according to the coordinate system transformed by the coordinate transformation unit.
The robot control system according to any one of claims 1-9 . the robot comprises an end effector;
the frame image depicts the end effector;
The operation image generator extracts the partial area so as to include the end effector.
The robot control system according to any one of claims 1-10 . A display method performed by a robot control system comprising at least one processor, comprising:
a step of receiving a frame image in which the imaging range changes according to the motion of the robot that operates based on the command;
generating said directive;
generating an operation image by extracting a partial area from the frame image according to the command so as to represent the delay of the robot's motion with respect to the command;
including
In the step of generating the operational image, a virtual mark fixed in position on the operational image and representing the delay, and a reference fixed in position on the frame image and representing a reference point of the robot. superimposing a mark on the partial area to generate the operation image;
Robot control method. a step of receiving a frame image in which the imaging range changes according to the motion of the robot that operates based on the command;
generating said directive;
generating an operation image by extracting a partial area from the frame image according to the command so as to represent the delay of the robot's motion with respect to the command;
on the computer , and
In the step of generating the operational image, a virtual mark fixed in position on the operational image and representing the delay, and a reference fixed in position on the frame image and representing a reference point of the robot. superimposing a mark on the partial area to generate the operation image;
robot control program.

Images