JP7537192B2 - Robot system and picking method - Google Patents

Description

The present invention relates to a robot system and a picking method.

Patent documents 1 to 3 disclose technology for recognizing pickable workpieces based on photographed images of bulk-stacked workpieces and then picking the workpieces.

Patent No. 5767464 JP 2017-42859 A Patent No. 5196156

However, with the above-mentioned conventional technology, it is difficult to reliably detect the position and orientation of a workpiece using an image sensor or depth sensor, especially in the case of small workpieces such as screws, metal workpieces, or shiny workpieces, making it difficult to reliably pick up a single workpiece. Picking failures can also occur due to failure to estimate the clearance from surrounding obstructions or to calculate the possibility of grasping. Furthermore, picking a workpiece often changes the arrangement of surrounding workpieces, so it also takes time to detect the position and orientation of the workpiece before each picking operation.

The present invention has been made in consideration of the above points, and aims to provide a robot system and a picking method that can pick a workpiece with high accuracy without having to detect the position and orientation of the workpiece in advance each time.

The first aspect of the disclosure is a robot system comprising: a robot mechanism having a gripping unit for gripping a workpiece; a detection unit for detecting a gripping state of the workpiece by the gripping unit; a recognition unit for recognizing the number of workpieces gripped by the gripping unit based on the detection result of the detection unit; and a control unit for controlling the movement of the robot mechanism such that the number of gripped workpieces becomes the specified number after the robot mechanism is caused to grip the workpieces from a placement location where a plurality of the workpieces are placed, if the number of the workpieces recognized by the recognition unit differs from a pre-specified number.

In the first aspect above, the specified number may be 1.

In the first aspect, the gripping unit may include a plurality of fingers and a drive unit that drives at least one of the fingers so that the position of at least one of the fingers changes when the workpiece is gripped by the plurality of fingers.

In the first aspect, the detection unit may be a pressure distribution sensor provided on the gripping surface of at least one of the fingers and configured to detect the pressure distribution of the gripping surface with which the workpiece comes into contact, and the recognition unit may be configured to recognize the number of the workpieces based on the pressure distribution detected by the pressure distribution sensor.

In the first aspect, the pressure distribution sensor may be provided on each of the gripping surfaces of at least two of the multiple fingers.

In the first aspect, the recognition unit may be configured to include a trained model that uses the pressure distribution as an input and the number of workpieces gripped by the gripping unit as an output.

In the first aspect, the trained model may include a neural network.

In the first aspect, the detection unit may be an image capture unit that captures an image of the workpiece being held by the fingers, and the recognition unit may be configured to recognize the number of workpieces based on the image captured by the image capture unit.

In the first aspect, the control unit may be configured to move at least one of the photographing unit and the gripping unit so that the workpiece held by the multiple fingers can be photographed at a position where it can be photographed.

In the first aspect described above, the detection unit is a force sensor that detects the force acting on the multiple fingers when the multiple fingers are gripping the workpiece, and the recognition unit may be configured to recognize the number of the gripped workpieces based on the total weight of the gripped workpieces, calculated as the increase due to gripping in the vertical component of the force detected by the force sensor.

In the first aspect, the number of workpieces may be recognized based on the total weight of the workpieces and the weight of each of the workpieces.

In the first aspect, the control unit may be configured to control the movement of the robot mechanism to re-grasp the workpiece from the placement location when the number of the workpieces recognized by the recognition unit is different from the specified number.

In the first aspect, the control unit may be configured to control the movement of the robot mechanism to re-grasp the workpiece from the placement location when the number of the workpieces recognized by the recognition unit is less than the specified number.

In the first aspect, the device may further include an external force mechanism that applies an external force to the workpiece while the workpiece is being held by the multiple fingers, and the control unit may be configured to control the external force mechanism to apply an external force to at least a portion of the workpiece being held by the multiple fingers to cause it to fall when the number of the workpieces recognized by the recognition unit is greater than the specified number.

The second aspect of the disclosure is a picking method that detects the gripping state of a workpiece by a gripping unit that grips the workpiece, recognizes the number of works gripped by the gripping unit based on the detection result of the gripping state, and controls the movement of the robot mechanism equipped with the gripping unit to grip the workpiece from a placement location where multiple workpieces are placed, and if the recognized number of works differs from a pre-specified number, controls the movement of the robot mechanism so that the number of gripped works becomes the specified number.

According to the present invention, workpieces can be picked with high accuracy without having to detect the position and orientation of the workpiece in advance each time.

FIG. 1 is a configuration diagram of a robot system. FIG. 2 is a functional block diagram of a control device. FIG. 2 is a block diagram showing a hardware configuration of a control device. 13 is a flowchart showing the flow of a picking process.

Below, an example of an embodiment of the present invention will be described with reference to the drawings. Note that the same reference symbols are used for identical or equivalent components and parts in each drawing. Also, the dimensional ratios in the drawings may be exaggerated for the convenience of explanation and may differ from the actual ratios.

FIG. 1 is a configuration diagram of a robot system 10 according to this embodiment. As shown in FIG. 1, the robot system 10 includes a robot mechanism 20, a control device 30, and an imaging unit 40. In this embodiment, the robot system 10 functions as a picking device that picks up a workpiece W.

The robot mechanism 20 includes a robot arm AR as a mechanical part that is the subject of motion control when performing a picking operation, and a robot hand H attached to the tip of the robot arm AR. The robot hand H grasps the workpiece W from a box 50, which is an example of a placement location on which multiple workpieces W are placed. The robot arm AR is an example of a gripping unit. Note that in this embodiment, the workpiece W will be described as a relatively small part such as a screw that allows the robot hand H to grasp multiple workpieces W, but the workpiece W is not limited to this.

The robot hand H has multiple fingers, and in this embodiment, as an example, two fingers F1 and F2, but the number of fingers is not limited to two. Also, in this embodiment, as an example, the fingers F1 and F2 are made of plate-shaped members, but the shape of the fingers F1 and F2 is not limited to this.

The robot hand H also includes a drive unit 22-1 that drives the finger F1 to change its position when the fingers F1 and F2 grip the workpiece W, and a drive unit 22-2 that drives the finger F2 to change its position when the fingers F1 and F2 grip the workpiece W. Note that, in this embodiment, a case where a drive unit is provided on both fingers F1 and F2 is described, but a configuration in which a drive unit is provided on either finger F1 or F2 may also be used.

Furthermore, a pressure distribution sensor 24-1 is provided on the gripping surface of finger F1 to detect the pressure distribution on the gripping surface with which the workpiece W comes into contact. Similarly, a pressure distribution sensor 24-2 is provided on the gripping surface of finger F2 to detect the pressure distribution on the gripping surface with which the workpiece W comes into contact. Here, pressure distribution sensors 24-1 and 24-2 are an example of a detection unit that detects the gripping state of the workpiece W by fingers F1 and F2.

In this embodiment, a pressure distribution sensor is provided on each of the gripping surfaces of fingers F1 and F2, but a pressure distribution sensor may be provided on the gripping surface of either finger F1 or F2.

As an example of the robot mechanism 20, a vertical multi-joint robot with six degrees of freedom, a horizontal multi-joint robot, etc. are used, but the degrees of freedom and type of the robot are not limited to these.

The control device 30 controls the robot mechanism 20. As shown in FIG. 2, the control device 30 functionally includes a recognition unit 32 and a control unit 34.

The recognition unit 32 recognizes the number of workpieces W held by the fingers F1 and F2 based on the detection results of the pressure distribution sensors 24-1 and 24-2, i.e., the pressure distribution. In this embodiment, as an example, the recognition unit 32 recognizes the number of workpieces W held by the fingers F1 and F2 using a trained model that uses the pressure distributions detected by the pressure distribution sensors 24-1 and 24-2 as inputs and the number of workpieces W as output, for example, a trained model using a neural network.

After causing the robot mechanism 20 to grasp the workpieces W from the box 50, if the number of workpieces recognized by the recognition unit 32 differs from a pre-specified number, the control unit 34 controls the movement of the robot mechanism 20 so that the number of grasped workpieces W becomes the specified number. Note that in this embodiment, a case where the specified number is 1 will be described. In other words, if the number of workpieces recognized by the recognition unit 32 is multiple, the control unit 34 controls the movement of the robot mechanism 20 so that the workpieces W fall until the number of workpieces recognized by the recognition unit 32 becomes one.

Specifically, the control unit 34 controls at least one of the drive units 22-1, 22-2 so that at least one of the fingers F1, F2 moves in order to change the gripping state of the workpiece W gripped by the fingers F1, F2, for example. In other words, at least one of the fingers F1, F2 is shifted so that the relative position of the fingers F1, F2 changes. This changes the gripping state of the workpiece W gripped by the fingers F1, F2, allowing the workpiece W to be dropped.

The photographing unit 40 is installed in a position where it can photograph the work W inside the box 50 from above the box 50, and outputs a photographed image of the work W inside the box 50 to the control device 30 in response to instructions from the control device 30.

Next, a block diagram showing the hardware configuration of the control device 30 is shown.

As shown in FIG. 3, the control device 30 has a CPU (Central Processing Unit) 30A, a ROM (Read Only Memory) 30B, a RAM (Random Access Memory) 30C, a storage 30D, an input unit 30E, a monitor 30F, an optical disk drive 30G, and a communication interface 30H. Each component is connected to each other so as to be able to communicate with each other via a bus 30I.

In this embodiment, a picking program is stored in storage 30D. CPU 30A is a central processing unit that executes various programs and controls each component. That is, CPU 30A reads the program from storage 30D and executes the program using RAM 30C as a working area. CPU 30A controls each of the components and performs various calculation processes according to the program recorded in storage 30D.

ROM 30B stores various programs and data. RAM 30C temporarily stores programs or data as a working area. Storage 30D is composed of a HDD (Hard Disk Drive) or SSD (Solid State Drive) and stores various programs including the operating system and various data.

The input unit 30E includes a keyboard 30E1 and pointing devices such as a mouse 30E2, and is used to perform various inputs. The monitor 30F is, for example, a liquid crystal display, and displays various information such as the gripping state of the workpiece W. The monitor 30F may also function as the input unit 30E by adopting a touch panel system. The optical disk drive device 30G reads data stored on various recording media (such as CD-ROMs or Blu-ray disks) and writes data to the recording media.

The communication interface 30H is an interface for communicating with other devices, and uses standards such as Ethernet (registered trademark), FDDI, or Wi-Fi (registered trademark).

The functional components of the control device 30 shown in FIG. 2 are realized by the CPU 30A reading out the picking program stored in the storage 30D, expanding it into the RAM 30C, and executing it.

Next, the operation of the robot system 10 will be described.

Figure 4 is a flowchart showing the flow of the picking process by the robot system 10. When the user operates the input unit 30E to instruct the execution of the picking process, the CPU 30A reads the picking program from the storage 30D, expands it into the RAM 30C, and executes it, thereby executing the picking process.

In step S100, the CPU 30A, as the control unit 34, controls the robot mechanism 20 so that the robot hand H grips the workpiece W in the box 50. Specifically, for example, the image capturing unit 40 is instructed to capture an image of the workpiece W in the box 50, and the captured image is analyzed to identify the location of the workpiece W. At this time, it is not necessary to identify the position or posture of the workpiece W to be gripped, and it is sufficient to only identify the location of the workpiece W. Then, the robot arm AR is controlled so that the robot hand H moves to the location of the workpiece W, and then the drive units 22-1 and 22-2 are controlled so that the workpiece W is gripped with the fingers F1 and F2. Alternatively, the fingers F1 and F2 may be closed at any position in the box 50 without capturing an image with the imaging unit 40, and as a result, the workpiece W may be gripped with a certain probability. In particular, when a large number of workpieces W remain in the box 50, the workpiece W can be gripped with a high probability without even identifying the location of the workpiece W in advance.

In step S102, the CPU 30A, as the recognition unit 32, acquires the pressure distribution of the gripping surfaces of the fingers F1 and F2 from the pressure distribution sensors 24-1 and 24-2, respectively. Then, the number of workpieces W gripped by the fingers F1 and F2 is recognized from the acquired pressure distribution of the gripping surfaces of the fingers F1 and F2.

In step S104, the CPU 30A, as the control unit 34, determines whether the number of workpieces W recognized in step S102 is 0. If the number of recognized workpieces W is not 0, that is, if at least one workpiece W is being grasped, the process proceeds to step S106. On the other hand, if the number of recognized workpieces is 0, the process returns to step S100 and the workpieces W are grasped again.

In step S106, the CPU 30A, as the control unit 34, determines whether the number of workpieces W recognized in step S102 is the specified number, i.e., whether the number of recognized workpieces W is one. If the number of recognized workpieces W is one, the process proceeds to step S108. On the other hand, if the number of recognized workpieces W is not one, i.e., if there are multiple workpieces W, the process proceeds to step S110.

In step S108, the CPU 30A, as the control unit 34, controls the robot mechanism 20 to move the workpiece W grasped by the robot hand H to a predetermined location and place it there.

In step S110, the CPU 30A, as the control unit 34, controls at least one of the drive units 22-1, 22-2 to shift at least one of the fingers F1, F2. For example, the finger to be shifted, the direction, and the amount of movement may be determined in advance, or the finger to be shifted, the direction, and the amount of movement may be determined based on the pressure distribution on the gripping surfaces of the fingers F1, F2 acquired in step S102. This changes the gripping state of the workpiece W held by the fingers F1, F2, making the workpiece W more likely to fall.

In step S112, the CPU 30A, as the control unit 34, determines whether all of the workpieces W in the box 50 have been picked. In other words, it determines whether the box 50 is empty. Specifically, for example, the image captured by the imaging unit 40 is analyzed to determine whether any workpieces W remain in the box 50. If no workpieces W remain in the box 50, this routine ends. On the other hand, if any workpieces W remain in the box 50, the process returns to step S100, and the same process as above is repeated until all of the workpieces W have been picked.

In this manner, in this embodiment, rather than detecting the position and orientation of the workpiece W to be grasped in advance and then grasping the workpiece W, the workpiece W is first grasped, and then at least one of the fingers F1 and F2 is shifted until the number of grasped workpieces W reaches the specified number. This allows the workpieces W to be picked with a high degree of accuracy, without having to detect the position and orientation of the workpiece W in advance each time.

(Variation 1)

Next, we will explain variant 1 of this embodiment.

When recognizing the number of gripped workpieces W, the number of gripped workpieces W may be recognized, for example, based on the contact area of the workpiece W on the gripping surface of each of the fingers F1 and F2. For example, the contact area of the workpiece W on the gripping surface of each of the fingers F1 and F2 is calculated based on each of the pressure distributions detected by the pressure distribution sensors 24-1 and 24-2. Then, the number of gripped workpieces W may be recognized, for example, using table data or a formula that indicates the correspondence between each contact area and the number of gripped workpieces. For example, when the workpieces W are spherical, the contact area formed by pressing one workpiece W against the elastic gripping surface is relatively stable, so it is possible to recognize the number of workpieces W in this way.

(Variation 2)

Next, we will explain variant 2 of this embodiment.

When recognizing the number of gripped workpieces W, for example, the image of the workpieces W gripped by the fingers F1 and F2 may be captured by the image capturing unit 40, which is an example of a detection unit, and the number of gripped workpieces W may be recognized based on the captured image captured by the image capturing unit 40. In this case, the control unit 34 may move the robot hand H so that the workpieces W gripped by the fingers F1 and F2 are captured at a position where they can be captured by the image capturing unit 40, that is, at a position where the workpieces W are not hidden by the fingers F1 and F2. The image capturing unit 40 may also be configured to further include a movement mechanism, and the image capturing unit 40 may be moved to the position of the robot hand H. Both the robot hand H and the image capturing unit 40 may also be moved.

(Variation 3)

Next, we will explain variant 3 of this embodiment.

When recognizing the number of gripped workpieces W, for example, a six-axis force sensor may be provided as an example of a detection unit that detects the force acting on the fingers F1 and F2 when the fingers F1 and F2 are gripping the workpieces W, and the recognition unit 32 may recognize the number of workpieces W based on the amount by which the vertical component of the force detected by the force sensor increases after gripping compared to before gripping, i.e., the total weight of the workpieces W. In this case, for example, the number of workpieces W may be recognized based on the total weight of the workpieces W and the weight of each workpiece W. In other words, the number of workpieces W may be calculated by dividing the total weight of the workpieces W calculated from the force detected by the force sensor by the weight of each workpiece W.

(Variation 4)

Next, we will explain variant 4 of this embodiment.

When the number of workpieces W recognized by the recognition unit 32 is different from the designated number, for example, when the number is less than the designated number, the control unit 34 may control the movement of the robot mechanism 20 so that the robot mechanism 20 re-grabs the workpieces W from the box 50. That is, the control unit 34 may control the robot mechanism 20 to open the fingers F1 and F2 on the box 50, so that the robot mechanism 20 returns the workpieces W that were once grasped onto the box 50, and then grasps the workpieces W in the box 50 again.

(Variation 5)

Next, we will explain variant 5 of this embodiment.

When the number of workpieces W recognized by the recognition unit 32 is greater than the specified number, the control unit 34 may control the robot mechanism 20 to apply an external force to at least a portion of the workpieces held by the fingers F1 and F2 to cause them to drop. For example, the control unit 34 may control the robot mechanism 20 to collide the workpiece W with a fixed rod-shaped jig to cause the workpiece W to drop. In addition, the control unit 34 may be further configured to include an external force mechanism that applies an external force to the workpiece W using a rod-shaped member, and the external force mechanism may be controlled to cause the rod-shaped member to collide with the workpiece W to cause the workpiece W to drop.

In this embodiment, the picking process executed by the CPU by reading the software (program) may be executed by various processors other than the CPU. Examples of processors in this case include PLDs (Programmable Logic Devices) such as FPGAs (Field-Programmable Gate Arrays) whose circuit configuration can be changed after manufacture, and dedicated electrical circuits such as ASICs (Application Specific Integrated Circuits) that are processors with circuit configurations designed specifically to execute specific processes. The picking process may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, a combination of a CPU and an FPGA, etc.). The hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor elements.

In addition, in each of the above embodiments, the picking program is described as being pre-stored (installed) in storage 30D or ROM 30B, but this is not limiting. The program may be provided in a form recorded on a recording medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network.

10 Robot system 20 Robot mechanism 22-1, 22-2 Driving unit 24-1, 24-2 Pressure distribution sensor 30 Control device 32 Recognition unit 34 Control unit 40 Photography unit 50 Box AR Robot arms F1, F2 Fingers H Robot hand

Claims (5)

A robot mechanism having a gripping unit that grips a workpiece;
A detection unit that detects a gripping state of the workpiece by the gripping unit;
A recognition unit that recognizes the number of workpieces gripped by the gripping unit based on the detection result of the detection unit;
a control unit that controls the motion of the robot mechanism so that the number of the grasped workpieces becomes the designated number when the number of the workpieces recognized by the recognition unit differs from a predetermined designated number after the robot mechanism has performed an operation of grasping the workpieces from a placement location where the workpieces are placed;
Equipped with
The gripping portion is
Multiple fingers and
A drive unit that drives at least one finger so that a position of at least one finger changes in a state in which the workpiece is gripped by the plurality of fingers;
Including,
The detection unit is a plurality of pressure distribution sensors provided on gripping surfaces of at least two of the plurality of fingers, and detects a pressure distribution on the gripping surface with which the workpiece comes into contact;
The recognition unit calculates a contact area of the workpiece on each gripping surface based on each of the pressure distributions detected by the plurality of pressure distribution sensors, and recognizes the number of the workpieces based on each of the calculated contact areas ;
When the number of the workpieces recognized by the recognition unit is greater than the designated number, the control unit controls the robot mechanism to collide at least a part of the workpieces held by the gripping unit against a fixed jig, thereby dropping the workpieces.
Robot system. the specified number is 1;
The robot system of claim 1 . The robot system according to claim 1 or 2, wherein the control unit controls the movement of the robot mechanism so as to re-grasp the workpiece from the placement location when the number of the workpieces recognized by the recognition unit is different from the designated number. The robot system according to claim 3 , wherein the control unit controls the movement of the robot mechanism so as to re-grasp the workpiece from the placement location when the number of the workpieces recognized by the recognition unit is less than the specified number. Detecting a gripping state of the workpiece by a gripping unit that grips the workpiece;
Recognizing the number of workpieces gripped by the gripping unit based on the detection result of the gripping state;
After the robot mechanism having the gripping unit is caused to perform an operation of gripping the workpieces from a placement location where a plurality of the workpieces are placed, if the number of the workpieces recognized is different from a designated number designated in advance, the motion of the robot mechanism is controlled so that the number of the gripped workpieces becomes the designated number;
The gripping portion is
Multiple fingers and
A drive unit that drives at least one finger so that a position of at least one finger changes in a state in which the workpiece is gripped by the plurality of fingers;
Including,
The detection is performed by a plurality of pressure distribution sensors provided on gripping surfaces of at least two of the plurality of fingers, the pressure distribution sensors detecting a pressure distribution on the gripping surfaces with which the workpiece comes into contact;
The recognition includes calculating a contact area of the workpiece on each gripping surface based on each of the pressure distributions detected by the plurality of pressure distribution sensors, and recognizing the number of the workpieces based on each of the calculated contact areas;
When the number of the workpieces recognized by the recognition is greater than the specified number, the control includes controlling the robot mechanism to collide at least a part of the workpieces held by the gripping unit against a fixed jig, thereby dropping the workpieces.
Picking method.