JP7790914B2 - Information processing method, information processing device, control method, robot system, article manufacturing method, program, and recording medium - Google Patents

Description

The present invention relates to robotics technology and image processing technology.

On production lines, tasks such as removing or assembling wire-like components such as harnesses are difficult processes, and traditionally, such work has often been done manually. Therefore, efforts are being made to automate these tasks using machinery that utilizes cameras and recognition algorithms.

Patent Document 1 discloses a system for automating such work, in which the position and orientation of the tip of a linear member are measured based on an image captured by a camera, and the measurement results are used to have a robot grasp the tip of the linear member.

Patent Document 2 also discloses that when multiple linear components are mixed and overlapping, the shape of the robot's hand is taken into consideration to select the linear components to be grasped by the hand so that the part and the hand do not interfere with each other and so that the hand does not grasp two linear components at the same time.

Japanese Patent Application Laid-Open No. 2020-112470 International Publication No. 2019/098074

However, when multiple linear components intersect with each other, it can be difficult to determine the intersecting state in an image of these multiple linear components, and linear components may be held by the robot in a state where they are entangled with other linear components.

The present invention aims to prevent linear members from being held by a robot in a state where they are entangled with other linear members.

A first aspect of the present disclosure is an information processing method characterized in that a processing unit uses a trained model to acquire an image in which shades of light and dark are binarized to enable detection of multiple linear members, and identifies a higher-order linear member from among at least two linear members that intersect with each other based on portions in the image where the linear members are discontinuous.

A second aspect of the present disclosure is an information processing device characterized in that a processing unit uses a trained model to acquire an image in which shades of light and dark are binarized to enable detection of multiple linear members, and identifies a higher-ranking linear member from among at least two linear members that intersect with each other based on portions in the image where the linear members are discontinuous.

A third aspect of the present disclosure is a method for controlling a robot, comprising : using a trained model to obtain an image in which a binary shade is used to detect multiple linear members; identifying a higher-order linear member from among at least two intersecting linear members based on a portion in the image in which the linear members are discontinuous; and controlling the robot to hold the higher-order linear member.

A fourth aspect of the present disclosure is a robot system including a robot and a processing unit, wherein the processing unit uses a trained model to acquire an image in which shades are binarized to enable detection of multiple linear members, identifies a higher-order linear member from among at least two linear members that intersect with each other based on portions in the image where the linear members are discontinuous, and controls the robot to hold the higher-order linear member.

This invention makes it possible to prevent linear members from being held by a robot in a state where they are entangled with other linear members.

FIG. 1 is an explanatory diagram illustrating a schematic configuration of a robot system according to an embodiment. FIG. 1 is an explanatory diagram of an image processing apparatus according to an embodiment. FIG. 2 is a block diagram of a computer system in the robot system according to the embodiment. FIG. 10 is an explanatory diagram of an assembly operation according to the embodiment. FIG. 10 is an explanatory diagram of an assembly operation in a comparative example. 1A is a flowchart illustrating a process in a preparation stage according to an embodiment, and FIG. 1B is a flowchart illustrating a process in an assembly work stage according to an embodiment. FIG. 10 is an explanatory diagram of a preparatory stage according to the embodiment. 1A is an explanatory diagram showing an example of an input image according to an embodiment, FIG. 1B is an explanatory diagram showing an example of an output image according to an embodiment, and FIG. 1C is an explanatory diagram showing an image of a reference example. 10A to 10C are explanatory diagrams of an assembly work stage according to the embodiment. 5A to 5C are diagrams illustrating a method for extracting intersection candidates according to an embodiment. 5A and 5B are diagrams illustrating a method for extracting intersection candidates according to an embodiment. 5A and 5B are diagrams illustrating an intersection detection process according to the embodiment. 5A to 5D are explanatory diagrams of an example of three-dimensional measurement according to an embodiment. 6A to 6F are diagrams illustrating an example of a control method according to an embodiment.

Embodiments of the present invention will be described in detail below with reference to the drawings. FIG. 1 is an explanatory diagram showing the schematic configuration of a robot system 10 according to an embodiment. The robot system 10 includes a robot 100, an image processing device 200, a robot controller 300, which is an example of a control device, and an imaging device 400, which is an example of an imaging unit. The robot 100 is an industrial robot that is placed on a production line and used to manufacture an assembly W20, which is an example of an article. The assembly W20 is, for example, a device itself or a final or intermediate product of a component of that device. Examples of devices include electronic devices, electrical devices, and optical devices.

The robot 100 is a manipulator. The robot 100 is fixed to a stand (not shown). An intermediate product W10 is arranged around the robot 100. The intermediate product W10 has two linear members W1 and W2, which are examples of multiple linear members, and a main body W0 to which the linear members W1 and W2 are connected. The linear member W1 is an example of a first linear member, and the linear member W2 is an example of a second linear member. Each linear member W1 and W2 is a flexible linear member, and examples include electric wires such as signal lines and power lines, optical fibers, cables, and wire harnesses. The tip of each linear member W1 and W2 is a free end, and the base end of each linear member W1 and W2 is a fixed end fixed to the main body W0. Therefore, by holding specific portions of each linear member W1, W2 and operating, the robot 100 can determine the position and orientation of the tip of each linear member W1, W2.

The tip of linear member W1 is, for example, made up of connector A1, and the tip of linear member W2 is, for example, made up of connector A2. Main body W0 has connector A3, which is the assembly object to which connector A1 is to be assembled, and connector A4, which is the assembly object to which connector A2 is to be assembled. Robot 100 assembles connector A1 to connector A3, and connector A2 to connector A4, thereby producing assembly W20.

Note that while the multiple linear members are explained as two linear members W1 and W2, this is not limiting and the multiple linear members may be three or more linear members. Furthermore, while the multiple linear members W1 and W2 are explained as having connectors A1 and A2, they do not necessarily have to have connectors. Furthermore, the multiple linear members may be the same or different in color, material, and wire diameter.

The robot 100 and robot controller 300 are connected by wire so that they can communicate with each other. The robot controller 300 and image processing device 200 are connected by wire so that they can communicate with each other. The imaging device 400 and image processing device 200 are connected by wire or wirelessly so that they can communicate with each other.

The robot 100 has a robot arm 101 and a robot hand 102, which is an example of an end effector, i.e., an example of a holding mechanism. The robot arm 101 is, for example, a vertically articulated robot arm. The robot hand 102 is supported by the robot arm 101. The robot hand 102 is attached to a predetermined portion of the robot arm 101, for example, the tip of the robot arm 101. The robot hand 102 is configured to be able to hold each of the linear members W1 and W2. Note that while the case where the holding mechanism is the robot hand 102 will be described, this is not limiting. For example, the holding mechanism may be a suction mechanism that can hold each of the linear members W1 and W2 by suctioning them. In this embodiment, the robot hand 102 is configured to be able to grasp each of the linear members W1 and W2.

The imaging device 400 is configured as a stereo camera including a camera 401, which is an example of a first imaging unit, and a camera 402, which is an example of a second imaging unit. Each camera 401, 402 is a digital camera that captures an image of a subject and outputs the captured image generated by the image capture to the image processing device 200. Here, the image refers to digital image data in this embodiment.

The imaging device 400 is fixed to a frame (not shown). Each camera 401, 402 is positioned so that it can capture an image of an area including multiple linear members W1, W2. In other words, the intermediate product W10 is transported by a transport mechanism (not shown) to a predetermined position around the robot 100 so that the multiple linear members W1, W2 are included in the field of view of the imaging device 400. Therefore, the imaging device 400 can capture images of the multiple linear members W1, W2.

Each of the cameras 401 and 402 captures an image of a plurality of linear members W1 and W2 as subjects, thereby generating two-dimensional images _I11 and _I12 as examples of first images. Each of the cameras 401 and 402 may be a color camera that generates a color image, or a monochrome camera that generates a monochrome image. While the imaging device 400 will be described with two cameras, it may also include one camera or three or more cameras.

The image processing device 200 sends an imaging command to each of the cameras 401 and 402 to cause each of the cameras 401 and 402 to capture an image. The image processing device 200 acquires each of the images _I11 and _I12 generated by each of the cameras 401 and 402 and performs image processing on each of the acquired images _I11 and _I12 . By performing image processing on each of the images _I11 and _I12 , the image processing device 200 calculates the position and orientation of a predetermined portion of each of the linear members W1 and W2 in the coordinate system of the robot 100. The robot controller 300 causes the robot 100 to selectively hold one of the predetermined portions of the linear members W1 and W2 based on the position and orientation of the predetermined portion of each of the linear members W1 and W2 calculated by the image processing device 200. The robot controller 300 controls the operation of the robot 100 so that the connectors of the linear members held by the robot 100 are assembled to the corresponding connectors. A plurality of linear members W1 and W2 are successively assembled to the main body W0 to produce an assembly W20.

In this embodiment, the image processing device 200 is configured as a computer. Figure 2 is an explanatory diagram of the image processing device 200 according to this embodiment. The image processing device 200 has a main body 201, a display 202 which is an example of a display device connected to the main body 201, and a keyboard 203 and a mouse 204 which are examples of input devices connected to the main body 201.

In this embodiment, the robot controller 300 shown in Figure 1 is configured as a computer. The robot controller 300 is configured to be able to control the movement of the robot 100, i.e., the posture of the robot 100.

Figure 3 is a block diagram of a computer system in a robot system 10 according to an embodiment. The main body 201 of the image processing device 200 includes a CPU (Central Processing Unit) 251, which is an example of a processor. The CPU 251 is an example of a processing unit. The main body 201 also includes a ROM (Read Only Memory) 252, a RAM (Random Access Memory) 253, and a HDD (Hard Disk Drive) 254 as storage units. The main body 201 also includes a recording disk drive 255 and an interface 256, which is an input/output interface. The CPU 251, ROM 252, RAM 253, HDD 254, recording disk drive 255, and interface 256 are connected via a bus to enable communication with each other.

ROM 252 stores basic programs related to the operation of the computer. RAM 253 is a storage device that temporarily stores various data, such as the results of calculations performed by CPU 251. HDD 254 records the results of calculations performed by CPU 251 and various data obtained from external sources, as well as a program 261 that causes CPU 251 to execute various processes, which will be described later. Program 261 is application software that enables CPU 251 to perform various processes, which will be described later. Therefore, by executing program 261 recorded on HDD 254, CPU 251 is able to execute image processing, which will be described later. Recording disk drive 255 can read various data, programs, etc., recorded on recording disk 262.

In this embodiment, the non-transitory computer-readable recording medium is HDD 254, and program 261 is recorded on HDD 254, but this is not limited to this. Program 261 may be recorded on any non-transitory computer-readable recording medium. Examples of recording media that can be used to provide program 261 to a computer include flexible disks, hard disks, optical disks, magneto-optical disks, magnetic tape, and non-volatile memory.

The robot controller 300 includes a CPU 351, which is an example of a processor. The CPU 351 is an example of a control unit. The robot controller 300 also includes a ROM 352, a RAM 353, and a HDD 354 as storage units. The robot controller 300 also includes a recording disk drive 355 and an interface 356, which is an input/output interface. The CPU 351, ROM 352, RAM 353, HDD 354, recording disk drive 355, and interface 356 are connected to each other via a bus so that they can communicate with each other.

ROM 352 stores basic programs related to the operation of the computer. RAM 353 is a storage device that temporarily stores various data, such as the results of calculations performed by CPU 351. HDD 354 records the results of calculations performed by CPU 351 and various data acquired from external sources, and also records (stores) program 361 for causing CPU 351 to execute various processes. Program 361 is application software that enables CPU 351 to perform various processes described below. Therefore, by executing program 361 recorded on HDD 354, CPU 351 performs control processing and is able to control the operation of robot 100 in Figure 1. Recording disk drive 355 can read various data, programs, etc. recorded on recording disk 362.

In this embodiment, the non-transitory computer-readable recording medium is HDD 354, and program 361 is recorded on HDD 354, but this is not limited to this. Program 361 may be recorded on any non-transitory computer-readable recording medium. Examples of recording media that can be used to provide program 361 to a computer include flexible disks, hard disks, optical disks, magneto-optical disks, magnetic tape, and non-volatile memory.

In this embodiment, image processing and control processing are performed by multiple computers, i.e., multiple CPUs 251, 351, but this is not limited to this. Image processing and control processing may also be performed by a single computer, i.e., a single CPU. In this case, it is sufficient to configure the single CPU to function as both a processing unit and a control unit.

With reference to Figure 4, a case in which linear members W1 and W2 are assembled in this order to the main body W0 in such a robot system 10 will be described. Based on three-dimensional coordinate information indicating the holding position of linear member W1 calculated by the image processing device 200, the robot controller 300 controls the robot arm 101 to move the robot hand 102 above the holding position. The robot controller 300 then causes the robot hand 102 to hold the linear member W1. Next, the robot controller 300 controls the robot arm 101 to assemble connector A1 of linear member W1 to connector A3 of the main body W0. After completing the assembly of connector A1 to connector A3, the robot controller 300 repeats the same processing operation for linear member W2. This completes the assembly process and produces assembled product W20.

Here, a flexible linear member is soft and easily changes its posture. For this reason, at least two of the multiple linear members, for example, two linear members W1 and W2 as shown in Figure 5, may be arranged around the robot 100 in an intersecting manner. At the point where linear member W1 and linear member W2 intersect, linear member W2 may be positioned higher than linear member W1. Here, the higher linear member refers to the linear member that is positioned highest among the at least two linear members at the point where the at least two linear members intersect. Furthermore, the lower linear member refers to the linear member that is positioned lower than the linear member that is positioned highest among the at least two linear members at the point where the at least two linear members intersect. In the example of Figure 5, at the point where linear member W1 and linear member W2 intersect before assembly, linear member W2 is the higher linear member and linear member W1 is the lower linear member.

In this case, if the robot 100 holds the linear member W1 before the linear member W2 and assembles the connector A1 to the connector A3, the linear member W2 may become entangled with the linear member W1. In this state, it may be difficult for the robot 100 to hold the linear member W2 next, or even if the robot 100 can hold the linear member W2, it may be difficult to assemble the connector A2 to the connector A4. Furthermore, linear members are flexible and prone to change position, and the linear members themselves may be small (i.e., have a thin wire diameter). Therefore, a method using a spatial coding method with a projector or other method could be considered to measure the intersection state of multiple linear members in three dimensions with high accuracy, but this measurement method requires a long measurement process, reducing the productivity of the assembly W20.

In this embodiment, the CPU 251 of the image processing device 200 determines the intersection state of the linear members W1 and W2 using at least one of the two-dimensional images I1 ₁ and I1 ₂ captured by the imaging device 400 in order to avoid a situation in which the linear members become entangled during assembly work.

Figures 6(a) and 6(b) are flowcharts of processing according to an embodiment. The flowcharts include each process of the robot 100 control method, including the image processing method. The flowchart shown in Figure 6(a) shows the advance preparation stage, and the flowchart shown in Figure 6(b) shows the assembly work stage. Here, CPU 251 executes program 261 to perform the processing described below. Furthermore, CPU 351 executes program 361 to perform the processing described below.

First, the pre-preparation stage will be explained using the flowchart in Figure 6(a). Figure 7 is an explanatory diagram of the pre-preparation stage according to the embodiment. In the pre-preparation stage, a trained model 260 is created as an example of a predetermined model.

The subject prepared in this preliminary preparation stage is an intermediate product with a similar configuration to the intermediate product W10 that will be the target in the assembly stage. In other words, in the assembly stage, an assembled product W20 is manufactured from the intermediate product W10, but an intermediate product other than the intermediate product W10 used in the assembly stage may be prepared in the preliminary preparation stage as long as it has a similar configuration to the intermediate product W10 used in the assembly stage. Below, the intermediate product used in the preliminary preparation stage will be described using the same reference numerals as the intermediate product W10 used in the assembly stage, as it has a similar configuration.

In addition, it is preferable to set the exposure conditions, such as the imaging time and lens aperture, of the imaging device 400 in the pre-preparation stage to the same as those in the assembly work stage. In addition, it is preferable to set the usage environment, such as the installation position of the imaging device 400 and the illumination light irradiated onto the multiple linear members W1 and W2, in the pre-preparation stage to the same as those in the assembly work stage. Note that, while the pre-preparation stage will be described as using the imaging device 400 that will be used in the assembly work stage, this is not limited to this, and an imaging device different from the imaging device 400 may also be used. Furthermore, since no assembly work is performed in this pre-preparation stage, the robot 100 does not need to be present around the multiple linear members W1 and W2.

In step S11, the CPU 251 causes the imaging device 400 to capture images of multiple linear members W1 and W2, and acquires an image I1A, which is an image captured by the imaging device 400. The image I1A acquired in the processing of step S11 becomes part of the training data D1 used for learning. Therefore, the more images I1A acquired in step S11, the better.

In step S11, the CPU 251 causes the imaging device 400 to capture images of several patterns of linear members W1 and W2 with different intersection states of the linear members W1 and W2, obtains multiple images I1A, and stores these images I1A in a storage unit such as the HDD 254. The posture of the linear members W1 and W2 may be changed by a worker or a mechanism such as a robot.

The multiple images I1A may include images in which the linear members W1 and W2 do not intersect, or images in which neither of the linear members W1 nor W2 is visible. Figure 8(a) shows an example of one of the multiple images I1A. Image I1A shown in Figure 8(a) is a grayscale image. Image I1A includes images IW1A and IW2A corresponding to the linear members W1 and W2, and image IB1A corresponding to the background.

Next, in step S12, the CPU 251, under the operator's instructions, creates training data D1 based on the image I1A obtained in step S11. In this embodiment, a machine learning algorithm is used in the learning process, and within machine learning, a "supervised learning" algorithm is particularly used. Therefore, in step S12, training data D1 to be used for "supervised learning" is created. The training data D1 includes multiple images I1A and multiple images I2A that have a one-to-one correspondence with the multiple images I1A. A trained model 260 is created by machine learning so that each image I2A is output from each image I1A.

Each image I1A is an example of an input image. Each image I2A is an example of an output image. Each image I2A is created based on the corresponding image I1A by the operator operating the image processing device 200, following steps 1 to 3 below.

Step 1: An image corresponding to the background other than the linear members W1 and W2 (other than the linear members) is painted with a single color to create a background image.
Step 2: The images corresponding to the linear members W1 and W2 are filled with a single color to create linear images, which are different from the color of the background image.
Step 3: At the intersection of the linear image corresponding to linear member W1 and the linear image corresponding to linear member W2, the linear image corresponding to the lower linear member is discontinuous. The linear image corresponding to the higher linear member is continuous. The colors of the linear images need only be different from the color of the background image, and may be the same color or different colors.

Figure 8(b) shows an example of image I2A. Image I2A shown in Figure 8(b) was created based on image I1A shown in Figure 8(a). In image I2A, linear image L1A corresponding to the upper linear element and linear image L2A corresponding to the lower linear element are depicted differently depending on whether they are continuous or interrupted at the intersection C1A. Here, when linear image L2A is discontinuous, or interrupted, at intersection C1A, this refers to a state in which linear image L2A is divided into two, sandwiched between linear image L1A, so that linear image L2A does not contact linear image L1A. In this way, linear image L2A corresponding to the lower linear element is intermittent at intersection C1A. In image I2A shown in Figure 8(b), linear images L1A and L2A are a first color, for example, white, and background image B1A is a second color different from the first color, for example, black.

Note that when a computer performs image processing such as binarization on image I1A, the processed image obtained by that image processing does not necessarily become image I2A as shown in Figure 8(b). For example, in the processed image, there may be no boundary between two intersecting linear images, and the two linear images may become continuous. In this embodiment, in step 3, the operator may create image I2A directly from image I1A. Alternatively, in step 3, the computer may first perform image processing such as binarization on image I1A, and then the operator may modify the processed image to create image I2A.

By going through steps 1 to 3 above, a pair of one image I1A and one image I2A is created. By performing steps 1 to 3 above for all images I1A, multiple pairs of images I1A and I2A are created. These multiple pairs make up the training data D1.

Next, in step S13, CPU 251 performs machine learning to associate image I1A with image I2A. The machine learning algorithm used in step S13 is, for example, Pix2Pix (Pixel to Pixel). Pix2Pix is a type of "supervised learning" algorithm that performs machine learning based on training data D1 to infer an output value for each pixel of the input image. If learning progresses smoothly, the linear image corresponding to the upper linear element at the intersection will be filled in with a continuous solid color, and the linear image corresponding to the lower linear element at the intersection will be filled in with a break at the intersection.

Pix2Pix is used in this type of learning because it can automatically extract features from the shading information on images of linear members in an intersection to determine whether the linear members are continuous or discontinuous. Features include, for example, the shadow formed on the linear image corresponding to the lower linear member at the intersection, and the color continuity of the linear image corresponding to the upper linear member at the intersection. Therefore, in this embodiment, if the vertical relationship between two intersecting linear members does not appear as a shading feature in the captured image, as shown in Figure 8(c), the captured image is excluded from the training data D1. For example, if light is irradiated evenly from all directions, such as with dome lighting, and the image is captured with an equal amount of light, no shadows are formed near the intersection, and a situation like that shown in Figure 8(c) may occur, and therefore images obtained by such capture are excluded.

Furthermore, the algorithm used for machine learning is not limited to Pix2Pix, and any algorithm other than Pix2Pix may be used as long as it has the function of extracting the above-mentioned features.

The trained model 260 obtained by the training in step S13 is stored in a storage unit of the image processing device 200, for example, in the HDD 254.

Next, the assembly work stage will be described with reference to the flowchart in Figure 6(b). Figure 9 is an explanatory diagram of the assembly work stage according to the embodiment. Note that the intermediate product W10 is sequentially transported near the robot 100. In step S21, the CPU 251 causes the imaging device 400 to capture images of the linear members W1 and W2 of the intermediate product W10 that have been transported near the robot 100. As described above, it is preferable that the imaging conditions, such as exposure time and lens aperture, and the usage environment during this imaging are the same as those in step S11.

Through the imaging operation of step S21, the CPU 251 acquires an image I1 as shown in FIG. 9 from the imaging device 400. Image I1 is an example of a first image. Image I1 is a grayscale image. In this embodiment, since the imaging device 400 is a stereo camera, two images _I11 and _I12 are obtained through the imaging operation. Image I1 shown in FIG. 9 is one of the two images _I11 and I12. As shown in FIG. 9, two linear members W1 and _W2 are captured in image I1. That is, image I1 includes images IW1 and IW2 corresponding to the linear members W1 and W2, and an image IB1 corresponding to the background. In this embodiment, the image is captured so that the tip of each linear member W1 and W2, which is an example of a predetermined portion to be held by the robot 100, is captured. In image I1, end point IE1 of image IW1 corresponds to the tip of linear member W1, and end point IE2 of image IW2 corresponds to the tip of linear member W2.

Next, in step S22, the CPU 251 infers the intersection state of the linear members W1 and W2 from the image I1 captured in step S21. For the inference, the trained model 260 stored in the HDD 254 is used. That is, in step S22, the CPU 251 performs image processing on the image I1 based on the trained model 260 to generate an image I2. In other words, the CPU 251 uses the image I1 as an input image and processes the image I1 based on the trained model 260 to generate an image I2 as an output image. Image I2 is an example of a second image.

Image I2 includes linear images L1 and L2 corresponding to linear members W1 and W2. In Figure 9, linear images L1 and L2 are a first color, for example, white, and background image B1 is a second color different from the first color, for example, black. At the intersection of two linear images L1 and L2, the linear image corresponding to the higher linear member is a continuous monochromatic image, while the linear image corresponding to the lower linear member is a discontinuous monochromatic image. In other words, at the intersection, the linear image corresponding to the lower linear member is discontinuous, sandwiched between the linear image corresponding to the higher linear member.

In the example shown in Figure 9, image I2 includes multiple intersections C1 and C2. In intersection C1, the first linear image corresponding to the higher-level linear member is linear image L1, and the second linear image corresponding to the lower-level linear member is linear image L2. In intersection C2, the first linear image corresponding to the higher-level linear member is linear image L2, and the second linear image corresponding to the lower-level linear member is linear image L1.

The method for detecting intersections C1 and C2 in image I2 is described below. In steps S23 and S24, CPU 251 extracts intersection candidates from image I2, performs a predetermined calculation on the intersection candidates, and performs intersection determination processing to determine whether the intersection candidates are intersections based on the results of the calculation.

The processing of step S23 will now be described. First, the CPU 251 extracts intersection candidates in image I2. Below, the method for extracting intersection candidates will be described in detail using Figures 10(a) to 11(b).

First, as shown in Figure 10(a), CPU 251 extracts line segment images L11, L12, L21, and L22 from image I2. Next, as shown in Figure 10(b), CPU 251 extracts the endpoints of each area for line segment images L11, L12, L21, and L22. Various endpoint extraction algorithms, such as the Harrison corner detection method, are known as methods for extracting endpoints, but no particular method is limited. The endpoints extracted in this manner are shown in Figure 10(b) as endpoints L11-a, L11-b, L12-a, L21-a, L21-b, and L22-a. Next, CPU 251 defines a predetermined area containing each endpoint and sets this as an endpoint area for each endpoint. In Figure 10(c), these are shown as endpoint regions T11-a, T11-b, T12-a, T21-a, T21-b, and T22-a. Each endpoint region is, for example, a rectangular region of a predetermined size, with each endpoint located at the center of the endpoint region. Note that the shape of each endpoint region is not limited to a rectangle, and may be various shapes, such as a circle.

If there are at least two endpoint regions in image I2, CPU 251 determines whether the positional relationship between the positions and orientations of the line segment images contained in the endpoint regions for all combinations of endpoint regions is within a certain threshold. Based on the results of this determination, if the relationship is within the certain threshold, the region containing the endpoint regions of that combination is extracted as an intersection candidate.

The following is a specific explanation using two line segment images L21 and L22 as examples. Line segment image L21 is an example of a first line segment image, and line segment image L22 is an example of a second line segment image. Furthermore, endpoint L21-b of line segment image L21 is an example of a first endpoint, and endpoint L22-a of line segment image L22 is an example of a second endpoint. Endpoint region T21-b, which includes endpoint L21-b, is an example of a first region, and endpoint region T22-a, which includes endpoint L22-a, is an example of a second region.

As shown in FIG. 11(a), the CPU 251 calculates a regression line LR1 that follows the line segment image L21 in the endpoint region T21-b, and a regression line LR2 that follows the line segment image L22 in the endpoint region T22-a. To calculate the regression line LR1, the portion of the line segment image L21 that is included in the endpoint region T21-b is used. To calculate the regression line LR2, the portion of the line segment image L22 that is included in the endpoint region T22-a is used.

The CPU 251 calculates the distance Da between the perpendicular line Lp1 drawn from the regression line LR1 to the endpoint L21-b of the line segment image L21 and the perpendicular line Lp2 drawn from the regression line LR2 to the endpoint L22-a of the line segment image L22. The CPU 251 also calculates the intersection angle θb between the regression lines LR1 and LR2.

In this embodiment, the CPU 251 determines whether the predetermined conditions are met: the distance Da is equal to or less than the threshold value THa, and the intersection angle θb is equal to or less than the threshold value THb. If the predetermined conditions are met between the line segment images L21 and L22, the CPU 251 extracts the area that includes the endpoint regions T21-b and T22-a as the intersection candidate C11. The intersection candidate C11 is, for example, the smallest rectangular area that includes the endpoint regions T21-b and T22-a. Note that the shape of the area indicating the intersection candidate is not limited to a rectangle, and may be various shapes, such as a circle.

In Figure 11(b), the area including the combination of endpoint regions T21-b and T22-a satisfies the specified conditions and is therefore an intersection candidate C11, while the area including the combination of endpoint regions T11-b and T12-a satisfies the specified conditions and is therefore an intersection candidate C12. This makes it possible to infer the discontinuous area between the two line segment images. Note that in Figure 11(c), for example, endpoint region T11-a and endpoint region T21-a do not satisfy the specified conditions and are therefore not intersection candidates.

Next, for intersection candidate C11, CPU 251 interpolates line segment image L21 and line segment image L22 using a predetermined interpolation method. Linear interpolation is preferably used as the predetermined interpolation method. By linearly interpolating line segment image L21 and line segment image L22, a linear interpolated region R1 shown in Figure 11(a) is obtained. A similar interpolated region is obtained for intersection candidate C12 shown in Figure 11(b).

Through the above interpolation process, an interpolated region R1 for intersection candidate C11 and an interpolated region R2 for intersection candidate C12 are determined, as shown in Figure 12(a). Interpolated region R1 is a pixel region connecting line segment image L21 and line segment image L22. Interpolated region R2 is a pixel region connecting line segment image L11 and line segment image L12. Note that the method is not limited to linear interpolation, as long as it is possible to infer the discontinuous region between two line segment images.

Next, in step S24, the CPU 251 determines whether the line segment image L11 and the interpolated region R1 overlap in the intersection candidate C11. If the line segment image L11 and the interpolated region R1 overlap, the CPU 251 determines the intersection candidate C11 to be the intersection portion C1. Similarly, the CPU 251 determines whether the line segment image L22 and the interpolated region R2 overlap in the intersection candidate C12. If the line segment image L22 and the interpolated region R2 overlap, the CPU 251 determines the intersection candidate C12 to be the intersection portion C2. Note that depending on the state of the linear members W1 and W2, there may be one intersection portion, or multiple intersection portions C1 and C2 as shown in Figure 9. Furthermore, if one line segment image and the interpolated region do not overlap in the intersection candidate, the intersection candidate is determined not to be an intersection portion.

As shown in Figure 12(b), at the intersection C1, the continuous line segment image L11 is the line image L1 corresponding to the linear member W1, and the line segment images L21 and L22 connected by the interpolated region R1 are the line image L2 corresponding to the linear member W2. At the intersection C1, the CPU 251 identifies the linear member W1 corresponding to the linear image L1 as the upper linear member WU, and identifies the linear member W2 corresponding to the linear image L2 as the lower linear member WL.

Next, in step S25, the CPU 251 determines the position of a predetermined part of the upper linear member WU that the robot 100 is to hold, i.e., the holding position. In this embodiment, the CPU 251 calculates the holding position by three-dimensional measurement using images captured by the imaging device 400. As shown in FIG. 1, the imaging device 400 has multiple cameras 401, 402 arranged at intervals. The parallax between camera 401 and camera 402 enables three-dimensional measurement of the linear members W1, W2 that are the subjects.

13A to 13D are explanatory diagrams of an example of three-dimensional measurement according to an embodiment. In step S21 described above, the CPU 251 acquires image I11 shown in FIG. _13A , which is generated by the imaging operation of camera 401, and image I12 shown in FIG. _13A , which is generated by the imaging operation of camera 402. Each of images _I11 and _I12 is a grayscale image. Image _I11 includes image _IW11 corresponding to linear member W1 and image _IW21 corresponding to linear member W2. End point _IE11 of image _IW11 corresponds to the tip of linear member W1, and end point _IE21 of image _IW21 corresponds to the tip of linear member W2. Image _I12 includes image _IW12 corresponding to linear member W1 and image _IW22 corresponding to linear member W2. The end point IE1 ₂ of the image IW1 ₂ corresponds to the tip of the linear member W1, and the end point IE2 ₂ of the image IW2 ₂ corresponds to the tip of the linear member W2.

Then, in step S22 described above, the CPU 251 generates images I21 and _I22 shown in FIG. 13B from images _I11 and _I12 by inference based on the trained model 260. Images _I21 and _I22 are binarized images. Image _I21 includes a linear image L11 corresponding to linear member W1, i.e., image _IW11 , and a linear image _L21 corresponding to linear member _W2 , i.e., image _IW21 . Image _I22 includes a linear image _L12 corresponding to linear member W1, i.e., image _IW12 , and a linear image _L22 corresponding to linear member W2, i.e., image _IW22 . Then, in steps S23 and _S24 described above, the CPU 251 performs intersection determination processing. As a result, in image _I2-1 , an intersection _C1-1 between linear image _L1-1 and linear image _L2-1 is determined, and in image _I2-2 , an intersection _C1-2 between linear image _L1-2 and linear image _L2-2 is determined. At the intersection _C1-1 , it is determined that linear image _L1-1 corresponds to the higher linear member WU, and it is determined that linear image _L2-1 corresponds to the lower linear member WL. Furthermore, at the intersection _C1-2 , it is determined that linear image _L1-2 corresponds to the higher linear member WU, and it is determined that linear image _L2-2 corresponds to the lower linear member WL.

Next, in each image _I21 , _I22 , the end points _E11 , _E12 of each linear image _L11 , _L12 corresponding to the upper linear member WU are determined. Here, it is assumed that the position where the main body W0 is installed around the robot 100 and the extension direction of the linear members W1, W2 relative to the main body W0 are predetermined. This determines the approximate area in each image _I21 , _I22 where the end points _E11 , E12 corresponding to the connectors A1, A2 at the ends of the linear members W1, _W2 are located. As shown in Figure 13(c), the CPU 251 interlaces scans each image _I21 , _I22 from a position opposite the position where the main body W0 is located to search for each linear image _L11 , _L12 corresponding to the upper linear member WU. Then, the CPU 251 determines the first points of contact in each of the images I2 ₁ and I2 ₂ as the end points E1 ₁ and E1 ₂ .

Note that even if the linear member W1 is a linear member without a connector, the end points _E11 and _E12 can be searched for in a similar manner. Furthermore, the holding position of the portion of the linear member that the robot 100 is to hold may be a position offset from the tip by a predetermined length from the tip toward the base end of the linear member. In this case, as shown in FIG. 13( d ), the CPU 251 detects the end points _E11 and _E12 , and then determines points _F11 and _F12 that are offset by a predetermined length from the end points _{E11 and E12} along the linear images _L11 and _L12 .

Based on the positions of the endpoints _E11 , _E12 or the points _F11 , _F12 on the images _I21 , _I22 obtained by the above process, the three-dimensional position of the holding position can be measured by the principle of triangulation. The CPU 251 transmits information about the holding position to the CPU 351 of the robot controller 300. In step S26, the CPU 351 of the robot controller 300 controls the robot 100 to hold a predetermined portion of the linear member W1, which is the upper linear member WU, based on the received information about the holding position. The CPU 351 then controls the robot 100 to assemble the connector A1 of the linear member W1 to the connector A3. This prevents the linear members W1, W2 from being held by the robot 100 in an entangled state, allowing for smooth assembly work.

Note that while the imaging device 400 has been described as a stereo camera, this is not limiting. For example, the imaging device 400 may be a monocular camera. In this case, the holding position may be measured using a separate device.

In the above example, a case was described where there is one intersection between linear members W1 and W2, but there may be multiple intersections between linear members W1 and W2. In such cases, robot 100 may be operated to eliminate the intersection between linear members W1 and W2. The processing for such cases will be described with reference to Figures 14(a) to 14(f). Note that Figures 14(a) to 14(f) illustrate and explain only image I1 obtained from one of the multiple cameras 401, 402 shown in Figure 1, or image I2 obtained by processing image I1. The image obtained from the other camera is also processed in the same way as image I1 obtained from one camera, so its description will be omitted.

First, the CPU 251 acquires the image I1 shown in FIG. 14(a) from the imaging device 400. In image I1, the two images IW1 and IW2 corresponding to the two linear members W1 and W2 are twisted so that they intersect twice. Also, in image I1, the endpoint IE1 of image IW1 corresponds to the tip of linear member W1, and the endpoint IE2 of image IW2 corresponds to the tip of linear member W2.

The CPU 251 generates image I2 shown in FIG. 14(b) from image I1 based on the trained model 260 and performs intersection detection processing. This detects multiple intersections, for example, two intersections C1 and C2, between linear image L1 corresponding to linear member W1 and linear image L2 corresponding to linear member W2.

Next, as shown in Figure 14(c), the CPU 251 selects the only intersection CC of the intersections C1 and C2 to be focused on. The twisted linear members W1 and W2 need to be unwound from their tips. Therefore, the CPU 251 searches for an intersection by scanning image I2 in an interlaced manner from the opposite position from where the main body W0 is located, and selects the first intersection C1 encountered as the intersection CC to be focused on. Note that the intersection CC to be focused on may also be determined by tracing the linear image L1 from the endpoint E1 of the linear image L1 that is first encountered by interlaced scanning.

Once the unique intersection CC of interest is determined in this way, the above-described process defines the upper linear member WU and the lower linear member WL at that intersection CC, as shown in Figure 14(d). In this way, the CPU 251 identifies the upper linear member WU at the first intersection point by tracing from the tips of the linear members W1 and W2. Note that at the intersection C1, the linear image L1 corresponds to the upper linear member WU, and the linear image L2 corresponds to the lower linear member WL. Then, by scanning interlacedly, the endpoint E1 is searched for, and the grip position is determined.

Because there are two intersections C1 and C2, CPU 351 controls robot 100 to hold linear member W1, which is the upper linear member WU, at intersection C1. CPU 351 then controls robot 100 to eliminate the intersection of the upper linear member WU with the lower linear member WL. The position or location to which the upper linear member WU is retracted is determined based on the positional relationship between linear members W1 and W2 on the two-dimensional image related to intersection C1. By repeating the above operations, the number of intersections is reduced by one.

The CPU 251 again causes the imaging device 400 to capture images of the linear members W1 and W2, and performs image processing on the captured image based on the trained model 260 to generate a new image I2 as shown in FIG. 14(e). The CPU 251 then performs the intersection determination process again. In the example shown in FIG. 14(e), only one intersection, C2, is detected. At intersection C2, the linear image L2 corresponds to the upper linear member WU, and the linear image L1 corresponds to the lower linear member WL. Then, as shown in FIG. 14(f), interlaced scanning is performed to search for the endpoint E2 of the linear image L2, and the gripping position is determined. Thereafter, the linear members W2 and W1 can be assembled into the main body W0 in the order of W2, W1. The above operations eliminate tangles between the linear members W1 and W2, facilitating the assembly of the linear members W1 and W2.

As described above, according to this embodiment, the overlapping state of linear members W1 and W2 can be determined with high accuracy using image I1 captured of linear members W1 and W2, allowing the robot 100 to hold the upper linear member of linear members W1 and W2. In other words, this prevents the robot 100 from holding a linear member entangled with other linear members. For example, when assembling connector A2 to connector A4 and connector A1 to connector A3, it is possible to avoid a situation where linear members W1 and W2 become entangled, as shown in Figure 5. This improves the productivity of assembly W20.

Furthermore, when determining the intersection state of the linear members W1 and W2, image I2 is generated using the trained model 260, so there is no need to use a high-pixel camera for the imaging device 400. Furthermore, to determine the holding position of the linear member, there is no need to prepare a separate projector, nor is there a need to prepare a highly accurate three-dimensional measurement system that measures the thickness of thin wires. In this embodiment, since the intersection state of multiple linear members W1 and W2 can be determined using image I2 generated based on the trained model 260, it is possible to have the robot 100 hold the linear members without tangling them, with a simple and portable configuration. This improves the productivity of the assembly W20.

The present invention is not limited to the embodiments described above, and many modifications are possible within the technical concept of the present invention. Furthermore, the effects described in the embodiments are merely a list of the most favorable effects resulting from the present invention, and the effects of the present invention are not limited to those described in the embodiments.

In the above embodiment, we have described a case where image I2A used in training data D1 is a binary image, but this is not limited to this. In image I2A, each of the multiple linear images corresponding to multiple linear components may be a different color. In this case, in image I2 generated based on the trained model 260, the multiple linear images will be a different color.

In addition, in the above embodiment, the robot arm 101 is described as a vertically articulated robot arm, but this is not limited to this. The robot arm 101 may be various robot arms, such as a horizontally articulated robot arm, a parallel link robot arm, or an orthogonal robot. Furthermore, instead of a robot arm, the present invention may be applied to a machine that can automatically perform movements such as extension and contraction, bending and stretching, vertical movement, horizontal movement, or rotation, or a combination of these movements, based on information stored in a storage device provided in the control device.

Furthermore, in the above embodiment, a case was described in which the imaging device 400 was fixed to a frame other than the robot 100, but this is not limited to this, and the imaging device 400 may be fixed to, for example, the robot 100.

(Other Examples)
The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more of the functions.

10...Robot system, 100...Robot, 200...Image processing device, 251...CPU (processing unit), 351...CPU (control unit), 400...Image capture device (image capture unit)

Claims (22)

The processing unit
Using the trained model, a binary image with gray levels capable of detecting multiple linear components is obtained,
identifying a higher-ranking linear member from among at least two mutually intersecting linear members based on a portion where the linear members are discontinuous in the image;
1. An information processing method comprising: A portion where at least two linear members intersect in the image is discontinuous.
2. The information processing method according to claim 1, the images include a first linear image corresponding to an upper linear member and a second linear image corresponding to a lower linear member intersecting the upper linear member;
The processing unit
determining that the linear member corresponding to the second linear image is discontinuous when the second linear image is divided into two parts with the first linear image sandwiched between them so as not to come into contact with the first linear image;
Identifying a linear member corresponding to the second linear image that is discontinuous as a lower linear member, and identifying a linear member corresponding to the first linear image that is not discontinuous as a higher linear member.
3. The information processing method according to claim 1, wherein: The processing unit
extracting an area including a first area in which a first end point of a first line segment image exists and a second area in which a second end point of a second line segment image exists as an intersection candidate if a predetermined condition is satisfied between the first area and the second area in the image;
If an area obtained by interpolating the first line segment image and the second line segment image using a predetermined interpolation method overlaps with another line segment image, the intersection candidate is determined to be an intersection portion.
4. The information processing method according to claim 1, wherein the first and second inputs are input to the first and second inputs. The predetermined condition is a condition that a distance between a perpendicular line drawn from a first regression line along the first line segment image to the first endpoint and a perpendicular line drawn from a second regression line along the second line segment image to the second endpoint is within a threshold range, and an intersection angle between the first regression line and the second regression line is within a threshold range.
5. The information processing method according to claim 4. the predetermined interpolation is linear interpolation;
6. The information processing method according to claim 4 or 5. Each of the linear members has a free end,
taking an image of the linear members so that the tips of the linear members can be detected from the image;
7. The information processing method according to claim 1, wherein: the at least two linear members include a first linear member and a second linear member that intersect with each other;
When there are a plurality of points where the first linear member and the second linear member intersect, the processing unit traces back from the tip of each of the first linear member and the second linear member to identify a linear member that is higher than the first intersecting point.
8. The information processing method according to claim 1, wherein: The processing unit
having the robot retain the first detected end point of the identified upper linear member relative to the first intersection;
9. The information processing method according to claim 8. The image is binarized by making the color of the linear members different from the color of the background of the linear members.
10. The information processing method according to claim 1, wherein: The processing unit
acquiring a first image of the linear member;
converting the first image into the image for detecting linear members using the trained model;
11. The information processing method according to claim 1. The first image is used as an input image, and a second image in which the shading is binarized so that a plurality of linear members can be detected is used as training data to perform machine learning as an output image, thereby generating the trained model;
The processing unit
converting the first image into the image for detecting linear members using the trained model;
12. The information processing method according to claim 11. The second image used as the training data can be modified by a user.
13. The information processing method according to claim 12 . The processing unit
controlling the robot to hold the upper linear member, and then controlling the robot to assemble the tip of the upper linear member to the assembly target;
13. The information processing method according to claim 1 , wherein the first and second inputs are input to the first and second inputs. The processing unit
controlling the robot to hold the upper linear member, and then controlling the robot to prevent the upper linear member from crossing the lower linear member;
13. The information processing method according to claim 1 , wherein the first and second inputs are input to the first and second inputs. The first image is acquired by a stereo camera.
13. The information processing method according to claim 12 , wherein the information processing method comprises: The processing unit
Using the trained model, a binary image with gray levels capable of detecting multiple linear components is obtained,
identifying a higher-ranking linear member from among at least two mutually intersecting linear members based on a portion where the linear members are discontinuous in the image;
1. An information processing device comprising: A method for controlling a robot, comprising:
Using the trained model, a binary image with gray levels capable of detecting multiple linear components is obtained,
Identifying a higher-ranking linear member from among at least two mutually intersecting linear members based on a portion where the linear members are discontinuous in the image;
controlling the robot to hold the upper linear member;
A control method comprising: A robot system including a robot and a processing unit,
The processing unit
Using the trained model, a binary image with gray levels capable of detecting multiple linear components is obtained,
Identifying a higher-ranking linear member from among at least two mutually intersecting linear members based on a portion where the linear members are discontinuous in the image;
controlling the robot to hold the upper linear member;
A robot system characterized by: A method for manufacturing an article, which comprises manipulating a linear member using the information processing method according to any one of claims 1 to 16 to manufacture an article. A program for causing a computer to execute the information processing method according to any one of claims 1 to 16 or the control method according to claim 18 . 2. A computer-readable recording medium on which the program according to claim 1 is recorded.