JP6562619B2 - Information processing apparatus, information processing method, and program - Google Patents

Description

  The present invention relates to a technique for detecting a component shortage in component picking by a robot.

  In recent years, techniques for picking parts using a robot have been actively proposed to automate the assembly and classification of parts and the like of industrial products. As an example, there are components in which components are piled up in a component container, which is photographed / recognized, gripped and taken out by a robot, and assembled and classified. Here, “stacking” of parts refers to a state in which parts are collected and stacked without being aligned.

  In such a robot picking system, when parts piled up in a container are taken out, there will eventually be a situation where the parts disappear from the container. If an attempt is made to photograph / recognize a part in the absence of the part, an unexpected error may occur, or an unpredictable action may occur when the robot tries to grip.

  As a countermeasure against the shortage of parts, there is a method of supplying a new part every time the part is grasped and taken out by using a part supply feeder. In this case, a custom-made component supply feeder corresponding to the type of component is required. Further, since it is necessary to handle the robot picking system and the component supply feeder as a set, there is a problem that the system becomes huge.

  In Patent Document 1, in order to detect whether there is a part in the container, a light transmitting container is used to illuminate the light source from the upper and lower sides of the container, and the camera is photographed from the opposite side. And it has disclosed detecting the presence or absence of the component in a container by detecting the presence or absence of the shadow of a component from the image | photographed image.

Japanese Patent Laid-Open No. 2001-300877

  In the method according to Patent Document 1, it is necessary to limit the container to one that transmits light. Further, there is a restriction that a light source must be arranged on one side of the container and a camera on the other side, and there is a problem that the apparatus cannot be simplified.

  The present invention has been made in view of the above problems, and an object thereof is to detect a shortage of parts with a simple configuration.

  In order to solve the above problems, an information processing apparatus according to the present invention includes, for example, an acquisition unit that acquires an image including a container in which a target object is stored, and a position and orientation that estimates the position and orientation of the object based on the image An estimation means, a control means for controlling the holding means to hold a part based on the position and orientation estimated by the position and orientation estimation means, and a state estimation means for estimating the state in the container based on the image And a judging means for judging whether or not there are insufficient parts in the container based on the result estimated by the state estimating means, and an output means for outputting the result judged by the judging means. The state estimating means detects a bottom surface area where the object is not stacked in the container by detecting a pattern drawn on the bottom surface of the container, and the judging means Based on the detected result to determine whether or not the object in the container is insufficient.

  According to the present invention, a shortage of parts can be detected with a simple configuration.

The figure which shows the structural example of the robot picking system of 1st Embodiment. The flowchart which shows the flow of a process of 1st Embodiment. The figure which shows the example of a components area | region and a bottom face area | region. The schematic diagram which shows the example of distribution on the three-dimensional space of pile components. The figure which shows the structural example of the robot picking system of 3rd Embodiment. The flowchart which shows the flow of the process of 3rd Embodiment. It is a figure which shows the hardware constitutions of the information processing apparatus of this invention.

  Prior to describing each embodiment according to the present invention, a hardware configuration in which the information processing apparatus shown in each embodiment is mounted will be described with reference to FIG.

  FIG. 7 is a hardware configuration diagram of the information apparatus according to the present embodiment. In the figure, a CPU 710 comprehensively controls each device connected via a bus 700. The CPU 1710 reads and executes processing steps and programs stored in a read only memory (ROM) 720. In addition to the operating system (OS), each processing program, device driver, and the like according to the present embodiment are stored in the ROM 720, temporarily stored in a random access memory (RAM) 730, and appropriately executed by the CPU 710. The input I / F 740 is input as an input signal in a format that can be processed by the information processing apparatus 1 from an external device (display device, operation device, or the like). The output I / F 750 is output as an output signal in a format that can be processed by the display device to an external device (display device).

  Each of these functional units is realized by the CPU 710 developing a program stored in the ROM 720 on the RAM 730 and executing processing according to each flowchart described below. Further, for example, when hardware is configured as an alternative to software processing using the CPU 710, an arithmetic unit or a circuit corresponding to the processing of each functional unit described here may be configured.

(First embodiment)
An information processing apparatus 3 according to the first embodiment and a picking system including the information processing apparatus 3 will be described with reference to FIG. The information processing apparatus 3 according to the first embodiment includes an image acquisition unit 31, a component position / posture estimation unit 32, a piled state estimation unit 33, and a component state determination unit 34. The information processing device 3 is connected to the imaging device 2, the component supply device 5, and the robot control device 41 to constitute a picking system.

  The robot control unit 41 calculates each rotation angle to be moved by the robot hand 42 based on the estimation result of the position and orientation of the component 11 by the component position and orientation estimation unit 32, and controls the robot hand 42 to hold the component 11. To do. The hardware of the robot control unit 41 is composed of a general-purpose computer (hardware) having a storage device such as a CPU, a memory, and a hard disk, and various interfaces for input / output, but the same hardware as the information processing device 3 It may be configured as internal software.

  The robot hand 42 receives a command from the robot control unit 41, grips (holds) the component 11 in the container, takes it out, and performs assembly. The robot hand 42 is, for example, an arm with six axes of freedom, and includes a two-finger hand mechanism as an end effector. Further, a magnet type or adsorption type hand that holds the component 11 by pressing it against the flat surface portion may be used.

  The component (object) 11 is a target to be gripped and taken out by the robot hand 42 and assembled.

  The container 12 is a container for storing a plurality of parts 11. The container 12 has a bottom surface portion 13. In the following description of the embodiment, the description will be made on the assumption that a plurality of parts 11 are piled up in the container 12, but the present invention is not limited to the container, and any area can be used as long as the parts can be placed on a table or the like. Good. The bottom surface portion 13 represents a surface of an area where the component 11 can be placed.

  The imaging device 2 is arranged so as to keep the target component 1 and the container 12 at an angle of view, and images the target component 1 and the container 12. The imaging device 2 is an imaging device arranged so that the container 12 can be imaged from directly above, and is, for example, a digital camera. The captured image signal is sent to the image acquisition unit 31.

  The image acquisition unit 31 captures a digital image signal sampled and quantized by the imaging device 2. Furthermore, it has a function of acquiring image data represented by luminance (density value) of each pixel from the captured image signal and storing it in the memory. Note that the image acquisition unit 32 has a function of controlling the operation (such as imaging timing) of the imaging apparatus 2 via a general-purpose communication interface such as RS232C or IEEE488.

  The component position / orientation estimation unit 32 determines a component to be gripped from the plurality of components 11 using at least one piece of image data obtained from the image acquisition unit 31, and estimates the position and orientation thereof.

  The pile state estimation unit 33 detects at least one of the component region and the bottom region in the region of the target component unit 1 using at least one piece of image data obtained from the image acquisition unit 31. Thereby, the pile state estimation part 33 estimates the pile state of the components in a container (state estimation).

  The component state determination unit 34 determines whether or not the component 11 in the container 12 is insufficient based on the component region and the bottom region detected by the piled state estimation unit 33. If the robot hand 42 continues to take out the parts, the parts will eventually run out. For this reason, the component state determination unit 34 determines the lack of components based on the image data captured by the imaging device 2 and outputs the determined result to the component supply unit 5.

  The component supply device 5 includes a component tank. When the component state determination unit 34 determines that the component is insufficient, the component supply device 5 supplies the component by sending a certain amount of component from the component tank into the container 12. To do. Regarding the supply timing, it is considered to be most efficient from the viewpoint of cycle time to be performed while the assembly is performed after the robot hand 42 grasps and takes out.

  With reference to the flowchart of FIG. 2, the overall flow up to the supply of components according to the first embodiment will be described.

(Step S1)
In step S <b> 1, the imaging device 2 captures an area including the part 11. The area to be photographed is an area including the entire container bottom surface 13 on which the part 11 can be placed. The number of images to be photographed differs depending on the data handled in the subsequent component position / orientation estimation processing and stacking state estimation processing. The photographed data is sent to the image acquisition unit 32 as an image signal.

(Step S2)
In step S2, the component position / orientation estimation unit 32 estimates the position / orientation of the component to be gripped based on the captured image. As a position / orientation estimation method, estimation is performed by performing 2D model fitting from image features using feature amounts such as SIFT extracted from partial regions of image data. Further, for example, it may be estimated by holding posture data of a plurality of parts in advance and comparing the feature amount of each posture data with the feature amount of the partial area of the image data. However, the position / orientation estimation method is not limited to this. For example, a stereo method in which the imaging apparatus is configured by a plurality of cameras, or a projection unit such as a projector is prepared, and an active stereo such as a phase shift method or a spatial encoding method is prepared. A distance image may be obtained using a method, and the position and orientation of a component to be grasped may be estimated by model fitting with the distance image. The component position / orientation estimation unit 32 sends the estimated position / orientation information to the robot control unit 41.

(Steps S3a, 3b)
In step S3a, the robot control unit 41 outputs a command to the robot hand 42 according to the output from the component position / orientation estimation unit 32, and the robot hand 42 holds the component 11 according to the received command. Take out. This process is performed in parallel with step S3 described below. If there is an end command in step S3b, the process is ended. On the other hand, if there is no end instruction, it joins the main loop and continues processing.

(Step S3)
In step S <b> 3, the piled state estimation unit 33 detects a component region and a bottom region as a piled state of components in the container 1 based on the image data. FIG. 3 shows an example of the detection result of the component area and the bottom area. As a method for detecting the component region and the bottom region, the difference between the colors of the component 11 and the container bottom surface 13 is used to divide the region according to the color and detect the component region and the bottom region. However, it is not limited to this method, for example, by drawing a texture such as a circular marker on the bottom surface of the container at regular intervals, and detecting the circular marker from the image data, the bottom region may be detected, Based on the distance image obtained by the stereo method or the active stereo method, the component region and the bottom surface are determined from the difference between the distance measurement value in the region where the component 11 exists in the pile and the distance measurement value to the bottom surface 13 of the container. An area may be detected.

  As the image data used here, by using the same image data as that used for estimating the component position and orientation, it is possible to efficiently detect the component region and the bottom region by reducing the number of photographing. However, the present invention is not limited to this, and after the component is grasped and taken out by the robot hand 42, the image may be taken again by the imaging device 2 and detected based on the image data obtained from the image acquisition unit 31.

(Step S4)
In step S4, the component state determination unit 34 determines a component shortage state based on the ratio of the component region and the bottom region on the bottom surface of the container. As a determination method of the shortage state, the ratio of the part area occupying the entire area of the bottom surface 13 is obtained. Further, it may be obtained by calculating backward from the ratio of the bottom region. When the piled state is piled up high in a narrow area, the number of parts is sufficiently large, but there are few parts that are easy to grip, so in this case as well, it is determined that there are insufficient parts.

  If it is determined in step S4 that there is a shortage of parts, the process proceeds to the next step S5. If it is determined that there is no shortage of parts, the process returns to step S1, and the gripping and taking out of the parts are continued.

(Step S5)
In step S5, the component supply device 5 supplies components. Parts are supplied by feeding a certain amount of parts from the parts tank into the container 12. When the supply of parts and the gripping step by the robot are finished, the process returns to step S1 and starts again from the image capturing step.

  As described above, the gripping process is repeatedly performed while automatically supplying parts until an end command is given.

  According to the first embodiment, since the shortage of parts can be detected and the parts can be automatically supplied from the ratio of the parts area and the bottom area before the parts are lost, the amount of parts to be replenished at a time Less gripping is required, and gripping can be continued without pausing the entire picking system.

(Second Embodiment)
In the second embodiment, as a pile state estimation method, a distribution of distance images obtained by measuring the component 11 and the container 12 is configured, and the component state is determined based on the result. The schematic configuration of a robot picking system for detecting a component shortage according to the second embodiment is the same as that shown in FIG. Since the flowchart of the second embodiment is substantially the same as that of FIG. 2 described in the first embodiment, the description of the same processing will be omitted, and steps S3 and S4 having differences will be described.

  In step S3 in the second embodiment, a distance image calculated using a stereo method or an active stereo method is constructed based on the image data, and a distribution in a three-dimensional space of piled parts is constructed. FIG. 4 schematically shows the distribution in the three-dimensional space obtained from the distance image. For the target component part 1, 1 a represents this as a distribution in a three-dimensional space. In the same figure, two examples of distribution in a three-dimensional space are given, but 1b is a state in which parts are evenly arranged, it is easy to select candidates for gripping, and there are many parts that are easy to grip. . On the other hand, 1c is a piled-up state in which parts are densely stacked in a narrow area, there are few candidates for gripping, and there are not enough parts that are easy to grip.

  In step S4, the lack of parts is determined based on the distribution of the parts obtained in step S3 in the three-dimensional space. If a distribution such as 1b in FIG. 4 is obtained, it is determined that there are not enough parts, and the process returns to step S1, and if a distribution such as 1c is obtained, there are insufficient parts that are easy to grip. And the process proceeds to the next step S5. In step S5, upon receiving a warning, the user supplies a part or issues an end command.

  By determining the component shortage state based on the distribution of the component in the three-dimensional space in this manner, it is possible to detect the component shortage in substantially the same manner as in the first embodiment, and thus other descriptions are omitted.

  Although an example of the ease of gripping in the distribution of the parts in the three-dimensional space has been described with reference to FIG. 4, the present invention is not limited to this distribution, and the arrangement of the parts 11 and the robot hand 42 and the shape of the parts 11 are described. Since the distribution in the three-dimensional space of components that are easily gripped differs depending on the above, such a distribution may be newly defined and used.

  The above is the description of the robot picking system for detecting the shortage of parts according to the second embodiment of the present invention.

(Third embodiment)
In the third embodiment, when there is a shortage of parts, a warning is given to the user. FIG. 5 is a schematic configuration of a robot picking system for detecting a component shortage according to the third embodiment. Although it is substantially the same as FIG. 1 demonstrated in 1st Embodiment, in this embodiment, the point which has the components shortage warning part 6 instead of the components supply part 5 differs. FIG. 6 is a flowchart of the third embodiment. Since this flowchart is also substantially the same as FIG. 2 described in the first embodiment, the description of the same processing is omitted, and step S6 (corresponding to step S5 in the first embodiment) having a difference will be described.

  In step S6 in the third embodiment, if it is determined in the previous step that there is a shortage of parts, the user is warned of the shortage of parts. A warning is displayed on the display used when the user uses the robot picking system, and an alarm sound is sounded together with a warning indicating that there is a shortage of parts. However, the notification method is not limited to this, and other methods such as lighting a component shortage warning lamp may be used.

  The above is the description of the picking system for detecting a component shortage according to the third embodiment of the present invention.

(Modification)
The bottom surface 13 of the container does not need to be a horizontal plane and may be a slope. In that case, it is possible to detect the piled-up state of components and determine the shortage of components by the following method. If a part is placed on a slope, the part will naturally slide on the slope with its weight, and the part will naturally gather at a low position on the slope, and if the number of parts increases, it can also exist at a high position on the slope. Therefore, by drawing a line at a certain height of the bottom surface 13 and detecting whether or not the line is visible from the image captured by the imaging device 2, it is possible to determine the shortage of parts.

  As another modification, as in the second embodiment, the distribution of parts in a three-dimensional space is obtained, and if it is determined that there are not enough easy-to-grip parts, parts are supplied. In addition to warning, for example, the container 12 may be vibrated or the robot hand 42 may be piled up to increase the number of parts that are easy to grip.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

<Effect of embodiment>
In the first embodiment, the component region and the bottom region are detected to determine the shortage of components. Since only the ratio of the component area or the bottom area is calculated, processing can be performed at a very high speed. Further, it is possible to determine the shortage of parts even with only one two-dimensional image as an input.

  In the second embodiment, the shortage state of the parts is determined based on the distribution of the stacked parts in the three-dimensional space. In parts shortage detection by measuring the number and weight of parts, if the number of parts is large, it cannot be regarded as shortage, but the present invention detects that there are few parts that are easy to grip. can do.

  In the third embodiment, a warning is given when there are insufficient parts. As a result, the user can freely deal with the shortage of parts in addition to the parts supply.

Claims (10)

Acquisition means for acquiring an image including a container in which an object is stored;
State estimating means for estimating a state in the container based on the image;
Determination means for determining whether or not an object in the container is insufficient based on the result estimated by the state estimation means;
Output means for outputting the result determined by the determination means ,
The state estimation means detects a bottom surface area in which the object is not stacked in the container by detecting a pattern drawn on the bottom surface of the container, and the determination means determines the result of the detection. the information processing apparatus according to claim that you determine whether or not the object in the container is insufficient based.   The state estimation means detects at least one of the object region and the bottom surface region of the container, and determines whether or not the object in the container is insufficient based on the detected result. The information processing apparatus according to claim 1. The state estimating means acquires the distribution of the region of the object and the bottom region of the container in the container, and determines whether or not the object in the container is insufficient based on the acquired distribution. The information processing apparatus according to claim 1, wherein:   4. The apparatus according to claim 1, further comprising supply means for supplying an object to the container when the determination means determines that an object in the container is insufficient. The information processing apparatus described.   Furthermore, when the judgment means judges that the object in the said container is insufficient, it has an alerting | reporting means to alert | report that the object in the said container is insufficient. 5. The information processing apparatus according to any one of 4. Position and orientation estimation means for estimating the position and orientation of the object based on the image;
6. The apparatus according to claim 1, further comprising a control unit configured to control the holding unit so as to hold an object based on the position and orientation estimated by the position and orientation estimation unit. Information processing device.   The information processing apparatus according to claim 6, wherein the position and orientation estimation unit extracts a feature amount from the image and estimates the position and orientation of the object based on the extracted feature amount.   8. The control unit according to claim 6, wherein the control unit executes the process of controlling the holding unit so as to hold the object in parallel with the process of estimating the state in the container by the state estimation unit. The information processing apparatus described. An acquisition step of acquiring an image including a container in which an object is stored;
A position and orientation estimation step of estimating the position and orientation of the object based on the image;
A state estimation step of estimating a state in the container based on the image;
A determination step of determining whether or not an object in the container is insufficient based on the result estimated in the state estimation step;
An output step of outputting the result determined in the determination step ,
The state estimation step detects a bottom surface area where the object is not stacked in the container by detecting a pattern drawn on the bottom surface of the container, and the determination means an information processing method characterized that you determine whether or not the object in the container is insufficient based.   The program for functioning a computer as each means of the information processing apparatus of any one of Claims 1 thru | or 8.

Images