JP7549221B2 - Recognition System - Google Patents

Description

The disclosure in this specification relates to a recognition system.

Patent document 1 discloses a position and orientation measurement device that can measure position and orientation at high speed using two-dimensional images and range images. The contents of the prior art documents are incorporated by reference as explanations of the technical elements in this specification.

JP 2015-132523 A

In the configuration of the prior art document, while capturing an image for distance measurement, processing of a two-dimensional image is performed in parallel, thereby speeding up overall processing. As processing of a two-dimensional image, processing is disclosed in which edges are detected from a two-dimensional image to obtain approximate values of position and orientation. In this way, in position and orientation recognition, it is required to complete the recognition quickly. However, there is not just one method for speeding up position and orientation recognition, and various methods can be adopted depending on the required accuracy, computational processing power, etc. In the above-mentioned respects, or in other respects not mentioned, further improvements are required in recognition systems.

One disclosed objective is to provide a recognition system capable of quickly recognizing position and orientation.

The recognition system disclosed herein is a recognition system that performs position and orientation recognition for recognizing a position and orientation of a measurement object (41),
a measurement unit (20, 220) that captures a surface image of a measurement object and calculates three-dimensional coordinates of positions corresponding to pixels of the surface image as a three-dimensional point cloud of the measurement object;
a two-dimensional extraction unit (73) that extracts two-dimensional edge information that is information indicating the position of an edge portion in the measurement object based on the surface image;
a three-dimensional extraction unit (74) that extracts three-dimensional edge information that is information indicating the positions of edge portions in a three-dimensional point cloud;
A recognition execution unit (79) for recognizing a position and orientation of a measurement object based on three-dimensional edge information,
The three-dimensional extraction unit extracts three-dimensional edge information from the three-dimensional point cloud based on the two-dimensional edge information;
The measurement unit acquires distance information indicating a distance between the measurement target and the measurement unit for each pixel of the surface image;
The two-dimensional extraction unit determines whether or not an area is an edge portion from the distance information, and extracts two-dimensional edge information .
The disclosed recognition system is a recognition system that performs position and orientation recognition for recognizing a position and orientation of a measurement object (41),
a measurement unit (20, 220) that captures a surface image of a measurement object and calculates three-dimensional coordinates of positions corresponding to pixels of the surface image as a three-dimensional point cloud of the measurement object;
a two-dimensional extraction unit (73) that extracts two-dimensional edge information that is information indicating the position of an edge portion in the measurement object based on the surface image;
a three-dimensional extraction unit (74) that extracts three-dimensional edge information that is information indicating the positions of edge portions in a three-dimensional point cloud;
A recognition execution unit (79) for recognizing a position and orientation of a measurement object based on three-dimensional edge information,
The three-dimensional extraction unit extracts three-dimensional edge information from the three-dimensional point cloud based on the two-dimensional edge information;
The measurement unit acquires, for each pixel of the surface image, luminance information indicating luminance and distance information indicating a distance between the measurement object and the measurement unit;
The two-dimensional extraction unit determines whether or not an area is an edge portion from both the luminance information and the distance information, and extracts two-dimensional edge information.

The disclosed recognition system includes a three-dimensional extraction unit that extracts three-dimensional edge information from a three-dimensional point cloud based on two-dimensional edge information. Therefore, by extracting three-dimensional edge information using two-dimensional edge information, it is possible to extract three-dimensional edge information at higher speeds than when three-dimensional edge information is extracted without using two-dimensional edge information. Therefore, extraction of three-dimensional edge information in position and orientation recognition control can be completed at high speeds. Therefore, a recognition system capable of recognizing position and orientation at high speeds can be provided.

The two-dimensional extraction unit may also be configured to determine whether or not an edge portion is present from the luminance information and extract two-dimensional edge information. In this case, it is not necessary to calculate the distance for each pixel of the surface image in two-dimensional measurement and determine whether or not an edge portion is present from the difference in distance. This makes it easier to extract two-dimensional edge information at high speed.

The two-dimensional extraction unit may also be configured to determine whether or not an area is an edge part from distance information and extract two-dimensional edge information. In this case, it is possible to determine whether or not an area is an edge part from information other than luminance information. In addition, by adjusting the edge distance difference used to determine whether or not an area is an edge part, it is easy to adjust the amount of data in the two-dimensional edge information.

The two-dimensional extraction unit may also be configured to extract two-dimensional edge information by determining whether or not an area is an edge part from both the luminance information and the distance information. In this case, it is easier to adjust the accuracy of the determination of whether or not an area is an edge part, compared to when the determination is based on either the luminance information or the distance information alone. This makes it easier to adjust the balance between the accuracy of position and orientation recognition and the time required for position and orientation recognition.

The recognition system disclosed herein is a recognition system that performs position and orientation recognition for recognizing a position and orientation of a measurement object (41),
A measurement unit (20, 220) capable of capturing a surface image of a measurement object and calculating a three-dimensional point cloud at positions corresponding to pixels of the surface image;
A control unit (70) for controlling the recognition system,
The control unit
Perform shape measurement using the measurement unit ,
extracting two-dimensional edge information that is information indicating the position of an edge portion of the measurement object based on the surface image;
Calculate three-dimensional edge information, which is a three-dimensional point cloud of the edge portion ,
Recognize the position and orientation of the measurement target based on the 3D edge information ,
The control unit performs two-dimensional measurement, which is shape measurement for extracting two-dimensional edge information, and then performs three-dimensional measurement, which is shape measurement for calculating three-dimensional edge information, and calculates a three-dimensional point cloud of pixels corresponding to pixels of the measurement object in the three-dimensional measurement that are determined to be an edge part based on the two-dimensional edge information, and does not calculate a three-dimensional point cloud of pixels corresponding to pixels that are determined not to be an edge part.

The disclosed recognition system includes a three-dimensional calculation unit that calculates a three-dimensional point cloud for a portion of the measurement object that is determined to be an edge portion based on two-dimensional edge information, and does not calculate a three-dimensional point cloud for a portion that is determined not to be an edge portion. This makes it easier to calculate three-dimensional edge information quickly compared to a case where a three-dimensional point cloud for the entire measurement object, including planar portions, is calculated and then three-dimensional edge information is extracted. Therefore, a recognition system capable of quickly recognizing position and orientation can be provided.

The measurement unit may also be configured to calculate a three-dimensional point cloud based on a color light pattern. In this case, there is no need to change the phase of the light pattern and repeat light projection and reception multiple times. Therefore, shape measurement in position and orientation recognition control can be completed quickly.

The measurement unit may also be provided on the robot. In this case, the position of the measurement unit during shape measurement can be adjusted by controlling the movement of the robot. Therefore, there is no need to provide a dedicated device for adjusting the position of the measurement unit.

The various aspects disclosed in this specification employ different technical means to achieve their respective objectives. The claims and the reference symbols in parentheses in this section are illustrative of the corresponding relationships with the embodiments described below, and are not intended to limit the technical scope. The objectives, features, and advantages disclosed in this specification will become clearer with reference to the detailed description that follows and the accompanying drawings.

FIG. 1 is a diagram showing a schematic configuration of a recognition system. FIG. 2 is a block diagram relating to control of the recognition system. 11 is a flowchart relating to control of the recognition system. FIG. 13 is a diagram showing a schematic configuration of a recognition system according to a second embodiment. 10 is a flowchart relating to control of a recognition system in a second embodiment. FIG. 13 is a block diagram relating to control of the recognition system in the third embodiment. 13 is a flowchart relating to control of a recognition system in a third embodiment.

Several embodiments will be described with reference to the drawings. In several embodiments, functionally and/or structurally corresponding and/or associated parts may be given the same reference numerals or reference numerals that differ in the hundredth or higher digit. For corresponding and/or associated parts, the descriptions of other embodiments may be referred to.

1, a recognition system 1 is a system for recognizing the position and orientation of a measurement object 41, which has a known shape. The position and orientation of the measurement object 41 means the position and orientation of the measurement object 41. However, it is not always necessary to recognize both the position and orientation, and a configuration in which either the position or the orientation is recognized may be used.

The recognition system 1 performs shape measurement to calculate the three-dimensional coordinates of a measurement object 41 captured using the measurement unit 20 for multiple measurement points. Furthermore, the recognition system 1 holds model data that is data indicating the model shape of the measurement object 41. As a result, the recognition system 1 matches the three-dimensional coordinates obtained by the shape measurement with the model data to recognize the position and orientation of the measurement object 41. The recognition system 1 may perform matching using multiple different types of model data. This makes it possible to perform a discrimination task of determining which part the measurement object 41 is by identifying which model data the measurement object 41 matches. Details of the measurement unit 20 will be described later.

The recognition system 1 can be used for various tasks using a robot 10. Examples of tasks using a robot 10 include an assembly task in which a part is assembled in a specific location, a picking task in which a part is picked, a visual inspection in which a part is inspected for scratches or chips, and a gripping task in which a part is gripped. However, the recognition system 1 may also be used for tasks that do not use a robot 10.

The robot 10 comprises a base 11, a robot arm 15, and a robot hand 16. The robot 10 is a vertical articulated robot. However, a horizontal articulated robot or the like may also be used as the robot 10. The base 11 is fixed to the installation surface on which the robot 10 is installed using bolts or the like.

The robot arm 15 is configured by connecting multiple shaft parts 13 via joint parts 12. The joint parts 12 include a motor, and the angle of the joint part 12 can be freely controlled by controlling the motor. The multiple shaft parts 13 are connected to each other via the joint parts 12 so that they can rotate relative to each other. The robot arm 15 is connected to the base 11.

The robot hand 16 is attached to the tip of the robot arm 15. The robot hand 16 includes a motor. The robot hand 16 is configured to be able to rotate relatively around the axis of the shaft part 13 provided on the robot arm 15 by controlling the motor.

The robot hand 16 has a claw portion. This claw portion rotates around the rotation axis of the robot hand 16. The robot hand 16 is configured to be able to grasp a measurement object 41 placed on the work table 31 by performing an opening and closing operation to expand and contract the distance between the claw portions. Furthermore, the robot 10 can move the robot arm 15 and the robot hand 16 with the robot hand 16 grasping the measurement object 41. This makes it possible to change the position and orientation of the measurement object 41. However, in order to grasp the measurement object 41, it is necessary to accurately recognize the position and orientation of the measurement object 41.

The measurement object 41 is placed on the work table 31. The work table 31 is a rectangular plate member. However, the type and shape of the work table 31 are not limited to the above example. A box-shaped parts box may be used as the work table 31. A movable device such as a belt conveyor may also be used as the work table 31. The number of measurement objects 41 placed on the work table 31 is not limited to one. For example, multiple measurement objects 41 may be placed on the work table 31 in a state where they overlap each other. The measurement object 41 is located at a standard position set approximately in the center of the work table 31. The standard position is a standard position where the measurement object 41 is placed. The standard position is set in advance in the recognition system 1.

The measurement object 41 can be an object of any shape, but as an example, an object in the shape of two disks of different diameters arranged side by side on the same axis is used. The measurement object 41 is placed on the work table 31 so that the disk part with the smaller diameter is located on the upper side.

The measurement unit 20 is a device for measuring the shape of the measurement object 41. The measurement unit 20 has a function for capturing a surface image of the measurement object 41 and a function for calculating a three-dimensional point cloud of the measurement object 41. A three-dimensional point cloud is data obtained by calculating the three-dimensional coordinates at multiple measurement points of the measurement object 41. In other words, the measurement points are each point that expresses the coordinates of the measurement object 41. The three-dimensional coordinates have X, Y, and Z coordinates as coordinate components.

Shape measurement by the measurement unit 20 includes two-dimensional measurement to obtain a surface image and three-dimensional measurement to obtain three-dimensional coordinates. The two-dimensional measurement and the three-dimensional measurement can be performed simultaneously in parallel with each other. The measurement unit 20 can be, for example, a distance image sensor that calculates distance using the principle of triangulation.

In two-dimensional measurement, the measurement unit 20 captures a surface image of the top surface of the measurement object 41. The amount of light reflected from the surface of the measurement object 41 varies depending on the shape of the measurement object 41. For this reason, the brightness in the surface image varies for each pixel depending on the shape of the measurement object 41. More specifically, at the edge parts of the measurement object 41, the brightness in the surface image varies significantly compared to the surrounding brightness. Also, at the flat parts of the measurement object 41, the brightness in the surface image varies very little compared to the surrounding brightness. An example of an edge part is the outer edge part of the measurement object 41. An example of an edge part is a step part such as a rib or an opening.

The surface image includes a set of two-dimensional coordinates and brightness information for each pixel that constitutes the image. Each pixel that constitutes the surface image corresponds to each measurement point on the measurement object 41. The brightness information is information about the brightness of the reflected light that is reflected by the surface of the measurement object 41 and received by the measurement unit 20.

The information that the surface image has for each pixel in a set with two-dimensional coordinates is not limited to luminance information. For example, the surface image may have a set of two-dimensional coordinates and distance information for each pixel. Distance information is information on the distance from the reference point of the measurement unit 20 to the measurement point. For example, the surface image may have a set of two-dimensional coordinates and edge information for each pixel. Edge information is information indicating whether the measurement point is an edge portion. The surface image may also be configured to have multiple pieces of information, luminance information, distance information, and edge information, in a set with two-dimensional coordinates.

The measurement unit 20 calculates a three-dimensional point cloud on the measurement object 41. The three-dimensional point cloud on the measurement object 41 can be obtained by a method using the principle of triangulation such as the phase shift method. When the phase shift method is adopted, the measurement unit 20 is equipped with a light projecting device 21 that projects a pattern light on the measurement object 41, and a light receiving device 22 that receives the pattern light reflected by the measurement object 41. The details of the light projecting device 21 and the light receiving device 22 will be described later. In the phase shift method, a pattern light such as a striped pattern is projected on the measurement object 41, and the pattern light reflected by the measurement object 41 is received. At this time, the phase of the projected pattern light is changed and the projection and reception are repeated. As a result, the three-dimensional shape of the measurement object 41 is measured from the difference between the projected pattern light and the received pattern light.

As one type of phase shift method, a color phase shift method can be used in which a color pattern light is projected from the light projector 21. A color pattern light is a pattern light that combines light of different wavelengths and different phases, such as red, green, and blue. In a monochromatic pattern light, the luminance of one of the colors red, green, and blue varies sinusoidally in one direction of the image, and the luminance is constant in the direction perpendicular to that direction.

In addition, the color pattern light has three monochromatic pattern lights whose phases are shifted by a predetermined angle. As an example, the red monochromatic pattern light has the most advanced phase, and the green monochromatic pattern light lags behind by 2π/3. The blue monochromatic pattern light lags behind the green monochromatic pattern light by an additional 2π/3 in phase. In the color phase shift method, instead of repeatedly projecting and receiving pattern light, the phase of the luminance in the direction of luminance change of the color pattern light is changed to project and receive light. The following describes an example of calculating three-dimensional coordinates using the color phase shift method.

The measurement unit 20 calculates the three-dimensional coordinates of the measurement points on the entire top surface of the measurement object 41 to generate a three-dimensional point cloud. Therefore, the three-dimensional coordinates are calculated for multiple measurement points including the edge and flat portions of the measurement object 41.

The angle of view at the time of imaging in the two-dimensional measurement is made to match the angle of view at the time of imaging in the three-dimensional measurement. Furthermore, the position and orientation of the measurement object 41 at the time of imaging in the two-dimensional measurement is made to match the position and orientation of the measurement object 41 at the time of imaging in the three-dimensional measurement. With this configuration, the surface image captured by the measurement unit 20 and the calculated three-dimensional coordinates are made to correspond in a pixel-by-pixel coordinate system. In other words, the pixels of the surface image obtained by the two-dimensional measurement are made to correspond to the three-dimensional coordinates obtained by the three-dimensional measurement. For example, it can be said that the two-dimensional coordinates (x1, y1) of the surface image and the three-dimensional coordinates (x1, y1, z1) are made to correspond in a pixel-by-pixel coordinate system. This is because the X-coordinate component of the surface image and the X-coordinate component of the three-dimensional coordinates match at x1, and the Y-coordinate component of the surface image and the Y-coordinate component of the three-dimensional coordinates match at y1.

The measurement unit 20 is provided on the robot hand 16. Therefore, the position of the measurement unit 20 can be adjusted by controlling the positions of the robot arm 15 and the robot hand 16. In other words, the direction and position of light projection and light reception by the measurement unit 20 can be adjusted by controlling the movement of the robot 10.

In FIG. 2, the control unit 70 that controls the recognition system 1 is connected to the robot 10 and the measurement unit 20. The control unit 70 acquires the rotational position of each rotation axis of the robot 10, etc. The control unit 70 also controls the movement of the robot 10.

The measurement unit 20 includes a light-projecting device 21 and a light-receiving device 22. The light-projecting device 21 may be a projector capable of projecting a color pattern light used in the color phase shift method. It is preferable that the light projected from the light-projecting device 21 be distinguishable from fluorescent light and sunlight in terms of intensity and wavelength.

The light receiving device 22 may be a camera that captures an image of the measurement target 41 with color pattern light projected onto it. The light receiving device 22 may be a camera that has a large number of light detection elements, such as photodiodes, arranged vertically and horizontally on the light receiving surface. Each light detection element of the light receiving device 22 corresponds to one pixel. The light receiving device 22 is a device that can receive light of at least the same wavelength as the wavelength of the light projected by the light projecting device 21.

The measurement unit 20 has the light-projecting device 21 and the light-receiving device 22 housed in a single housing. Therefore, the distance between the light-projecting device 21 and the light-receiving device 22 is always constant. However, the light-projecting device 21 and the light-receiving device 22 may be housed in separate housings. In this case, it is possible to adopt a configuration in which only one of the light-projecting device 21 and the light-receiving device 22 is provided on the robot 10. The control unit 70 acquires the measurement results of the shape measurement performed by the measurement unit 20. The control unit 70 controls the measurement unit 20 to perform the shape measurement.

The control unit 70 includes a measurement execution unit 71, a data acquisition unit 72, a two-dimensional extraction unit 73, a three-dimensional extraction unit 74, and a recognition execution unit 79. The measurement execution unit 71 executes shape measurement including two-dimensional measurement and three-dimensional measurement using the measurement unit 20. The data acquisition unit 72 acquires the measurement results of the shape measurement using the measurement unit 20. The measurement results include a surface image and a three-dimensional point cloud.

The two-dimensional extraction unit 73 extracts two-dimensional edge information from the surface image obtained by the two-dimensional measurement. The two-dimensional edge information is information that indicates the portion of the captured image where the edge portion of the measurement object 41 is imaged. The two-dimensional edge information does not include the portion where the planar portion of the measurement object 41 is imaged. The three-dimensional extraction unit 74 extracts three-dimensional edge information from the three-dimensional point cloud obtained by the three-dimensional measurement. The three-dimensional edge information is information that indicates the portion of the three-dimensional point cloud that forms the edge portion of the measurement object 41. The three-dimensional edge information does not include the three-dimensional coordinates that form the planar portion of the measurement object 41.

The recognition execution unit 79 performs position and orientation recognition by matching the three-dimensional coordinates of the three-dimensional edge information calculated by the three-dimensional extraction unit 74 with the model data. The more three-dimensional coordinates used for matching, the higher the matching accuracy. On the other hand, the fewer the three-dimensional coordinates used for matching, the shorter the time required for matching. For this reason, it is preferable to adjust the number of three-dimensional coordinates used for matching, taking into account the matching accuracy and the time required for matching.

The control unit 70 includes a robot operation unit 81. The robot operation unit 81 controls the opening and closing of the claws of the robot hand 16, and performs operations such as gripping the measurement target 41 whose position and orientation have been recognized.

The flow of position and orientation recognition for recognizing the position and orientation of the measurement object 41 using the recognition system 1 will be described below. In FIG. 3, when control related to position and orientation recognition is started, for example by turning on the power of the robot 10 by the user, the measurement execution unit 71 executes shape measurement of the measurement object 41 in step S101. More specifically, a surface image is captured by two-dimensional measurement, and a three-dimensional point cloud is calculated by three-dimensional measurement. The following describes an example in which the surface image includes a set of two-dimensional coordinates and brightness information. After executing the shape measurement, proceed to step S111.

In step S111, the data acquisition unit 72 acquires the measurement results obtained by shape measurement. More specifically, surface image data obtained by two-dimensional measurement and three-dimensional point cloud data obtained by three-dimensional measurement are acquired. After acquiring the measurement results, the process proceeds to step S115.

In step S115, the two-dimensional extraction unit 73 extracts two-dimensional edge information from the surface image. More specifically, it determines whether or not each pixel in the surface image is an edge portion from the luminance of the pixel, and extracts the pixels of the edge portion of the measurement object 41. A portion where the difference in luminance between adjacent pixels is equal to or greater than a predetermined edge luminance difference can be determined to be an edge portion, and a portion where the difference is less than the edge luminance difference can be determined to be a flat portion. After extracting the two-dimensional edge information, proceed to step S135.

In step S135, the three-dimensional extraction unit 74 extracts three-dimensional edge information, which is the three-dimensional coordinates of the edge portion, from the three-dimensional point cloud. Whether or not it is the three-dimensional coordinates of the edge portion is determined based on the two-dimensional edge information. More specifically, the three-dimensional coordinates of the positions corresponding to the pixels included in the two-dimensional edge information are left, and the other three-dimensional coordinates are removed from the three-dimensional point cloud to extract the three-dimensional edge information. After extracting the three-dimensional edge information, proceed to step S141.

In step S141, the recognition execution unit 79 performs position and orientation recognition. More specifically, the three-dimensional edge information is matched with the model data. In this matching, the three-dimensional coordinates of the edge portion of the measurement object 41 are matched with the three-dimensional coordinates of the edge portion of the model data. For example, after matching the coordinates of one specific point, the process of matching the coordinates of points adjacent to that point is repeated to match the coordinates of all points. However, since the three-dimensional edge information contains some measurement error, it is not necessary to match the coordinates of each point perfectly. For example, the matching result may be the position and orientation in which the sum of the distances between corresponding points in the model data and the three-dimensional edge information is minimized.

In the position and orientation recognition, the matching accuracy may be changed in stages. For example, the position and orientation recognition can be divided into first recognition and second recognition and executed in two stages. The first recognition is matching for recognizing the rough position and orientation of the measurement object 41. The second recognition is a process performed after the completion of the first recognition, and is matching for making fine adjustments from the rough position. This makes it easier to recognize the accurate position and orientation smoothly compared to a configuration in which the matching accuracy is not changed in stages. Since the first recognition uses three-dimensional edge information, it is easier to recognize the rough position and orientation with high accuracy compared to a case in which the rough position and orientation is recognized using two-dimensional edge information. In addition, since the second recognition uses three-dimensional edge information, it is easier to recognize the position and orientation quickly compared to a case in which fine adjustments are made using a three-dimensional point cloud including a planar portion. After executing the position and orientation recognition, a series of controls related to the position and orientation recognition is terminated.

After the series of position and orientation recognition steps are completed, the robot operation unit 81 operates the robot 10 to perform the necessary tasks. More specifically, the robot operation unit 81 performs tasks such as gripping the measurement object 41 whose position and orientation have been recognized. After the task using the robot 10 is completed, if there are other measurement objects 41 whose position and orientation need to be recognized, control related to the position and orientation recognition is repeated.

The effects of the above-described embodiment will be described below. According to the above-described embodiment, the three-dimensional extraction unit 74 extracts three-dimensional edge information from the three-dimensional point cloud based on the two-dimensional edge information. As a method for determining whether or not an edge portion is present from three-dimensional coordinates without using two-dimensional edge information, a method of calculating the distance between adjacent measurement points for all measurement points in the three-dimensional point cloud can be assumed. However, the method of calculating the distance between adjacent measurement points for all measurement points in the three-dimensional point cloud requires a large amount of distance data to be calculated, and the processing load is likely to be large. For this reason, by extracting three-dimensional edge information using two-dimensional edge information, it is possible to extract three-dimensional edge information at a high speed compared to the case where three-dimensional edge information is extracted without using two-dimensional edge information. Therefore, the extraction of three-dimensional edge information in the control of position and orientation recognition can be completed at a high speed. Therefore, a recognition system 1 capable of recognizing position and orientation at a high speed can be provided.

The two-dimensional extraction unit 73 extracts two-dimensional edge information by determining whether or not an edge portion is present from the luminance information for each two-dimensional coordinate. Therefore, it is not necessary to calculate the distance for each pixel of the surface image in two-dimensional measurement and determine whether or not an edge portion is present from the difference in distance. Therefore, two-dimensional edge information can be extracted at high speed. In particular, even if the unevenness of the edge portion relative to the flat portion is small, it is easy to accurately determine whether or not an edge portion is present.

The measurement unit 20 calculates a three-dimensional point cloud based on the color pattern light received by the light receiving device 22. This eliminates the need to change the phase of the pattern light and repeat light projection and reception multiple times. This makes it possible to quickly complete shape measurement in position and orientation recognition control.

The measurement unit 20 is provided on the robot 10. Therefore, the position of the measurement unit 20 during shape measurement can be adjusted by controlling the movement of the robot 10. Therefore, there is no need to provide a dedicated device for the purpose of adjusting the position of the measurement unit 20. In addition, the position of the measurement unit 20 can be adjusted according to the size and position of the measurement object 41. Therefore, by placing the measurement unit 20 in a position where it is easy to measure the measurement object 41, it is easy to improve the accuracy of the shape measurement.

In addition, when the measurement unit 20 is provided on the robot 10, the robot 10 will repeat movements for shape measurement and movements for tasks such as grasping. For this reason, quickly completing position and orientation recognition, including shape measurement, is extremely important in order to quickly start tasks such as grasping using the robot 10.

Second Embodiment This embodiment is a modification based on the preceding embodiment, in which a two-dimensional extraction unit 73 determines whether or not an edge portion is present from both the luminance information and the distance information for each two-dimensional coordinate, and extracts two-dimensional edge information.

In FIG. 4, the measurement unit 220 is fixed to the upper side of the work table 31 and the measurement object 41 independently of the robot 10. Therefore, shape measurement can be performed using the measurement unit 220 regardless of the movement of the robot 10.

The flow of position and orientation recognition for recognizing the position and orientation of the measurement object 41 using the recognition system 1 will be described below. In FIG. 5, when control related to position and orientation recognition is started, for example when the power of the robot 10 is turned on by the user, the measurement execution unit 71 executes shape measurement of the measurement object 41 in step S101. Then, the process proceeds to step S111, where the data acquisition unit 72 acquires the measurement results obtained by the shape measurement. Then, the process proceeds to step S212.

In step S212, the control unit 70 calculates distance information for each pixel of the surface image based on the three-dimensional coordinates. More specifically, the control unit 70 calculates the distance between the three-dimensional coordinates corresponding to the pixel of the surface image and the reference point of the measurement unit 220 for each pixel. This results in the surface image having a set of two-dimensional coordinates, brightness information, and distance information for each pixel. After calculating the distance information, proceed to step S215.

In step S215, the two-dimensional extraction unit 73 extracts two-dimensional edge information from the surface image. More specifically, it determines whether or not it is an edge portion from both the brightness information and distance information contained in the surface image, and extracts pixels of the edge portion of the measurement object 41. A portion where the brightness difference between adjacent pixels is equal to or greater than a predetermined edge brightness difference is likely to be an edge portion, and if it is less than the edge brightness difference, it can be determined that it is unlikely to be an edge portion. Furthermore, a portion where the distance difference between adjacent pixels is equal to or greater than a predetermined edge distance difference is likely to be an edge portion, and if it is less than the edge distance difference, it can be determined that it is unlikely to be an edge portion.

A part that is determined to be highly likely to be an edge part based on both the luminance information and the distance information is determined to be an edge part. On the other hand, a part that is determined to be unlikely to be an edge part based on both the luminance information and the distance information is determined to be a flat part. Furthermore, if the judgment results based on the luminance information and the distance information differ, the part is determined to be an unknown part.

When extracting two-dimensional edge information, the parts determined to be planar or unclear parts are removed, and only the parts determined to be edge parts are left, thereby extracting the two-dimensional edge information. However, if the number of edge parts extracted as the two-dimensional edge information is extremely small, the recognition accuracy may be reduced. For this reason, the two-dimensional edge information may be extracted by removing the parts determined to be planar parts, and leaving the parts determined to be edge parts or unclear parts. After extracting the two-dimensional edge information, proceed to step S135.

In step S135, the three-dimensional extraction unit 74 extracts three-dimensional edge information, which is the three-dimensional coordinates of the edge portion, from the three-dimensional point cloud. Then, the process proceeds to step S141, where the recognition execution unit 79 executes position and orientation recognition. After executing position and orientation recognition, control related to position and orientation recognition is terminated, and as necessary, a task using the robot 10, such as a grasping task, is performed.

The effects of the above-described embodiment are described below. According to the above-described embodiment, the two-dimensional extraction unit 73 extracts two-dimensional edge information by determining whether or not an edge portion is present from both the luminance information and the distance information for each two-dimensional coordinate. For this reason, it is easier to adjust the accuracy of determining whether or not an edge portion is present, compared to when determining whether or not an edge portion is present based on either the luminance information or the distance information alone. For example, by not including an unknown portion in an edge portion, it is possible to prevent a portion determined to be an edge portion from erroneously including a flat portion, making it easier to increase the accuracy of the two-dimensional edge information. On the other hand, by including an unknown portion in an edge portion, it is possible to increase the number of measurement points determined to be edge portions, making it easier to secure a large amount of data for the two-dimensional edge information.

Although the above description takes as an example a case where distance information is calculated from three-dimensional coordinates, the measurement unit 220 may be configured to obtain brightness information and distance information by two-dimensional measurement. In this case, the brightness information and distance information can be obtained independently of the three-dimensional measurement.

Third embodiment This embodiment is a modification of the preceding embodiment as a basic form. In this embodiment, the two-dimensional extraction unit 73 judges whether or not an edge portion is present from both the luminance information and the distance information for each two-dimensional coordinate, and extracts two-dimensional edge information. In addition, three-dimensional measurement is performed on the edge portion extracted as the two-dimensional edge information, and three-dimensional edge information is calculated.

In FIG. 6, the control unit 70 includes a three-dimensional calculation unit 374. The three-dimensional calculation unit 374 calculates a three-dimensional point cloud based on three-dimensional measurements performed only on the edge portion of the measurement object 41.

The flow of position and orientation recognition for recognizing the position and orientation of the measurement object 41 using the recognition system 1 will be described below. In FIG. 7, when control related to position and orientation recognition is started, for example by turning on the power of the robot 10 by the user, the measurement execution unit 71 executes two-dimensional measurement of the measurement object 41 in step S301. More specifically, a surface image is captured by the two-dimensional measurement. In the following, an example will be described in which the surface image includes a set of two-dimensional coordinates and distance information. After executing the shape measurement, proceed to step S311.

In step S311, the data acquisition unit 72 acquires the measurement results obtained by the two-dimensional measurement. Since the surface image includes a set of two-dimensional coordinates and distance information, information on the distance from the reference point of the measurement unit 20 to the measurement point is obtained for each pixel. After acquiring the information measurement results, proceed to step S315.

In step S315, the two-dimensional extraction unit 73 extracts two-dimensional edge information from the surface image. More specifically, it determines whether or not the pixel is an edge portion from the distance information included in the surface image, and extracts pixels of the edge portion of the measurement object 41. A portion where the difference in distance between adjacent pixels is equal to or greater than a predetermined edge distance difference is an edge portion, and a portion where the difference is less than the edge distance difference can be determined to be a flat portion.

When determining whether or not an edge part is present, the larger the edge distance difference is set, the less likely it is to be determined as an edge part, and the smaller the data amount of the two-dimensional edge information. On the other hand, the smaller the edge distance difference is set, the more likely it is to be determined as an edge part, and the larger the data amount of the two-dimensional edge information. For this reason, by adjusting the magnitude of the edge distance difference, it is possible to increase or decrease the part determined as an edge part. After extracting the two-dimensional edge information, proceed to step S321.

In step S321, the measurement execution unit 71 performs three-dimensional measurement on the portion of the measurement object 41 that is determined to be an edge portion. In other words, three-dimensional measurement is performed on a portion of the measurement object 41, rather than on the entire measurement object 41. More specifically, the three-dimensional coordinates of a position corresponding to a pixel included in the two-dimensional edge information are calculated.

In step S335, the three-dimensional extraction unit 74 calculates three-dimensional edge information. Only the three-dimensional coordinates of pixels corresponding to pixels included in the two-dimensional edge information are calculated by the three-dimensional measurement. Therefore, the three-dimensional coordinates obtained by the three-dimensional measurement can be used as the three-dimensional edge information as is. After calculating the three-dimensional edge information, proceed to step S141.

In step S141, the recognition execution unit 79 performs position and orientation recognition by matching the three-dimensional edge information with the model data. After control related to position and orientation recognition is completed, a task using the robot 10, such as a grasping task, is performed as necessary.

The effects of the above-described embodiment are described below. According to the above-described embodiment, the two-dimensional extraction unit 73 extracts two-dimensional edge information by determining whether or not an area is an edge from distance information for each two-dimensional coordinate. This makes it possible to determine whether or not an area is an edge from information other than luminance information. In addition, by adjusting the edge distance difference used to determine whether or not an area is an edge, it is easy to adjust the amount of data in the two-dimensional edge information.

The three-dimensional calculation unit 374 calculates a three-dimensional point cloud for a portion of the measurement object 41 that is determined to be an edge portion based on the two-dimensional edge information, and does not calculate a three-dimensional point cloud for a portion that is determined not to be an edge portion. This makes it easier to calculate three-dimensional edge information quickly compared to a case where a three-dimensional point cloud for the entire measurement object 41, including the planar portions, is calculated and then three-dimensional edge information is extracted. This makes it possible to provide a recognition system 1 that is capable of recognizing position and orientation quickly.

Other embodiments The disclosure in this specification and drawings, etc. is not limited to the exemplified embodiments. The disclosure includes the exemplified embodiments and modifications by those skilled in the art based thereon. For example, the disclosure is not limited to the combination of parts and/or elements shown in the embodiments. The disclosure can be implemented by various combinations. The disclosure can have additional parts that can be added to the embodiments. The disclosure includes the omission of parts and/or elements of the embodiments. The disclosure includes the substitution or combination of parts and/or elements between one embodiment and another embodiment. The disclosed technical scope is not limited to the description of the embodiments. Some disclosed technical scopes are indicated by the description of the claims, and should be interpreted as including all modifications within the meaning and scope equivalent to the description of the claims.

The disclosure in the specification and drawings, etc. is not limited by the claims. The disclosure in the specification and drawings, etc. encompasses the technical ideas described in the claims, and extends to more diverse and extensive technical ideas than the technical ideas described in the claims. Therefore, various technical ideas can be extracted from the disclosure in the specification and drawings, etc., without being bound by the claims.

The control unit and the method described in the present disclosure may be realized by a dedicated computer comprising a processor programmed to execute one or more functions embodied in a computer program. Alternatively, the device and the method described in the present disclosure may be realized by a dedicated hardware logic circuit. Alternatively, the device and the method described in the present disclosure may be realized by one or more dedicated computers configured by a combination of a processor that executes a computer program and one or more hardware logic circuits. In addition, the computer program may be stored in a computer-readable non-transient tangible recording medium as instructions executed by the computer.

1 Recognition system, 10 Robot, 12 Joint part, 13 Axis part, 15 Robot arm, 16 Robot hand, 20 Measurement unit, 21 Light projector, 22 Light receiver, 41 Measurement object, 70 Control unit, 71 Measurement execution unit, 72 Data acquisition unit, 73 Two-dimensional extraction unit, 74 Three-dimensional extraction unit, 79 Recognition execution unit, 81 Robot operation unit, 220 Measurement unit, 374 Three-dimensional calculation unit

Claims (5)

A recognition system for performing position and orientation recognition for recognizing a position and orientation of a measurement object (41), comprising:
a measurement unit (20, 220) that captures a surface image of the measurement object and calculates three-dimensional coordinates of positions corresponding to pixels of the surface image as a three-dimensional point cloud of the measurement object;
a two-dimensional extraction unit (73) that extracts two-dimensional edge information that is information indicating a position of an edge portion in the measurement object based on the surface image;
a three-dimensional extraction unit (74) that extracts three-dimensional edge information that is information indicating the position of the edge portion in the three-dimensional point cloud;
a recognition execution unit (79) for recognizing a position and orientation of the measurement object based on the three-dimensional edge information;
The three-dimensional extraction unit extracts the three-dimensional edge information from the three-dimensional point cloud based on the two-dimensional edge information,
the measurement unit acquires distance information indicating a distance between the measurement object and the measurement unit for each pixel of the surface image;
The two-dimensional extraction unit is a recognition system that determines whether or not an object is an edge portion from the distance information and extracts the two-dimensional edge information . A recognition system for performing position and orientation recognition for recognizing a position and orientation of a measurement object (41), comprising:
a measurement unit (20, 220) that captures a surface image of the measurement object and calculates three-dimensional coordinates of positions corresponding to pixels of the surface image as a three-dimensional point cloud of the measurement object;
a two-dimensional extraction unit (73) that extracts two-dimensional edge information that is information indicating a position of an edge portion in the measurement object based on the surface image;
a three-dimensional extraction unit (74) that extracts three-dimensional edge information that is information indicating the position of the edge portion in the three-dimensional point cloud;
a recognition execution unit (79) for recognizing a position and orientation of the measurement object based on the three-dimensional edge information;
The three-dimensional extraction unit extracts the three-dimensional edge information from the three-dimensional point cloud based on the two-dimensional edge information,
the measurement unit acquires, for each pixel of the surface image, luminance information indicating luminance and distance information indicating a distance between the measurement object and the measurement unit;
The two-dimensional extraction unit extracts the two-dimensional edge information by determining whether or not the object is an edge portion from both the luminance information and the distance information . A recognition system for performing position and orientation recognition for recognizing a position and orientation of a measurement object (41), comprising:
a measurement unit (20, 220) capable of capturing a surface image of the measurement object and calculating a three-dimensional point cloud at positions corresponding to pixels of the surface image;
A control unit (70) that controls the recognition system,
The control unit is
A shape measurement is performed by the measurement unit ;
extracting two-dimensional edge information that is information indicating a position of an edge portion of the measurement object based on the surface image;
Calculating three-dimensional edge information, which is the three-dimensional point cloud of the edge portion ;
Recognizing the position and orientation of the measurement object based on the three-dimensional edge information;
The control unit performs two-dimensional measurement, which is the shape measurement to extract the two-dimensional edge information, and then performs three-dimensional measurement, which is the shape measurement to calculate the three-dimensional edge information, and calculates the three-dimensional point cloud of pixels corresponding to pixels of the measurement object in the three-dimensional measurement that are determined to be the edge portion based on the two-dimensional edge information, and does not calculate the three-dimensional point cloud of pixels corresponding to pixels that are determined not to be the edge portion. 4. The recognition system according to claim 1, wherein the measurement unit includes a light-projecting device (21) that projects a color pattern light, which is a combination of a plurality of lights having different wavelengths and different phases of luminance change, onto the measurement object, and a light-receiving device (22) that receives the color pattern light reflected by the measurement object, and calculates the three -dimensional point cloud based on the color pattern light received by the light-receiving device. The robot (10) includes a plurality of shaft parts (13) connected to each other via joint parts (12) so as to be capable of rotating relative to each other;
The recognition system according to claim 1 , wherein the measurement unit is provided on the robot.

Images