JP7724056B2 - How to generate data for machine learning - Google Patents

Description

The present invention relates to a method for generating data for machine learning.

In object grasping systems, which use a robotic arm to grasp multiple loosely stacked objects (workpieces), artificial intelligence such as convolutional neural networks may be used to recognize the type, position, and orientation of an object from photographed images. To ensure the proper use of systems that use artificial intelligence, machine learning using a huge amount of training data is essential, and evaluation data is also required to assess whether the results of the machine learning are appropriate. This results in significant costs for preparing the data.

Japanese Patent Application Laid-Open No. 2020-103284 JP 2008-281399 A

Creating training data or evaluation data requires captured images of the object, as well as the relative position and orientation of the camera and the object. One method for identifying the object's position and orientation is to measure its three-dimensional shape using a 3D scanner camera or similar device and match the results with three-dimensional data of the object previously created using CAD or similar software. However, in the case of a group of objects containing multiple loosely stacked objects, overlapping objects can make it difficult to match the three-dimensional shape measurement results of each object with the three-dimensional data. For example, if multiple objects are placed close together, the three-dimensional shape of the combined objects may be mistakenly matched with the three-dimensional data of the object, making it difficult to accurately measure the object's position and orientation.

The present invention has been made in consideration of the above, and aims to provide a method for generating machine learning data that can accurately acquire image data including multiple objects.

In order to solve the above-mentioned problems and achieve the object, a method for generating data for machine learning according to one aspect of the present invention includes: irradiating a first object coated with a first paint that emits light using first visible light when irradiated with light having a wavelength shorter than that of visible light and that is colorless and transparent to visible light, and a second object coated with a second paint that emits light using second visible light when irradiated with light having a wavelength shorter than that of visible light and that is colorless and transparent to visible light, with light having a wavelength shorter than that of visible light in a predetermined pattern; photographing the first object and the second object with a shape acquisition camera; and determining the shape of the first object from the photographed image. The method includes a first step of measuring a three-dimensional shape of an object and a three-dimensional shape of a second object, a second step of irradiating the first object and the second object with visible light and photographing the first object and the second object with a position acquisition camera that is a stereo camera , and a third step of separating the three-dimensional shape of the first object and the three-dimensional shape of the second object from the three-dimensional shapes measured in the first step based on a difference between the first visible light and the second visible light, and matching the three-dimensional data of the first object and the three-dimensional data of the second object with pre-stored three-dimensional data of the first object and the second object. The direction in which the first object and the second object are photographed is changed, and the measurement of the three-dimensional shapes of the first object and the second object in the first step and the photographing of the first object and the second object in the second step are repeated, and the data is linked and updated by point cloud data matching. The relative position or orientation of the object with respect to the position acquisition camera is calculated by measuring the three-dimensional shape of the first object and the three-dimensional shape of the second object in the first step, photographing the first object and the second object in the second step, and separating the point cloud data of the first object and the second object and matching the point cloud data with the three-dimensional data of the object in the third step.

According to one aspect, image data containing multiple objects can be accurately acquired.

FIG. 1 is an external view showing an example of an object grasping system. FIG. 2 is a block diagram showing an example of the configuration of an object grasping system. FIG. 3 is a diagram illustrating an example of a process related to control of a robot arm. FIG. 4 is a diagram illustrating another example of processing related to control of a robot arm. FIG. 5 is a diagram illustrating an example of a detection model. FIG. 6 is a diagram showing an example of a feature map output by the feature detection layer (u1). FIG. 7 is a diagram showing an example of the estimation result of the position and orientation of the object. FIG. 8 is a diagram showing another example of the estimation result of the grip position of the target object. FIG. 9 is a diagram showing an example of a bulk pile image captured by a stereo camera. FIG. 10 is a diagram showing an example of the relationship between the bulk pile image and the matching map. FIG. 11 is a flowchart illustrating an example of the estimation process. FIG. 12 is a diagram illustrating an example of the estimation process. FIG. 13 is a diagram showing an example of a bulk pile image including a tray according to a modified example. FIG. 14 is a diagram showing an example of a positional deviation estimation model according to a modified example. FIG. 15 is a diagram showing another example of a position shift estimation model according to a modified example. FIG. 16 is a block diagram showing an example of the configuration of a system for acquiring (measuring) three-dimensional data of a group of objects and generating a three-dimensional model of the group of objects. FIG. 17 is a diagram illustrating an example of the measurement process. FIG. 18 is a diagram showing an example of three-dimensional data of an object. FIG. 19 is a flowchart illustrating an example of the pre-processing. FIG. 20 is a diagram illustrating an example of the calibration process. FIG. 21 is a diagram showing an example of a plurality of objects to which paint is applied. FIG. 22 is a flowchart showing an example of a process for acquiring three-dimensional data of an object group. FIG. 23 is a diagram showing an example of a state in which a UV pattern is irradiated onto a group of subjects from a projector. FIG. 24 is a diagram showing an example of a state in which visible light is irradiated onto a group of subjects. FIG. 25 is a diagram showing an example of a captured image of a virtual space in which a group of objects is arranged. FIG. 26 is a block diagram showing an example of an information processing device that executes a measurement program.

Below, a method for generating data for machine learning and a device for generating data for machine learning according to an embodiment will be described with reference to the drawings. Note that the present invention is not limited to these embodiments. Furthermore, the dimensional relationships and ratios of elements in the drawings may differ from reality. Even between drawings, there may be parts where the dimensional relationships and ratios differ. Furthermore, the content described in one embodiment or variant applies, in principle, to other embodiments or variants as well.

(Object grasping system)
FIG. 1 is an external view showing an example of an object grasping system 1. The object grasping system 1 shown in FIG. 1 includes an image processing device 10, a camera 20, and a robot arm 30 (not shown). The camera 20 is provided, for example, in a position where it can capture images of both the robot arm 30 and bulk-stacked workpieces 41, 42, etc., which are objects to be grasped by the robot arm 30. The camera 20 captures images of the robot arm 30 and the workpieces 41, 42, for example, and outputs the images to the image processing device 10. Note that the robot arm 30 and the bulk-stacked workpieces 41, 42, etc. may be captured by separate cameras. Hereinafter, an image of a randomly arranged object may be referred to as a "bulk-stack image." As shown in FIG. 1, the camera 20 is a camera capable of capturing multiple images, such as a known stereo camera. The image processing device 10 uses the images output from the camera 20 to estimate the positions and orientations of the workpieces 41, 42, etc. The image processing device 10 outputs a signal to control the operation of the robot arm 30 based on the estimated positions and orientations of the workpieces 41, 42, etc. The robot arm 30 performs an operation to grip the workpieces 41, 42, etc. based on the signal output from the image processing device 10. Note that although FIG. 1 discloses multiple different types of workpieces 41, 42, etc., there may be only one type of workpiece. Here, a case where there is only one type of workpiece will be described. Furthermore, the workpieces 41, 42, etc. are arranged so that their positions and orientations are irregular. For example, as shown in FIG. 1, multiple workpieces may be arranged so that they overlap in a top view. Furthermore, the workpieces 41, 42 are an example of a target object.

Figure 2 is a block diagram showing an example of the configuration of the object grasping system 1. As shown in Figure 2, the image processing device 10 is communicatively connected to a camera 20 and a robot arm 30 via a network NW. Also as shown in Figure 2, the image processing device 10 includes a communication I/F (interface) 11, an input I/F 12, a display 13, a memory circuit 14, and a processing circuit 15.

The communication I/F 11 controls data input/output communication with external devices via the network NW. For example, the communication I/F 11 is implemented by a network card, network adapter, NIC (Network Interface Controller), etc., and receives image data output from the camera 20 and transmits signals to be output to the robot arm 30.

The input I/F 12 is connected to the processing circuit 15 and converts input operations received from an administrator (not shown) of the image processing device 10 into electrical signals and outputs them to the processing circuit 15. For example, the input I/F 12 is a switch button, mouse, keyboard, touch panel, etc.

The display 13 is connected to the processing circuit 15 and displays various information and image data output from the processing circuit 15. For example, the display 13 may be implemented as an LCD monitor, a CRT (Cathode Ray Tube) monitor, a touch panel, etc.

The memory circuitry 14 is realized, for example, by a storage device such as a memory. The memory circuitry 14 stores various programs executed by the processing circuitry 15. The memory circuitry 14 also temporarily stores various data used when the processing circuitry 15 executes the various programs. The memory circuitry 14 has a machine (deep) learning model 141. The machine (deep) learning model 141 further includes a neural network structure 141a and learning parameters 141b. The neural network structure 141a is, for example, an application of a known network such as the convolutional neural network b1 in FIG. 5, and is the network structure shown in FIG. 12, which will be described later. The learning parameters 141b are, for example, the weights of the convolutional filter of the convolutional neural network, and are parameters that are learned and optimized to estimate the position and orientation of an object. The neural network structure 141a may be provided in the estimation unit 152. Note that, although the machine (deep) learning model 141 in this invention will be described as a trained model, it is not limited thereto. In the following, the machine (deep) learning model 141 may be referred to simply as the "learning model 141."

The learning model 141 is used in the process of estimating the position and orientation of a workpiece from images output by the camera 20. The learning model 141 is generated, for example, by learning using the positions and orientations of multiple workpieces and images of the multiple workpieces as training data. Here, the learning model 141 is generated, for example, by the processing circuit 15, but is not limited to this and may also be generated by an external computer.

The processing circuitry 15 is implemented by a processor such as a CPU (Central Processing Unit). The processing circuitry 15 controls the entire image processing device 10. The processing circuitry 15 reads various programs stored in the memory circuitry 14 and executes the read programs to perform various processes. For example, the processing circuitry 15 includes an image acquisition unit 151, an estimation unit 152, and a robot control unit 153.

The image acquisition unit 151 acquires a bulk pile image, for example, via the communication I/F 11, and outputs it to the estimation unit 152. The image acquisition unit 151 is an example of an acquisition unit.

The estimation unit 152 estimates the position and orientation of the object using the output image of the bulk pile. The estimation unit 152 performs estimation processing on the image of the object using, for example, the learning model 141, and outputs the estimation result to the robot control unit 153. Note that the estimation unit 152 may also estimate, for example, the position and orientation of a tray or the like on which the object is placed. The configuration for estimating the position and orientation of the tray will be described later.

The robot control unit 153 generates a signal to control the robot arm 30 based on the estimated position and orientation of the object, and outputs it to the robot arm 30 via the communication I/F 11. The robot control unit 153, for example, acquires information regarding the current position and orientation of the robot arm 30. The robot control unit 153 then generates a trajectory for the robot arm 30 to move when grasping the object, based on the current position and orientation of the robot arm 30 and the estimated position and orientation of the object. The robot control unit 153 may also modify the trajectory for the robot arm 30 to move based on the position and orientation of a tray, etc.

Figure 3 is a diagram showing an example of processing related to the control of a robot arm. As shown in Figure 3, the estimation unit 152 estimates the position and orientation of a target object from the bulk pile image. Similarly, the estimation unit 152 may estimate the position and orientation of a tray or other object on which the object is placed from the bulk pile image. The robot control unit 153 calculates the coordinates and orientation of the position of the hand of the robot arm 30 based on the estimated models of the object and tray, etc., and generates a trajectory for the robot arm 30.

In addition, the robot control unit 153 may further output a signal to control the operation of the robot arm 30 to align the grasped objects after the robot arm 30 has grasped the objects. Figure 4 is a diagram showing another example of processing related to robot arm control. As shown in Figure 4, the image acquisition unit 151 acquires an image of the object grasped by the robot arm 30, taken by the camera 20. The estimation unit 152 estimates the position and posture of the target object grasped by the robot arm 30, and outputs the image to the robot control unit 153. The image acquisition unit 151 may also acquire an image of a destination tray, etc., taken by the camera 20, to which the grasped object will be moved. In this case, the image acquisition unit 151 further acquires an image (aligned image) of the objects already aligned on the destination tray, etc. From the destination image or the aligned image, the estimation unit 152 estimates the position and posture of the destination tray, etc., and the position and posture of the objects already aligned. The robot control unit 153 then calculates the coordinates and orientation of the position of the tip of the robot arm 30 based on the estimated position and orientation of the object held by the robot arm 30, the position and orientation of the tray or other object to be aligned, and the positions and orientations of objects that have already been aligned, and generates a trajectory for the robot arm 30 when aligning the objects.

Next, the estimation process performed by the estimation unit 152 will be described. The estimation unit 152 extracts features of an object using, for example, a model that applies known downsampling, upsampling, and skip connection object detection models. Figure 5 illustrates an example of a detection model. In the object detection model shown in Figure 5, the d1 layer divides, for example, a bulk pile image P1 (320 x 320 pixels) into a 40 x 40 grid by downsampling via a convolutional neural network b1, and calculates multiple features (e.g., 256 types) for each grid. The d2 layer, which is a layer below the d1 layer, divides the grid divided by the d1 layer more coarsely (e.g., into a 20 x 20 grid) and calculates the features of each grid. Similarly, the d3 and d4 layers, which are layers below the d1 and d2 layers, divide the grid divided by the d2 layer more coarsely. The d4 layer uses upsampling to calculate features at finer divisions, and at the same time, skip connections s3 are used to integrate these with the features of the d3 layer to generate the u3 layer. The skip connections may be simple additions or concatenations of features, or they may involve transformations such as those used in convolutional neural networks being applied to the features of the d3 layer. Similarly, the u2 layer is generated by integrating the features calculated by upsampling the u3 layer with the features of the d2 layer using skip connections s2. The u1 layer is then generated in a similar manner. As a result, in the u1 layer, features for each grid divided into a 40x40 grid are calculated, just like in the d1 layer.

Figure 6 shows an example of a feature map output by the feature detection layer (u1). The horizontal direction of the feature map shown in Figure 6 indicates each horizontal grid of the bulk pile image P1, which is divided into a 40 x 40 grid, and the vertical direction indicates each vertical grid. Furthermore, the depth direction of the feature map shown in Figure 6 indicates the feature element in each grid.

Figure 7 shows an example of the estimation results for the position and orientation of an object. As shown in Figure 7, the estimation unit outputs two-dimensional coordinates (Δx, Δy) indicating the position of the object, quaternions (qx, qy, qz, qw) indicating the orientation of the object, and classification scores (C0, C1, ..., Cn). Note that, among the coordinates indicating the position of the object, the depth value indicating the distance from the camera 20 to the object is not calculated as the estimation result. The configuration for calculating the depth value will be explained later. Note that depth here refers to the distance from the z coordinate of the camera to the z coordinate of the object in the z-axis direction parallel to the optical axis of the camera. Note that the classification score is a value output for each grid and is the probability that the center point of the object is included in that grid. For example, if there are n types of objects, the probability that the center point of the object is not included is added to this, and n+1 classification scores are output. For example, if there is only one type of object, two classification scores are output. Also, if there are multiple objects in the same grid, the probability of the object stacked higher is output.

In Figure 7, point C indicates the center of grid Gx, and point ΔC with coordinates (Δx, Δy) indicates, for example, the center point of a detected object. That is, in the example shown in Figure 7, the center of the object is offset from center point C of grid Gx by Δx in the x-axis direction and Δy in the y-axis direction.

Instead of Figure 7, as shown in Figure 8, arbitrary points a, b, and c other than the center of the object can be set, and the coordinates (Δx1, Δy1, Δz1, Δx2, Δy2, Δz2, x3, Δy3, Δz3) of the arbitrary points a, b, and c from point C, the center of grid Gx, can be output. The arbitrary points can be set anywhere on the object, and can be one point or multiple points.

Note that if the grid divisions are too coarse compared to the size of the objects, multiple objects may fit into one grid, causing the features of each object to blend together and result in false detection. Therefore, here we only use the feature map, which is the output of the feature extraction layer (u1), which calculates the fine (40x40 grid) features ultimately generated.

Here, the distance from the camera 20 to the object is determined by capturing two types of left and right images using, for example, a stereo camera. Figure 9 is a diagram showing an example of a bulk pile image captured by a stereo camera. As shown in Figure 9, the image acquisition unit 151 acquires two types of bulk pile images, a left image P1L and a right image P1R. The estimation unit 152 also performs estimation processing using the learning model 141 on both the left image P1L and the right image P1R. When performing the estimation processing, some or all of the learning parameters 141b used for the left image P1L may be shared as weighting for the right image P1R. Instead of a stereo camera, a single camera may be used, and images equivalent to two types of left and right images may be captured at two different positions by shifting the camera's position.

Here, the estimation unit 152 suppresses erroneous recognition of objects by using a matching map that combines the feature amounts of the left image P1L and the feature amounts of the right image P1R. The matching map indicates the strength of correlation between the feature amounts of the right image P1R and the left image P1L for each feature amount. In other words, by using the matching map, it is possible to match the left image P1L and the right image P1R by focusing on the feature amounts in each image.

Figure 10 shows an example of the relationship between a bulk pile image and a matching map. As shown in Figure 10, in the matching map ML, which is based on the left image P1L and corresponds to the right image P1R, the grid MLa that has the highest correlation between the feature of the grid containing the center point of the object W1L in the left image P1L and the feature of the right image P1R is highlighted. Similarly, in the matching map MR, which is based on the right image P1R and corresponds to the left image P1L, the grid MRa that has the highest correlation between the feature of the grid containing the center point of the object W1R in the right image P1R and the feature of the left image P1L is highlighted. Furthermore, the grid MLa with the highest correlation in the matching map ML corresponds to the grid where the object W1L in the left image P1L is located, and the grid MRa with the highest correlation in the matching map MR corresponds to the grid where the object W1R in the right image P1R is located. This makes it possible to determine that the grid on which object W1L is located in the left image P1L matches the grid on which object W1R is located in the right image P1R. That is, in FIG. 9, the matching grids are grid G1L in the left image P1L and grid G1R in the right image P1R. This makes it possible to determine the parallax for object W1 based on the X coordinate of object W1L in the left image P1L and the X coordinate of object W1R in the right image P1R, and therefore the depth z from camera 20 to object W1 can be determined.

Figure 11 is a flowchart showing an example of the estimation process. Figure 12 is a diagram showing an example of the estimation process. The following description will be made using Figures 9 to 12. First, the image acquisition unit 151 acquires left and right images of the object, such as the left image P1L and right image P1R shown in Figure 9 (step S201). Next, the estimation unit 152 calculates feature amounts for each grid in the horizontal direction of each of the left and right images. Here, as described above, if each image is divided into a 40 x 40 grid and 256 feature amounts are calculated for each grid, a 40-by-40 matrix is obtained in the horizontal direction of each image, as shown in the first and second terms on the left side of equation (1) (the product of the matrix of the first term and the matrix of the second term).

Next, the estimation unit 152 executes process m shown in FIG. 12. First, the estimation unit 152 calculates the matrix product of the feature quantities extracted from a specific column of the left image P1L and the transposed feature quantities extracted from the same column of the right image P1R, for example, using equation (1). In equation (1), the first term on the left side represents the feature quantities l11 to l1n in the first grid in the horizontal direction of a specific column of the left image P1L, arranged in the row direction. Meanwhile, the second term on the left side of equation (1) represents the feature quantities r11 to r1n in the first grid in the horizontal direction of a specific column of the right image P1R, arranged in the column direction. In other words, the matrix in the second term on the left side is the transposed matrix of the feature quantities r11 to r1m in the horizontal direction of a specific column of the right image P1R, arranged in the row direction. The right side of equation (1) is calculated by multiplying the matrix in the first term on the left side by the matrix in the second term on the left side. The first column on the right side of equation (1) represents the correlation between the feature amount of the first grid extracted from the right image P1R and the feature amount of each horizontal grid in a specific column extracted from the left image P1L, and the first row represents the correlation between the feature amount of the first grid extracted from the left image P1L and the feature amount of each horizontal grid in a specific column extracted from the right image P1R. In other words, the right side of equation (1) represents a correlation map between the feature amount of each grid in the left image P1L and the feature amount of each grid in the right image P1R. Note that in equation (1), the subscript "m" indicates the horizontal grid position of each image, and the subscript "n" indicates the feature number in each grid. In other words, m is 1 to 40, and n is 1 to 256.

Next, the estimation unit 152 uses the calculated correlation map to calculate a matching map ML of the right image P1R for the left image P1L, as shown in matrix (1). The matching map ML of the right image P1R for the left image P1L is calculated, for example, by applying the Softmax function to the row direction of the correlation map. This normalizes the correlation values in the horizontal direction. In other words, the values in the row direction are converted so that the sum of all values equals 1.

Next, the estimation unit 152 convolves the feature quantities extracted from the right image P1R into the calculated matching map ML, for example, using equation (2). The first term on the left side of equation (2) is the transpose of matrix (1), and the second term on the left side is the matrix of the second term on the left side of equation (1). Note that in the present invention, the feature quantities used to determine correlation and the feature quantities used to convolve into the matching map are the same; however, new feature quantities for determining correlation and feature quantities for convolution may be generated separately from the extracted feature quantities using a convolutional neural network or the like.

Next, the estimation unit 152 concatenates the feature obtained by equation (2) with the feature extracted from the left image P1L to generate new feature using, for example, a convolutional neural network. In this way, by integrating the feature values of the left and right images, the accuracy of position and orientation estimation is improved. Note that process m in Figure 12 may be repeated multiple times.

Next, the estimation unit 152 estimates the position, orientation, and class classification from the feature quantities obtained here, for example, using a convolutional neural network. Additionally, the estimation unit 152 uses the calculated correlation map to calculate a matching map MR of the left image P1L with respect to the right image P1R, as shown in matrix (2) (step S202). The matching map MR of the left image P1L with respect to the right image P1R is calculated, for example, by applying the Softmax function to the row direction of the correlation map, similar to the matching map ML of the right image P1R with respect to the left image P1L.

Next, the estimation unit 152 convolves the feature quantities of the left image P1L into the calculated matching map, for example, using equation (3). The first term on the left side of equation (3) is matrix (2), and the second term on the left side is the matrix of the second term on the left side of equation (1) before transposition.

The estimation unit 152 then selects and compares a preset threshold with the grid with the largest estimated result of the target (object) class classification estimated from the left image P1L (step S203). If the threshold is not exceeded, it is determined that there is no target and the process ends. If the threshold is exceeded, the grid with the largest value is selected from the matching map ML between that grid and the right image P1R (step S204).

Next, for the selected grid, the estimated result of the target's class classification in the right image P1R is compared with a preset threshold (step S208). If the threshold is exceeded, the grid with the largest value is selected from the matching map ML for that grid and the left image P1L (step S209). If the threshold is not exceeded, the class classification score of the grid selected from the estimated result of the left image P1L is set to 0 and the process returns to step S203 (step S207).

Next, the grid of the matching map ML selected in step S209 is compared with the grid selected from the estimation results of the left image P1L in step S204 to determine whether they are equal (step S210). If the grids are different, the classification score of the grid selected from the estimation results of the left image P1L in step S204 is set to 0, and the process returns to grid selection in step S203 (step S207). Finally, the parallax is calculated from the detection results of position information (e.g., the horizontal x value in Figure 1) of the grids selected in the left image P1L and right image P1R (step S211).

Next, the depth of the target is calculated based on the parallax calculated in step S211 (step S212). Note that if depth is being calculated for multiple targets, after step S211, the class classification score of the grid selected from the estimation results of the left image P1L and the right image P1R is set to 0, and then the process returns to step S203, and the process is repeated up to step S212.

As described above, the image processing device 10 includes an acquisition unit and an estimation unit. The acquisition unit acquires a first image and a second image of randomly stacked workpieces. The estimation unit generates a matching map between the feature amounts of the first image and the feature amounts of the second image, estimates the position, orientation, and class classification score of each target workpiece for each of the first and second images, and calculates the depth from the stereo camera to the workpiece by estimating the workpiece position based on the matching results and position estimation results using the attention map. This makes it possible to reduce false detections in object recognition.

(Image Processing Variations)
The object grasping system 1 has been described above, but is not limited to the above description and various modifications are possible without departing from the spirit of the system. For example, while the description has been given of a single type of object (workpiece), the present invention is not limited to this configuration and the image processing device 10 may be configured to detect multiple types of workpieces. Furthermore, the image processing device 10 may not only detect the object but also detect the position and orientation of a tray on which the object is placed. FIG. 13 is a diagram showing an example of a bulk pile image including a tray according to a modified example. In the example shown in FIG. 13, the image processing device 10 identifies the position and orientation of the tray on which the object is placed, thereby enabling the robot arm 30 to set a trajectory that will prevent the robot arm 30 from colliding with the tray. The tray, which is the object to be detected, is an example of an obstacle. The image processing device 10 may also be configured to detect obstacles other than trays.

Furthermore, while an example has been described in which the image processing device 10 divides the bulk image into, for example, a 40 x 40 grid, this is not limiting and the object may be detected by dividing it into finer or coarser grids, or estimation processing may be performed on a pixel-by-pixel basis. This allows the image processing device 10 to calculate the distance between the camera and the object with greater accuracy. Figure 14 is a diagram showing an example of a positional deviation estimation model according to a modified example. As shown in Figure 14, the image processing device 10 may cut out and combine portions of the left image P1L and right image P1R that are smaller in size than the grid around the estimated position. Then, estimation processing may be performed in the same manner as the estimation processing described above, and the positional deviation may be estimated based on the processing results.

Furthermore, when performing estimation processing in fine or coarse grid units or pixel units, estimation processing may be performed separately for each of the left image P1L and the right image P1R, as described above. Figure 15 is a diagram showing another example of a position shift estimation model according to a modified example. In the example shown in Figure 15, the image processing device 10 performs estimation processing separately for each of the left image P1L and the right image P1R. Even in this case, the image processing device 10 may share the weighting for the left image P1L with the weighting for the right image P1R when performing each estimation processing, as described above.

Furthermore, the above-described fixed processing may be performed not on images of the bulk piled workpieces 41 and 42, but on the robot arm 30, the workpieces 41 and 42 held by the robot arm 30, or the workpieces 41 and 42 aligned at the alignment destination.

(Acquisition and learning of three-dimensional data of a subject group)
The following describes a configuration for acquiring three-dimensional data of an object group including multiple objects, and a configuration for generating three-dimensional data of the object group using the acquired data. Fig. 16 is a block diagram showing an example of the configuration of a system for acquiring (measuring) three-dimensional data of an object group and generating a three-dimensional model of the object group. In Fig. 16, a processing device 110 and a three-dimensional data measuring device 140 are communicatively connected via a network NW. The processing device 110 also includes a communication I/F (interface) 111, an input I/F 112, a display 113, a memory circuit 114, and a processing circuit 115.

The three-dimensional data measuring device 140 is connected to the projector 120, the stereo cameras 131 and 132, and the 3D scanner camera 150. The three-dimensional data measuring device 140 includes a communication I/F 146, an input I/F 142, a display 143, a memory circuit 144, and a processing circuit 145. Note that, below, when there is no need to distinguish between the stereo cameras 131 and 132, they may be simply referred to as camera 130.

Projector 120 is capable of irradiating (projecting) a predetermined pattern using light with a shorter wavelength than visible light (for example, UV (ultraviolet) light), and is also capable of illumination using visible light. Illumination using visible light may be performed using white light, or may be illumination using red, blue, or green light, or illumination that matches the color of the expected indoor lighting. A visible light illumination device may be provided separately from projector 120. Camera 130 is a color or monochrome camera capable of capturing visible light.

The stereo cameras 131 and 132 are color or monochrome cameras capable of capturing visible light. Distance information to the group of objects is acquired using the 3D scanner camera 150. The 3D scanner camera 150 is a color camera that captures images of a group of objects illuminated by the projector 120 with light having a shorter wavelength than visible light. The 3D scanner camera 150 may form part of the stereo camera 130, or at least one of the cameras 131 and 132 may be configured to capture images of a group of objects illuminated with light having a shorter wavelength than visible light.

In this embodiment, the 3D scanner camera 150 captures an image of a group of objects including multiple objects (workpieces) illuminated by the projector 120 with light having a wavelength shorter than that of visible light. Similarly, the stereo cameras 131 and 132 capture images of the group of objects illuminated by the projector 120 with visible light. Figure 17 is a diagram showing an example of the measurement process. As shown in Figure 17, the stereo cameras 131 and 132 capture images of a group of objects including multiple objects W1 and W2. Similarly, the 3D scanner camera 150 captures images of the multiple objects W1 and W2 to obtain point cloud data indicating the distance to each of the objects W1 and W2.

In the processing device 110, the communication I/F 111 controls data input/output communication with external devices via the network NW. For example, the communication I/F 111 is realized by a network card, network adapter, NIC (Network Interface Controller), etc.

The input I/F 112 is connected to the processing circuit 115 and converts input operations received from an administrator (not shown) of the processing device 110 into electrical signals and outputs them to the processing circuit 115. For example, the input I/F 112 is a switch button, mouse, keyboard, touch panel, etc.

The display 113 is connected to the processing circuit 115 and displays various information and image data output from the processing circuit 115. For example, the display 113 may be implemented as an LCD monitor, a CRT (Cathode Ray Tube) monitor, a touch panel, etc.

The memory circuitry 114 is realized, for example, by a storage device such as a memory. The memory circuitry 114 stores various programs executed by the processing circuitry 115. The memory circuitry 114 also temporarily stores various data used when the processing circuitry 115 executes the various programs. The memory circuitry 114 has image, position, and orientation data 1141 of the object group and a machine (deep) learning model 1142.

Furthermore, the machine (deep) learning model 1142 has a neural network structure 1142a and learning parameters 1142b. The neural network structure 1142a is, for example, an application of a known network such as the convolutional neural network b1 in Figure 5, and is the network structure shown in Figure 12. The learning parameters 1142b are, for example, the weights of the convolutional filter of the convolutional neural network, and are parameters that are learned and optimized to estimate the position and orientation of each object included in the object group.

The machine (deep) learning model 1142 is used in the object grasping system 1 (Figures 1 and 2) to estimate the position and orientation of a workpiece from images output from the camera 20 (Figures 1 and 2). The machine (deep) learning model 1142 is generated, for example, by learning from the positions and orientations of multiple workpieces and images of the multiple workpieces as training data. Note that here, the machine (deep) learning model 1142 is generated, for example, by the processing circuit 115, but is not limited to this and may also be generated by an external computer.

The processing circuitry 115 is realized by a processor such as a CPU (Central Processing Unit). The processing circuitry 115 controls the entire processing device 110. The processing circuitry 115 reads various programs stored in the memory circuitry 114 and executes the read programs to perform various processes. For example, the processing circuitry 115 has a learning unit 1151 and a data output unit 1152.

The learning unit 1151 performs machine learning of the machine (deep) learning model 1142 based on the image, position, and posture data 1141 of the target group (including both newly measured and stored data by the 3D data measuring device 140 and data previously stored), and updates the learning parameters 1142b.

The data output unit 1152 outputs the image, position, and orientation data 1141 of the object group stored in the memory circuitry 114 and the data of the machine (deep) learning model 1142 in response to instructions from the operator or an external request.

In the three-dimensional data measuring device 140, the communication I/F 146 controls data input/output communication with external devices via the network NW. For example, the communication I/F 146 is implemented by a network card, network adapter, NIC, etc. The communication I/F 146 also transmits control signals to be output to the projector 120 in accordance with standards such as HDMI (registered trademark) (High-Definition Multimedia Interface), and receives status signals from the projector 120. The communication I/F 146 also transmits control signals to the camera 130 and receives image data output from the camera 130.

The input I/F 142 is connected to the processing circuit 145 and converts input operations received from an administrator (not shown) of the three-dimensional data measuring device 140 into electrical signals and outputs them to the processing circuit 145. For example, the input I/F 142 is a switch button, mouse, keyboard, touch panel, etc.

The display 143 is connected to the processing circuit 145 and displays various information and image data output from the processing circuit 145. For example, the display 143 may be implemented as an LCD monitor, a CRT monitor, a touch panel, etc.

The memory circuitry 144 is realized, for example, by a storage device such as a memory. The memory circuitry 144 stores various programs executed by the processing circuitry 145. The memory circuitry 144 also temporarily stores various data used when the various programs are executed by the processing circuitry 145. The memory circuitry 144 has three-dimensional object group data 1441 and image, position, and orientation data 1442 of the object group. The image, position, and orientation data 1442 of the object group is the original data for part of the image, position, and orientation data 1141 of the object group stored in the memory circuitry 114 of the processing device 110, and is accumulated in the memory circuitry 114 via the communication I/F 146, the network NW, and the communication I/F 111 of the processing device 110.

In this embodiment, the three-dimensional data of the individual objects included in the object group may already be stored in the memory circuitry 144, for example. Figure 18 is a diagram showing an example of three-dimensional data of the objects. In this case, the object group three-dimensional data measurement unit 1451 may detect each of the objects W1 and W2 included in the object group by, for example, matching an image captured by the camera 130 or the 3D scanner camera 150 with the three-dimensional object model M already stored in the memory circuitry 144, as shown in Figure 18.

The processing circuitry 145 of the three-dimensional data measuring device 140 is realized by a processor such as a CPU. The processing circuitry 145 controls the entire three-dimensional data measuring device 140. The processing circuitry 145 reads various programs stored in the memory circuitry 144 and executes the read programs to perform various processes. For example, the processing circuitry 145 has an object group three-dimensional data measurement unit 1451.

The object group 3D data measurement unit 1451 controls the projector 120 and camera 130 via the communication I/F 146, captures images of the object group captured by the camera 130, and acquires point cloud data indicating the 3D shape of each object group from the images of the object group captured by the 3D scanner camera 150. The 3D shape of each object included in the object group is measured based on images obtained by irradiating each object with a predetermined pattern and capturing the image, for example, using a known grid method. The 3D shape of each object is measured while changing the capturing direction of each object. In this way, by changing the capturing direction of the camera 130 and the 3D scanner camera 150, even if some objects are piled up and therefore cannot be seen from a certain angle, capturing the image from another angle makes it easier to capture object data over a wide area, improving the accuracy of matching the point cloud data with the 3D data described below. In this process, the 3D scanner camera 150 distinguishes each object included in the object group based on differences in visible light, as described below.

As preparation for processing, the stereo cameras 131 and 132 and the 3D scanner camera 150 are calibrated, and a predetermined paint is applied to each object. The predetermined paint is a fluorescent paint that emits visible light when irradiated with light with a shorter wavelength than visible light (e.g., ultraviolet (UV) light) and is colorless and transparent to visible light. This can also be called invisible paint. In this embodiment, each object is coated with an invisible paint that emits different visible light. For example, object W1 is coated with a first invisible paint that emits light with a first visible light, and object W2 is coated with a second invisible paint that emits light with a second visible light. In the following, light with a shorter wavelength than visible light is described as UV light, but this is not limited to this. For example, fluorescent paint can be excited by two-photon excitation in the same way as light with a shorter wavelength than visible light, and this can be used to irradiate patterned light.

Figure 19 is a flowchart showing an example of pre-processing. As shown in Figure 19, the object group 3D data measurement unit 1451 of the 3D data measurement device 140 performs calibration processing of the stereo cameras 131 and 132 (step S101). Note that, hereinafter, the cameras 131 and 132 that make up the stereo camera may be referred to as a stereo set. Figure 20 is a diagram showing an example of calibration processing. As shown in Figure 20, calibration processing of the stereo set is performed, for example, by the stereo cameras 131 and 132 photographing a known calibration board CB under visible light. The calibration processing is performed using a commonly known method.

Next, the object group 3D data measurement unit 1451 performs a calibration process between the projector 120, which emits light with a wavelength shorter than that of visible light, and the 3D scanner camera 150, which acquires point cloud data (step S102). Note that, hereinafter, the combination of the projector 120 and the 3D scanner camera 150 may be referred to as a 3D scan set. Furthermore, the object group 3D data measurement unit 1451 also performs a calibration process between the stereo set and the 3D scan set (step S103). Note that the calibration process of the 3D scan set and the calibration process between the 3D scan set and the stereo set in steps S102 and S103 may be performed using visible light, or may be performed using light with a wavelength shorter than that of visible light using a calibration board printed with fluorescent paint.

Next, invisible paint is applied to each individual object included in the object group to be measured (step S111). In this embodiment, each object included in the object group is coated with invisible paint that emits different visible light. Each object coated with invisible paint is placed at a position where 3D data measurement will be performed (step S112). The processes of steps S111 and S112 are repeated until all objects included in the object group have been positioned (step S113). Figure 21 is a diagram showing an example of multiple objects coated with paint. As shown in Figure 21, multiple objects (workpieces) W1 and W2 included in the object group are each coated with a different invisible paint. In this embodiment, objects W1 and W2 are placed in any position, such as a position where they appear to overlap when photographed from stereo cameras 131 and 132, as shown in Figure 18.

After pre-processing, the object group 3D data measurement unit 1451 of the 3D data measurement device 140 performs processing to acquire 3D data of the arranged object group. Figure 22 is a flowchart showing an example of the processing for acquiring object group 3D data, and is an example of processing when photography is performed using a color camera 150.

In FIG. 22, when the 3D data measuring device 140 starts processing, it projects a UV pattern onto the object using the projector 120. Then, the 3D scanner camera 150, which is positioned in a predetermined positional relationship with the projector 120, captures each object included in the object group (step S301). FIG. 23 is a diagram showing an example of the state in which the projector projects a UV pattern onto the object group. The UV light emitted from the projector 120 causes the objects W1 and W2 to emit light in a predetermined pattern using invisible paint, and this state is captured by the 3D scanner camera 150. Because the visible light emission is due to the invisible paint, it does not matter if the object W is glossy or black. Also, in step S301, object W1, which is coated with a first invisible paint and emits first visible light, and object W2, which is coated with a second invisible paint and emits second visible light, are simultaneously captured.

Returning to FIG. 22, the photographing of the object (step S301) is repeated by changing the UV pattern emitted from the projector 120 (step S303) until the predetermined UV pattern is completed (step S302: No). Then, the three-dimensional data measuring device 140 calculates the three-dimensional shape of each object group based on the multiple images taken for each UV pattern and stores it in the memory circuit (step S304).

Once processing for the specified UV pattern is complete (step S302: Yes), the three-dimensional data measuring device 140 next performs processing by irradiating the object with white light from the projector 120 or an external light source and capturing an image with the camera 130 (step S311). The three-dimensional data measuring device 140 then stores the captured image in a memory circuit (step S312). Figure 24 is a diagram showing an example of a state in which visible light is irradiated onto a group of objects. Irradiation with white light from the projector 120 causes the first invisible paint on object W1 and the second invisible paint on object W2 to become colorless and transparent. This allows for the generation of machine learning data similar to the original state of objects W1 and W2, which is expected to improve the accuracy of workpiece position and orientation recognition when the workpiece is actually grasped by a robot arm.

Returning to FIG. 22, the three-dimensional data measuring device 140 then determines whether processing from the predetermined direction has been completed (step S321). If it determines that processing has not been completed (step S321: No), it changes the imaging direction for the object group (step S322) and repeats the process from irradiation of the UV pattern (step S301). The object's orientation may be changed manually by the operator or by a drive mechanism. If the three-dimensional data measuring device 140 determines that processing from the predetermined direction has been completed (step S321: Yes), it proceeds to step S323. Note that if the positions and orientations of all objects included in the object group can be identified by imaging from a single direction alone, it is not necessary to repeat the processes from step S322 onwards.

Next, the three-dimensional data measuring device 140 performs point cloud data matching between point cloud data measured from different directions and updates the data (step S323). In this case, the three-dimensional data measuring device 140 may perform point cloud data matching individually for the point cloud data of object W1 acquired based on the first visible light and the point cloud data of object W2 acquired based on the second visible light, or may perform point cloud data matching for the entire object group. Next, point cloud matching is performed between the point cloud data acquired based on the first visible light and the pre-stored three-dimensional data. From the position and orientation of the three-dimensional model at this time, the relative position with the stereo camera at the time of image capture is calculated, and data on the position and orientation of the object is obtained (S324). Next, point cloud matching is similarly performed between the point cloud data acquired based on the second visible light and the 3D model. If there is other data separated by emitted color, matching with the 3D model is repeated for each emitted color (S331, S332).

Then, the three-dimensional data measuring device 140 determines whether the predetermined number of images for generating data have been captured (step S341). If the three-dimensional data measuring device 140 determines that the predetermined number of images have not been captured (step S341: No), it repeats the process from UV pattern irradiation (step S301) after the bulk pile state of the objects has been changed (step S342). In step S342, for example, the processes in steps S112 to S113 shown in FIG. 19 are performed. On the other hand, if the three-dimensional data measuring device 140 determines that the predetermined number of images have been captured (step S341: Yes), it ends the process.

The above describes an embodiment of the present invention, but the present invention is not limited to the above embodiment, and various modifications are possible without departing from the spirit of the invention. For example, the number of objects included in the object group is not limited to that described, and the object group may include three or more objects.

As described above, the machine learning data generation method according to the embodiment includes a first step of irradiating a first object coated with a first paint that emits light using first visible light when irradiated with light having a wavelength shorter than that of visible light and is colorless and transparent in response to visible light, and a second object coated with a second paint that emits light using second visible light when irradiated with light having a wavelength shorter than that of visible light and is colorless and transparent in response to the visible light, with light having a wavelength shorter than that of visible light in a predetermined pattern, and capturing images of the first object and the second object. This makes it possible to easily measure three-dimensional data of an object group including the three-dimensional shapes of each individual object from an actual object group containing multiple objects.

The machine learning data generation method may also include a second step of irradiating the first object and the second object with visible light and capturing images of the first object and the second object. In the first step, the position and three-dimensional shape of the first object are measured by detecting the first visible light, and the position and three-dimensional shape of the second object are measured by detecting the second visible light. The machine learning data generation method may also include a third step of separating the three-dimensional shapes of the first object and the second object from the three-dimensional shapes measured in the first step based on the difference between the first visible light and the second visible light, and matching the three-dimensional data of the first object and the second object with pre-stored three-dimensional data. This makes it possible to easily measure three-dimensional data of a group of objects, including the three-dimensional shapes of each individual object.

In addition, in the first step, the three-dimensional shapes of the first object and the second object are measured by switching between multiple predetermined patterns for one orientation of the first object and the second object. This improves the accuracy of the three-dimensional shape measurement.

The machine learning data generation method may also include changing the direction in which the first object and the second object are photographed, repeating the measurement of the three-dimensional shapes of the first object and the second object in the first step and the photographing of the first object and the second object in the second step, and linking and updating the data by point cloud data matching. The machine learning data generation method may also include repeating the measurement of the three-dimensional shapes of the first object and the second object in the first step and the photographing of the first object and the second object in the second step, and linking and updating the data by matching the point cloud data separated for each object with the three-dimensional data. This makes it possible to obtain the three-dimensional shapes of each object that cannot be captured by photographing from only one side.

An object machine learning data measurement device according to an embodiment includes a first processing unit that irradiates a first object coated with a first paint that emits first visible light when irradiated with light having a wavelength shorter than that of visible light and is colorless and transparent in response to visible light, and a second object coated with a second paint that emits second visible light when irradiated with light having a wavelength shorter than that of visible light, with light having a wavelength shorter than that of visible light in a predetermined pattern, photographs the first object and the second object, and measures the position and three-dimensional shape of the first object and the position and three-dimensional shape of the second object from the photographed image. This allows the above-described object three-dimensional data measurement method to be realized as a device.

The various processes described in each embodiment can be realized by executing a pre-prepared program on an information processing device. Therefore, below, an example of an information processing device 900 that executes a program having the same functions as the motor control device 100 in each of the above embodiments is described. Figure 26 is a block diagram showing an example of an information processing device that executes a measurement program.

The information processing device 900 that executes the measurement program shown in FIG. 26 includes a communication device 910, an input device 920, an output device 930, a ROM 940, a RAM 950, a CPU 960, and a bus 970. For example, the communication device 910 functions as the communication I/F 146 shown in FIG. 16, the input device 920 functions as the input I/F 142 shown in FIG. 16, and the output device 930 functions as the display 143 shown in FIG. 16.

The ROM 940 is pre-stored with a measurement program that performs the same functions as the above embodiment. The measurement program may be stored not in the ROM 940 but in a storage device such as a hard disk built into the computer system, or in a recording medium readable by a drive (not shown). The recording medium may also be, for example, a CD-ROM, DVD disk, USB memory, portable recording medium such as an SD card, or semiconductor memory such as a flash memory. The measurement program is measurement program 940A, as shown in FIG. 26. Measurement program 940A may be integrated or distributed as appropriate.

The CPU 960 then reads the measurement program 940A from the ROM 940 and expands the read program in the work area of the RAM 950. Then, as shown in FIG. 26, the CPU 960 causes the measurement program 940A expanded in the RAM 950 to function as a measurement process 950A.

The CPU 960 determines whether the identification information included in the setting change information received from the maintenance device 200 belongs to the motor control device. The CPU 960 changes the settings using the setting change information corresponding to the identification information determined to belong to the motor control device. The CPU 960 transfers the setting change information including identification information of a motor control device other than the motor control device to another motor control device.

Note that while Figure 26 describes an information processing device 900 that executes a measurement program having the same functions as the three-dimensional data measuring device 140, the processing device 110 and image processing device 10 can also be realized by executing a pre-prepared program on the information processing device. Furthermore, while the information processing device 900 is, for example, a standalone computer, it is not limited to this and may be realized by multiple computers that can communicate with each other, or may be realized in a virtual machine on the cloud. Furthermore, all or part of the functions of the information processing device 900 may be realized by an integrated circuit, such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

Furthermore, the present invention is not limited to the above-described embodiments. The present invention also includes configurations that appropriately combine the above-described components. Furthermore, further effects and modifications can be easily derived by those skilled in the art. Therefore, the broader aspects of the present invention are not limited to the above-described embodiments, and various modifications are possible.

110 Processing device, 114 Memory circuit, 1141 Image, position, and orientation data of object group, 1142 Machine (deep) learning model, 1142a Neural network structure, 1142b Learning parameters, 115 Processing circuit, 1151 Learning unit, 1152 Data output unit, 120 Projector, 130 (131, 132) Stereo camera, 140 3D data measurement device, 144 Memory circuit, 1441 3D data of object group, 1442 Image, position, and orientation data of object group, 145 Processing circuit, 1451 3D data measurement unit of object group, 150 3D scanner camera, W1, W2 Object, M 3D object model

Claims (4)

a first step of irradiating a first object coated with a first paint that emits light using first visible light when irradiated with light having a wavelength shorter than that of visible light and that is colorless and transparent to visible light, and a second object coated with a second paint that emits light using second visible light when irradiated with light having a wavelength shorter than that of visible light and that is colorless and transparent to visible light, with light having a wavelength shorter than that of visible light in a predetermined pattern, photographing the first object and the second object with a shape acquisition camera, and measuring the three-dimensional shapes of the first object and the second object from the photographed images;
a second step of irradiating the first object and the second object with visible light and capturing images of the first object and the second object with a position acquisition camera that is a stereo camera ;
a third step of separating the three-dimensional shape measured in the first step into a three-dimensional shape of the first object and a three-dimensional shape of the second object based on a difference between the first visible light and the second visible light, and matching the three-dimensional data of the first object and the three-dimensional data of the second object that have been stored in advance;
Equipped with
changing a direction in which the first object and the second object are photographed, repeating the measurement of the three-dimensional shape of the first object and the three-dimensional shape of the second object in the first step and the photographing of the first object and the second object in the second step, and linking and updating the data by point cloud data matching;
measuring the three-dimensional shape of the first object and the three-dimensional shape of the second object in the first step, photographing the first object and the second object in the second step, and calculating the relative position or orientation of the object with the position acquisition camera in the third step, which separates point cloud data of the first object and the second object and performs point cloud data matching with the three-dimensional data of the object;
Methods for generating data for machine learning. The machine learning data generation method of claim 1, wherein the shape acquisition camera is a camera different from the position acquisition camera. The machine learning data generation method according to claim 1 or 2, wherein in the first step, the three-dimensional shape of the first object is measured by detecting the first visible light, and the three-dimensional shape of the second object is measured by detecting the second visible light. 4. The method for generating data for machine learning according to claim 1, wherein in the first step, the three-dimensional shape of the first object and the three-dimensional shape of the second object are measured by switching between a plurality of the predetermined patterns for one posture of the first object and the second object.