JP7474083B2 - Method for measuring three-dimensional data of an object, method for generating training data, and training data generating device - Google Patents

Description

The present invention relates to a method for measuring three-dimensional data of an object , a method for generating training data, and an apparatus for generating training data.

In object gripping systems, where a robot arm grips multiple randomly stacked objects (workpieces), artificial intelligence such as convolutional neural networks may be used to recognize the type, position, and orientation of an object from a photographed image. Machine learning using a huge amount of training data is essential for the proper use of systems that use artificial intelligence, and the cost of preparing the training data is significant.

In order to accurately recognize multiple randomly piled objects from captured images, it is necessary to prepare learning data of images that are close to how they will actually be placed on a workbench, etc., and 3D shape data of the objects alone is insufficient; texture data of the objects' surfaces is also required. Generally, a considerable amount of learning data is generated by processing 3D shape data and texture data measured from actual objects. Various technologies have been proposed for precisely measuring the 3D shape of objects without contact (see Patent Document 1, etc.).

Japanese Patent Application Laid-Open No. 7-280535

However, when obtaining 3D shape data and texture data from an actual object through optical measurement, it is difficult to measure if the object is shiny or black. In other words, if the object is shiny or black, the light emitted for measurement does not return normally, making it difficult to accurately measure the 3D shape.

In order to properly reflect the irradiated light, paint or other materials may be sprayed onto the surface of the object to change its color or texture, but in such cases, the texture of the object's surface cannot be measured accurately. This means that the measurement of the 3D shape and the texture must be carried out in separate processes, which increases the number of work steps and makes it difficult to align the 3D shape and texture.

The present invention has been made in consideration of the above, and aims to provide a method for easily measuring three-dimensional data of an object, including its three-dimensional shape and texture.

In order to solve the above-mentioned problems and achieve the object, a method for measuring three-dimensional data of an object according to one aspect of the present invention includes a first step and a second step. In the first step, an object that emits visible light when irradiated with light having a wavelength shorter than that of visible light and is coated with a paint that is colorless and transparent to visible light is irradiated with light having a wavelength shorter than that of visible light in a predetermined pattern, the object is photographed, and the three-dimensional shape of the object is measured from the photographed image. In the second step, the object is irradiated with visible light, the object is photographed, and the texture of the object is measured from the photographed image.

The method for measuring three-dimensional object data according to one aspect of the present invention can easily measure three-dimensional object data including three-dimensional shape and texture.

FIG. 1 is an external view showing an example of an object grasping system. FIG. 2 is a block diagram showing an example of the configuration of an object grasping system. FIG. 3 is a diagram illustrating an example of a process related to control of a robot arm. FIG. 4 is a diagram showing another example of the process relating to the control of the robot arm. FIG. 5 is a diagram illustrating an example of a detection model. FIG. 6 is a diagram showing an example of a feature map output by the feature detection layer (u1). FIG. 7 is a diagram showing an example of an estimation result of the position and orientation of an object. FIG. 8 is a diagram showing another example of the estimation result of the gripping position of the target object. FIG. 9 is a diagram showing an example of a bulk pile image captured by the stereo camera. FIG. 10 is a diagram showing an example of the relationship between the bulk pile image and the matching map. FIG. 11 is a flowchart illustrating an example of the estimation process. FIG. 12 is a diagram illustrating an example of the estimation process. FIG. 13 is a diagram showing an example of a bulk pile image including a tray according to a modified example. FIG. 14 is a diagram showing an example of a position shift estimation model according to a modified example. FIG. 15 is a diagram showing another example of a position shift estimation model according to a modified example. FIG. 16 is a block diagram showing an example of the configuration of a system for acquiring (measuring) three-dimensional data of an object and for learning. FIG. 17 is a flowchart showing an example of a process for acquiring three-dimensional data of an object. FIG. 18 is a diagram showing an example of a state in which a UV pattern is irradiated onto an object from a projector. FIG. 19 is a diagram showing an example of a state in which visible light is irradiated onto an object. FIG. 20 is a flowchart showing another example of the process of acquiring three-dimensional data of an object. FIG. 21 is a flowchart showing an example of learning data generation and learning processing. FIG. 22 is a diagram showing an example of three-dimensional data of an object. FIG. 23 is a diagram showing an example of a captured image of a virtual space in which a plurality of objects are arranged.

The object 3D data measuring method, object 3D data measuring device, learning data generating method, and learning data generating device according to the embodiments will be described below with reference to the drawings. Note that the present invention is not limited to these embodiments. Furthermore, the dimensional relationships and ratios of each element in the drawings may differ from reality. The drawings may also include parts with different dimensional relationships and ratios. Furthermore, the contents described in one embodiment or variant apply in principle to other embodiments or variants.

(Object Grasping System)
FIG. 1 is an external view showing an example of an object gripping system 1. The object gripping system 1 shown in FIG. 1 includes an image processing device 10, a camera 20, and a robot arm 30 (not shown). The camera 20 is provided at a position where it can capture images of both the robot arm 30 and the randomly piled workpieces 41, 42, etc., which are objects to be gripped by the robot arm 30. The camera 20 captures images of the robot arm 30 and the workpieces 41, 42, for example, and outputs the images to the image processing device 10. Note that the robot arm 30 and the randomly piled workpieces 41, 42, etc. may be captured by separate cameras. As shown in FIG. 1, the camera 20 is a camera capable of capturing multiple images, such as a known stereo camera. The image processing device 10 estimates the positions and orientations of the workpieces 41, 42, etc., using the images output from the camera 20. The image processing device 10 outputs a signal for controlling the operation of the robot arm 30 based on the estimated positions and orientations of the workpieces 41, 42, etc. The robot arm 30 performs an operation of gripping the workpieces 41, 42, etc. based on a signal output from the image processing device 10. Although FIG. 1 discloses a plurality of different types of workpieces 41, 42, etc., the type of workpiece may be one type. Here, a case where there is one type of workpiece will be described. The workpieces 41, 42, etc. are arranged so that their positions and postures are irregular. As shown in FIG. 1, for example, the plurality of workpieces may be arranged so as to overlap in a top view. The workpieces 41, 42 are examples of objects.

Figure 2 is a block diagram showing an example of the configuration of the object grasping system 1. As shown in Figure 2, the image processing device 10 is communicatively connected to a camera 20 and a robot arm 30 via a network NW. Also, as shown in Figure 2, the image processing device 10 includes a communication I/F (interface) 11, an input I/F 12, a display 13, a memory circuit 14, and a processing circuit 15.

The communication I/F 11 controls data input/output communication with external devices via the network NW. For example, the communication I/F 11 is realized by a network card, a network adapter, a NIC (Network Interface Controller), etc., and receives image data output from the camera 20 and transmits signals to be output to the robot arm 30.

The input I/F 12 is connected to the processing circuit 15, converts input operations received from an administrator (not shown) of the image processing device 10 into electrical signals, and outputs the electrical signals to the processing circuit 15. For example, the input I/F 12 is a switch button, a mouse, a keyboard, a touch panel, etc.

The display 13 is connected to the processing circuit 15 and displays various information and image data output from the processing circuit 15. For example, the display 13 is realized by a liquid crystal monitor, a CRT (Cathode Ray Tube) monitor, a touch panel, etc.

The memory circuit 14 is realized by, for example, a storage device such as a memory. Various programs executed by the processing circuit 15 are stored in the memory circuit 14. In addition, various data used when the various programs are executed by the processing circuit 15 is temporarily stored in the memory circuit 14. The memory circuit 14 has a machine (deep) learning model 141. Furthermore, the machine (deep) learning model 141 has a neural network structure 141a and learning parameters 141b. The neural network structure 141a is, for example, an application of a known network such as the convolutional neural network b1 of FIG. 5, and is a network structure shown in FIG. 12 described later. The learning parameters 141b are, for example, the weights of the convolution filter of the convolutional neural network, and are parameters that are learned and optimized to estimate the position and posture of the target object. The neural network structure 141a may be provided in the estimation unit 152. Note that the machine (deep) learning model 141 in the present invention will be described using a learned model as an example, but is not limited to this. In the following, the machine (deep) learning model 141 may be referred to simply as the "learning model 141."

The learning model 141 is used in a process of estimating the position and orientation of a workpiece from an image output from the camera 20. The learning model 141 is generated, for example, by learning from the positions and orientations of multiple workpieces and images of the multiple workpieces as teacher data. Note that here, the learning model 141 is generated, for example, by the processing circuit 15, but is not limited to this and may be generated by an external computer.

The processing circuitry 15 is realized by a processor such as a CPU (Central Processing Unit). The processing circuitry 15 controls the entire image processing device 10. The processing circuitry 15 reads various programs stored in the memory circuitry 14 and executes the read programs to perform various processes. For example, the processing circuitry 15 has an image acquisition unit 151, an estimation unit 152, and a robot control unit 153.

The image acquisition unit 151 acquires a bulk image, for example, via the communication I/F 11, and outputs the image to the estimation unit 152. The image acquisition unit 151 is an example of an acquisition unit.

The estimation unit 152 estimates the position and orientation of the target object using the outputted bulk image. The estimation unit 152 performs estimation processing on the image of the target object using, for example, the learning model 141, and outputs the estimation result to the robot control unit 153. Note that the estimation unit 152 may further estimate, for example, the position and orientation of a tray or the like on which the target object is placed. The configuration for estimating the position and orientation of the tray will be described later.

The robot control unit 153 generates a signal to control the robot arm 30 based on the estimated position and posture of the object, and outputs the signal to the robot arm 30 via the communication I/F 11. The robot control unit 153 acquires, for example, information on the current position and posture of the robot arm 30. Then, the robot control unit 153 generates a trajectory along which the robot arm 30 moves when grasping the object, according to the current position and posture of the robot arm 30 and the estimated position and posture of the object. The robot control unit 153 may also modify the trajectory along which the robot arm 30 moves, based on the position and posture of a tray or the like.

Figure 3 is a diagram showing an example of processing related to control of a robot arm. As shown in Figure 3, the estimation unit 152 estimates the position and orientation of a target object from the bulk image. Similarly, the estimation unit 152 may estimate the position and orientation of a tray or the like on which the object is placed from the bulk image. The robot control unit 153 calculates the coordinates and orientation of the position of the hand of the robot arm 30 based on the estimated models of the object and tray, etc., and generates a trajectory of the robot arm 30.

In addition, the robot control unit 153 may further output a signal to control the operation of the robot arm 30 to align the grasped object after the robot arm 30 grasps the object. FIG. 4 is a diagram showing another example of processing related to the control of the robot arm. As shown in FIG. 4, the image acquisition unit 151 acquires an image of the object grasped by the robot arm 30, which is photographed by the camera 20. The estimation unit 152 estimates the position and posture of the target object grasped by the robot arm 30, and outputs it to the robot control unit 153. The image acquisition unit 151 may further acquire an image of the tray or the like to which the grasped object is to be moved, which is photographed by the camera 20. At that time, the image acquisition unit 151 further acquires an image of the object already aligned on the tray or the like to which the object is to be aligned (aligned image). The estimation unit 152 estimates the position and posture of the tray or the like to which the object is to be aligned, and the position and posture of the object already aligned, from the image of the alignment destination or the aligned image. Then, the robot control unit 153 calculates the coordinates and orientation of the position of the hand of the robot arm 30 based on the estimated position and orientation of the object held by the robot arm 30, the position and orientation of the tray or other object to be aligned, and the position and orientation of objects that have already been aligned, and generates a trajectory for the robot arm 30 when aligning the objects.

Next, the estimation process in the estimation unit 152 will be described. The estimation unit 152 extracts the feature amount of the object by using, for example, a model that applies an object detection model having known downsampling, upsampling, and skip connections. FIG. 5 is a diagram showing an example of a detection model. In the object detection model shown in FIG. 5, the d1 layer divides, for example, a bulk pile image P1 (320×320 pixels) into 40×40 grids by downsampling via a convolutional neural network b1, and calculates multiple feature amounts (for example, 256 types) for each grid. In addition, the d2 layer, which is a layer lower than the d1 layer, divides the grid divided in the d1 layer more roughly (for example, into 20×20 grids) and calculates the feature amount of each grid. Similarly, the d3 layer and the d4 layer, which are layers lower than the d1 layer and the d2 layer, divide the grid divided in the d2 layer more roughly, respectively. The d4 layer calculates features in finer divisions by upsampling, and at the same time integrates them with the features of the d3 layer through skip connection s3 to generate the u3 layer. The skip connection may be a simple addition or concatenation of features, or a transformation such as a convolutional neural network may be applied to the features of the d3 layer. Similarly, the u3 layer is upsampled and the calculated features are integrated with the features of the d2 layer through skip connection s2 to generate the u2 layer. The u1 layer is then generated in a similar manner. As a result, in the u1 layer, features of each grid divided into a 40 x 40 grid are calculated, just like in the d1 layer.

Figure 6 shows an example of a feature map output by the feature detection layer (u1). The horizontal direction of the feature map shown in Figure 6 indicates each horizontal grid of the bulk pile image P1, which is divided into a 40 x 40 grid, and the vertical direction indicates each vertical grid. The depth direction of the feature map shown in Figure 6 indicates the feature element in each grid.

FIG. 7 is a diagram showing an example of the estimation result of the position and orientation of the object. As shown in FIG. 7, the estimation unit outputs two-dimensional coordinates (Δx, Δy) indicating the position of the object, quaternions (qx, qy, qz, qw) indicating the orientation of the object, and class classification scores (C0, C1, ..., Cn). Note that, here, as the estimation result, among the coordinates indicating the position of the object, the depth value indicating the distance from the camera 20 to the object is not calculated. The configuration for calculating the depth value will be described later. Note that, the depth here refers to the distance from the z coordinate of the camera to the z coordinate of the object in the z axis direction parallel to the optical axis of the camera. Note that the class classification score is a value output for each grid, and is the probability that the center point of the object is included in that grid. For example, if there are n types of objects, the "probability that the center point of the object is not included" is added to this and n+1 class classification scores are output. For example, if there is only one type of workpiece to be the object, two class classification scores are output. Also, if there are multiple objects in the same grid, the probability of the object stacked higher is output.

In FIG. 7, point C indicates the center of grid Gx, and point ΔC with coordinates (Δx, Δy) indicates, for example, the center point of a detected object. That is, in the example shown in FIG. 7, the center of the object is offset from center point C of grid Gx by Δx in the x-axis direction and Δy in the y-axis direction.

Instead of FIG. 7, arbitrary points a, b, and c other than the center of the object may be set as shown in FIG. 8, and the coordinates (Δx1, Δy1, Δz1, Δx2, Δy2, Δz2, x3, Δy3, Δz3) of the arbitrary points a, b, and c from point C at the center of grid Gx may be output. The arbitrary points may be set anywhere on the object, and may be one or multiple points.

Note that if the grid division is too coarse compared to the size of the object, multiple objects may fall into one grid, causing the features of each object to mix and result in false detection. Therefore, here we only use the feature map, which is the output of the feature extraction layer (u1), from which the fine (40 x 40 grid) features finally generated are calculated.

In addition, here, for example, a stereo camera is used to capture two types of left and right images, and the distance from the camera 20 to the object is identified. FIG. 9 is a diagram showing an example of a bulk image captured by a stereo camera. As shown in FIG. 9, the image acquisition unit 151 acquires two types of bulk images, a left image P1L and a right image P1R. The estimation unit 152 performs estimation processing using the learning model 141 for both the left image P1L and the right image P1R. Note that when performing the estimation processing, a part or all of the learning parameters 141b used for the left image P1L may be shared as a weighting for the right image P1R. Note that instead of a stereo camera, a single camera may be used, and images equivalent to two types of left and right images may be captured at two locations by shifting the position of the camera.

Here, the estimation unit 152 suppresses erroneous recognition of objects by using a matching map that combines the features of the left image P1L and the features of the right image P1R. The matching map indicates the strength of correlation between the features of the right image P1R and the left image P1L for each feature. In other words, by using the matching map, it is possible to match the left image P1L and the right image P1R by focusing on the features in each image.

Figure 10 is a diagram showing an example of the relationship between a bulk pile image and a matching map. As shown in Figure 10, in the matching map ML, which is based on the left image P1L and corresponds to the right image P1R, the grid MLa that has the highest correlation between the feature amount of the grid including the center point of the object W1L in the left image P1L and the feature amount included in the right image P1R is highlighted. Similarly, in the matching map MR, which is based on the right image P1R and corresponds to the left image P1L, the grid MRa that has the highest correlation between the feature amount of the grid including the center point of the object W1R in the right image P1R and the feature amount included in the left image P1L is highlighted. In addition, the grid MLa with the highest correlation in the matching map ML corresponds to the grid where the object W1L in the left image P1L is located, and the grid MRa with the highest correlation in the matching map MR corresponds to the grid where the object W1R in the right image P1R is located. This makes it possible to determine that the grid on which object W1L is located in the left image P1L matches the grid on which object W1R is located in the right image P1R. That is, in FIG. 9, the matching grids are grid G1L in the left image P1L and grid G1R in the right image P1R. This makes it possible to determine the parallax for object W1 based on the X coordinate of object W1L in the left image P1L and the X coordinate of object W1R in the right image P1R, and therefore the depth z from camera 20 to object W1 can be determined.

Fig. 11 is a flowchart showing an example of the estimation process. Fig. 12 is a diagram showing an example of the estimation process. Hereinafter, the description will be given with reference to Figs. 9 to 12. First, the image acquisition unit 151 acquires each of the left and right images of the object, such as the left image P1L and the right image P1R shown in Fig. 9 (step S201). Next, the estimation unit 152 calculates feature amounts for each grid in the horizontal direction of each of the left and right images. Here, as described above, when each image is divided into a 40 x 40 grid and 256 feature amounts are calculated for each grid, a 40-row, 40-column matrix is obtained in the horizontal direction of each image, as shown in the first and second terms on the left side of Equation (1) (the product of the matrix of the first term and the matrix of the second term).

Next, the estimation unit 152 executes the process m shown in FIG. 12. First, the estimation unit 152 calculates the matrix product of the feature amount of a specific column extracted from the left image P1L and the feature amount of the same column extracted from the right image P1R, for example, by using formula (1). In formula (1), the first term on the left side indicates that the feature amounts l11 to l1n in the first grid in the horizontal direction of a specific column of the left image P1L are arranged in the row direction. On the other hand, in the second term on the left side of formula (1), the feature amounts r11 to r1n in the first grid in the horizontal direction of a specific column of the right image P1R are arranged in the column direction. In other words, the matrix in the second term on the left side is a transposition of a matrix in which the feature amounts r11 to r1m of the grid in the horizontal direction of a specific column of the right image P1R are arranged in the row direction. In addition, the right side of formula (1) is a calculation of the matrix product of the matrix in the first term on the left side and the matrix in the second term on the left side. The first column on the right side of equation (1) represents the correlation between the feature amount of the first grid extracted from the right image P1R and the feature amount of each grid in the horizontal direction of a specific column extracted from the left image P1L, and the first row represents the correlation between the feature amount of the first grid extracted from the left image P1L and the feature amount of each grid in the horizontal direction of a specific column extracted from the right image P1R. In other words, the right side of equation (1) represents a correlation map between the feature amount of each grid in the left image P1L and the feature amount of each grid in the right image P1R. Note that in equation (1), the subscript "m" indicates the horizontal grid position of each image, and the subscript "n" indicates the feature number in each grid. In other words, m is 1 to 40, and n is 1 to 256.

Next, the estimation unit 152 uses the calculated correlation map to calculate a matching map ML of the right image P1R for the left image P1L as shown in matrix (1). The matching map ML of the right image P1R for the left image P1L is calculated, for example, by applying the Softmax function to the row direction of the correlation map. This normalizes the correlation value in the horizontal direction. In other words, the values in the row direction are converted so that the sum of all values is 1.

Next, the estimation unit 152 convolves the feature quantities extracted from the right image P1R into the calculated matching map ML, for example, using equation (2). The first term on the left side of equation (2) is the transposed matrix (1), and the second term on the left side is the matrix of the second term on the left side of equation (1). Note that in the present invention, the feature quantities used for correlation and the feature quantities used for convolution into the matching map are the same, but new feature quantities for correlation and feature quantities for convolution may be generated separately from the extracted feature quantities using a convolutional neural network or the like.

Next, the estimation unit 152 generates new features by linking the features obtained by formula (2) to the features extracted from the left image P1L, for example, by using a convolutional neural network. In this way, by integrating the features of the left and right images, the estimation accuracy of the position and orientation is improved. Note that process m in FIG. 12 may be repeated multiple times.

Next, the estimation unit 152 estimates the position, orientation, and class classification from the features obtained here, for example, by using a convolutional neural network. In addition, the estimation unit 152 uses the calculated correlation map to calculate a matching map MR of the left image P1L with respect to the right image P1R as shown in matrix (2) (step S202). The matching map MR of the left image P1L with respect to the right image P1R is also calculated, for example, by applying a Softmax function to the row direction of the correlation map, similar to the matching map ML of the right image P1R with respect to the left image P1L.

Next, the estimation unit 152 convolves the features of the left image P1L into the calculated matching map, for example, using equation (3). The first term on the left side of equation (3) is matrix (2), and the second term on the left side is the matrix of the second term on the left side of equation (1) before transposition.

The estimation unit 152 then selects and compares a preset threshold with the grid that has the largest estimated result of the class classification of the target (object) estimated from the left image P1L (step S203). If the threshold is not exceeded, it is determined that there is no target and the process ends. If the threshold is exceeded, the grid with the largest value is selected from the matching map ML with the right image P1R for that grid (step S204).

Next, for the selected grid, the estimated result of the class classification of the target in the right image P1R is compared with a preset threshold value (step S208). If the threshold value is exceeded, the grid with the largest value is selected from the matching map ML with the left image P1L for that grid (step S209). If the threshold value is not exceeded, the class classification score of the grid selected from the estimated result of the left image P1L is set to 0 and the process returns to step S203 (step S207).

Next, the grid of the matching map ML selected in step S209 is compared with the grid selected from the estimation result of the left image P1L in step S204 to see if they are equal (step S210). If the grids are different, the class classification score of the grid selected from the estimation result of the left image P1L in step S204 is set to 0, and the process returns to the selection of the grid in step S203 (step S207). Finally, the parallax is calculated from the detection results of the position information (e.g., the horizontal x value in Figure 1) of the grids selected in the left image P1L and the right image P1R (step S211).

Next, the depth of the target is calculated based on the parallax calculated in step S211 (step S212). Note that if the depth is to be calculated for multiple targets, after step S211, the class classification score of the grid selected from the estimation results of the left image P1L and the right image P1R is set to 0, and then the process returns to step S203, and thereafter, steps up to step S212 are repeated.

As described above, the image processing device 10 includes an acquisition unit and an estimation unit. The acquisition unit acquires a first image and a second image captured of randomly stacked workpieces. The estimation unit generates a matching map between the features of the first image and the features of the second image, estimates the position, orientation, and class classification score of each target workpiece for each of the first image and the second image, and calculates the depth from the stereo camera to the workpiece by estimating the workpiece position based on the matching result using the attention map and the position estimation result. This makes it possible to suppress erroneous detection in object recognition.

(Image Processing Variations)
The object grasping system 1 has been described above, but is not limited to the above description, and various modifications are possible without departing from the spirit thereof. For example, the case where there is one type of object (workpiece) has been described, but the present invention is not limited to this, and the image processing device 10 may be configured to detect multiple types of workpieces. In addition, the image processing device 10 may not only detect the object, but also detect the position and posture of a tray on which the object is placed. FIG. 13 is a diagram showing an example of a bulk pile image including a tray according to a modified example. In the example shown in FIG. 13, the image processing device 10 can set a trajectory such that the robot arm 30 does not collide with the tray by specifying the position and posture of the tray on which the object is placed. The tray, which is the object to be detected, is an example of an obstacle. The image processing device 10 may be configured to detect obstacles other than the tray.

Also, while an example has been described in which the image processing device 10 divides the bulk image into, for example, a 40 x 40 grid, this is not limiting, and the object may be detected by dividing it into finer or coarser grids, or the estimation process may be performed on a pixel-by-pixel basis. This allows the image processing device 10 to more accurately calculate the distance between the camera and the object. Figure 14 is a diagram showing an example of a position shift estimation model according to a modified example. As shown in Figure 14, the image processing device 10 may cut out and combine parts of the left image P1L and the right image P1R that are smaller in size than the grids around the estimated position. Then, an estimation process may be performed similarly to the estimation process described above, and the position shift may be estimated based on the processing results.

When performing estimation processing in fine or coarse grid units or pixel units, estimation processing may be performed separately for the left image P1L and the right image P1R, as described above. FIG. 15 is a diagram showing another example of a position shift estimation model according to a modified example. In the example shown in FIG. 15, the image processing device 10 performs estimation processing separately for the left image P1L and the right image P1R. In this case, the image processing device 10 may share the weighting for the left image P1L with the weighting for the right image P1R when performing each estimation processing, as described above.

The estimation process described above may also be performed on the robot arm 30, the workpieces 41 and 42 held by the robot arm 30, or the workpieces 41 and 42 aligned at the alignment destination, rather than on images of the randomly stacked workpieces 41 and 42.

(Acquisition and learning of 3D data of object)
Fig. 16 is a block diagram showing an example of the configuration of a system for acquiring (measuring) and learning three-dimensional data of an object. In Fig. 16, a processing device 110 and a three-dimensional data measuring device 140 are communicably connected through a network NW. The processing device 110 also includes a communication I/F (interface) 111, an input I/F 112, a display 113, a memory circuit 114, and a processing circuit 115.

The three-dimensional data measuring device 140 is connected to the projector 120 and the camera 130. The three-dimensional data measuring device 140 includes a communication I/F 141, an input I/F 142, a display 143, a memory circuit 144, and a processing circuit 145.

Projector 120 is capable of irradiating (projecting) a predetermined pattern using light with a shorter wavelength than visible light (for example, UV (ultraviolet) light), and is also capable of illumination using visible light. Visible light illumination is performed using white light when camera 130 is a color camera, and using red light, blue light, and green light when camera 130 is a monochrome camera. A visible light illumination device may be provided separately from projector 120. Camera 130 is a color or monochrome camera capable of capturing visible light.

In the processing device 110, the communication I/F 111 controls communication of data input/output with external devices via the network NW. For example, the communication I/F 111 is realized by a network card, a network adapter, a NIC (Network Interface Controller), etc.

The input I/F 112 is connected to the processing circuit 115, converts input operations received from an administrator (not shown) of the processing device 110 into electrical signals, and outputs the electrical signals to the processing circuit 115. For example, the input I/F 112 is a switch button, a mouse, a keyboard, a touch panel, etc.

The display 113 is connected to the processing circuit 115 and displays various information and image data output from the processing circuit 115. For example, the display 113 is realized by a liquid crystal monitor, a CRT (Cathode Ray Tube) monitor, a touch panel, etc.

The memory circuit 114 is realized by, for example, a storage device such as a memory. The memory circuit 114 stores various programs executed by the processing circuit 115. In addition, the memory circuit 114 temporarily stores various data used when the processing circuit 115 executes various programs. The memory circuit 114 has three-dimensional object data 1141 and a machine (deep) learning model 1142. The three-dimensional object data 1141 includes three-dimensional shape data and texture data for various objects. The texture data referred to here is RGB (Red, Green, Blue) color data, metal (reflection), roughness (surface roughness), transparency, etc., which represent texture. In addition, the texture data to be acquired may be only RGB color data, or may include either or both of data related to color and texture.

Furthermore, the machine (deep) learning model 1142 includes a neural network structure 1142a and learning parameters 1142b. The neural network structure 1142a is, for example, an application of a known network such as the convolutional neural network b1 in FIG. 5, and is the network structure shown in FIG. 12. The learning parameters 1142b are, for example, the weights of the convolutional filter of the convolutional neural network, and are parameters that are learned and optimized to estimate the position and orientation of the target object.

The machine (deep) learning model 1142 is used in the object grasping system 1 (Figs. 1 and 2) for processing to estimate the position and orientation of a workpiece from an image output from the camera 20 (Figs. 1 and 2). The machine (deep) learning model 1142 is generated, for example, by learning from the positions and orientations of multiple workpieces and images of the multiple workpieces as teacher data. Note that here, the machine (deep) learning model 1142 is generated, for example, by the processing circuit 115, but is not limited to this and may be generated by an external computer.

The processing circuitry 115 is realized by a processor such as a CPU (Central Processing Unit). The processing circuitry 115 controls the entire processing device 110. The processing circuitry 115 reads various programs stored in the memory circuitry 114 and executes the read programs to perform various processes. For example, the processing circuitry 115 has a learning unit 1151 and a data output unit 1152.

The learning unit 1151 generates images of the randomly piled workpieces using image generation software based on the object 3D data 1141 (including both data newly measured and accumulated by the 3D data measuring device 140 and data accumulated in the past), creates learning data for machine learning, performs machine learning on the machine (deep) learning model 1142 based on the learning data, and updates the learning parameters 1142b.

The data output unit 1152 outputs the three-dimensional object data 1141 and the machine (deep) learning model 1142 data stored in the memory circuitry 114 in response to an instruction from an operator or an external request.

In the three-dimensional data measuring device 140, the communication I/F 141 controls data input/output communication with external devices via the network NW. For example, the communication I/F 141 is realized by a network card, a network adapter, a NIC, etc. The communication I/F 141 also transmits control signals to be output to the projector 120 in accordance with standards such as HDMI (registered trademark) (High-Definition Multimedia Interface), and receives status signals from the projector 120. The communication I/F 141 also transmits control signals to the camera 130, and receives image data output from the camera 130.

The input I/F 142 is connected to the processing circuit 145, converts input operations received from an administrator (not shown) of the three-dimensional data measuring device 140 into electrical signals, and outputs the electrical signals to the processing circuit 145. For example, the input I/F 142 is a switch button, a mouse, a keyboard, a touch panel, etc.

The display 143 is connected to the processing circuit 145 and displays various information and image data output from the processing circuit 145. For example, the display 143 is realized by an LCD monitor, a CRT monitor, a touch panel, etc.

The memory circuitry 144 is realized by, for example, a storage device such as a memory. The memory circuitry 144 stores various programs executed by the processing circuitry 145. The memory circuitry 144 also temporarily stores various data used when the various programs are executed by the processing circuitry 145. The memory circuitry 144 has object three-dimensional data 1441. The object three-dimensional data 1441 is original data for a portion of the object three-dimensional data 1141 stored in the memory circuitry 114 of the processing device 110, and is accumulated in the memory circuitry 114 via the communication I/F 141, the network NW, and the communication I/F 111 of the processing device 110.

The processing circuitry 145 of the three-dimensional data measuring device 140 is realized by a processor such as a CPU. The processing circuitry 145 controls the entire three-dimensional data measuring device 140. The processing circuitry 145 reads various programs stored in the memory circuitry 144 and executes the read programs to perform various processes. For example, the processing circuitry 145 has an object three-dimensional data measuring unit 1451.

Three-dimensional object data measuring unit 1451 controls projector 120 and camera 130 via communication I/F 141, measures the three-dimensional shape and texture from the image of the object captured by camera 130, and acquires three-dimensional object data including three-dimensional shape data and texture data. The three-dimensional shape of the object is measured based on an image obtained by irradiating the object with a predetermined pattern and capturing the image, for example, using a known lattice method. The three-dimensional shape of the object is measured while changing the posture of the object.

Figure 17 is a flow chart showing an example of the process of acquiring three-dimensional data of an object, and is an example of the process when the image is captured by a color camera 130. As a preliminary preparation for the process, a specified paint is applied to the object. The specified paint is a fluorescent paint that emits visible light when irradiated with light with a shorter wavelength than visible light (e.g., UV (ultraviolet) light) and is colorless and transparent to visible light. This can also be called invisible paint. Note that, although light with a shorter wavelength than visible light is described below as UV light, this is not limited to this. For example, light with a shorter wavelength than visible light can be generated by two-photon excitation, and this can be used to irradiate the pattern light.

In FIG. 17, when the three-dimensional data measuring device 140 starts processing, it irradiates a UV pattern toward the object using the projector 120. Then, the object is photographed using the camera 130, which is in a predetermined positional relationship with the projector 120, and the three-dimensional shape of the object is calculated based on the photographed image and stored in a memory circuit (step S11). FIG. 18 is a diagram showing an example of a state in which a UV pattern is irradiated from the projector 120 onto the object W. The irradiation of UV light from the projector 120 causes the object W to emit light in a predetermined pattern due to invisible paint, and this state is photographed by the camera 130. Since the emission of visible light is due to invisible paint, it does not matter if the object W is a glossy object or a black object.

Returning to FIG. 17, after the second or subsequent UV patterns have been irradiated, the calculation of the three-dimensional shape of the object (step S11) is performed taking into account the images captured under the previous UV patterns. This improves the accuracy of the measurement of the three-dimensional shape. Alternatively, only image capture may be performed when the UV pattern is irradiated, and after image capture has been completed for all UV patterns, the three-dimensional shape of the object may be calculated based on all the captured images. Alternatively, when the UV pattern is irradiated and captured, processing may be performed only on the images captured under that UV pattern, and after image capture has been completed for all UV patterns, calculation may be performed taking into account all the images.

The three-dimensional data measuring device 140 then determines whether processing for the specified UV pattern has been completed (step S12), and if it determines that processing has not been completed (No in step S12), it repeats the process from irradiating another UV pattern (step S11). If it determines that processing for the specified UV pattern has been completed (Yes in step S12), it moves on to the next process.

As the next process, the three-dimensional data measuring device 140 irradiates the object with white light from the projector 120 or an external light source, captures an image with the camera 130, and acquires the texture based on the captured image and stores it in the memory circuit (step S13). Figure 19 is a diagram showing an example of the state in which visible light is irradiated onto the object W, in which the white light emitted from the projector 120 causes the invisible paint on the object W to become colorless and transparent, revealing the original texture of the object W, which is then captured by the camera 130.

In addition, since it can be assumed that the object is not moving based on the image captured under the UV pattern, it is easy to match the three-dimensional shape and texture of the object obtained from the image captured under the UV pattern.

Returning to FIG. 17, the three-dimensional data measuring device 140 performs point cloud data matching and updates the data in the second and subsequent processes after the posture change (step S14). That is, the three-dimensional data measuring device 140 links the data of the three-dimensional shape of the object acquired in the previous process and the data of the three-dimensional shape of the object acquired in the current process based on parts that can be determined to be identical, and updates the data of the three-dimensional shape of the object. This makes it possible to obtain the three-dimensional shape of the object that cannot be captured by photographing only from one side.

The three-dimensional data measuring device 140 then determines whether processing has been completed for the specified orientations that cover the entire appearance of the object (step S15), and if it determines that processing has not been completed (No in step S15), it changes the orientation of the object (step S16) and repeats the process from irradiation of the UV pattern (step S11). The orientation of the object may be changed manually by an operator or by a drive mechanism. If the three-dimensional data measuring device 140 determines that processing has been completed for the specified orientations (Yes in step S15), it ends the process.

Figure 20 is a flowchart showing another example of the process for acquiring three-dimensional data of an object, and is a processing example in which the image is captured by a monochrome camera 130. The preparations required before the process are the same as those described above.

In FIG. 20, when the three-dimensional data measuring device 140 starts processing, the projector 120 projects a UV pattern toward the object. Then, the camera 130, which is in a predetermined positional relationship with the projector 120, takes an image of the object, calculates the three-dimensional shape of the object based on the captured image, and stores it in a memory circuit (step S21). After the second or subsequent UV patterns are irradiated, the calculation of the three-dimensional shape of the object is performed taking into account the images captured under the previous UV pattern. Note that only the image is captured when the UV pattern is irradiated, and after the image capture is completed for all UV patterns, the three-dimensional shape of the object may be calculated based on all the captured images. Also, when the UV pattern is irradiated and photographed, processing is performed only on the image captured under that UV pattern, and after the image capture is completed for all UV patterns, calculation is performed taking into account all the images.

The three-dimensional data measuring device 140 then determines whether processing for the specified UV pattern has been completed (step S22), and if it determines that processing has not been completed (No in step S22), it repeats the process from irradiating another UV pattern (step S21). If it determines that processing for the specified UV pattern has been completed (Yes in step S22), it moves on to the next process.

As the next process, the three-dimensional data measuring device 140 irradiates the object with red light from the projector 120 or an external light source, takes an image using the camera 130, and acquires the red component of the texture based on the captured image and stores it in the memory circuit (step S23). Since it can be assumed that the object has not moved based on the image captured under UV pattern irradiation, it is easy to match the three-dimensional shape of the object acquired from the image captured under the UV pattern with the texture.

Next, the three-dimensional data measuring device 140 irradiates the object with blue light from the projector 120 or an external light source, captures an image with the camera 130, and acquires the blue component of the texture based on the captured image and stores it in the memory circuit (step S24).

Next, the three-dimensional data measuring device 140 irradiates the target with green light from the projector 120 or an external light source, captures an image using the camera 130, and obtains the green component of the texture based on the captured image and stores it in the memory circuit (step S25).

Next, in the second or subsequent processing after the posture change, the three-dimensional data measuring device 140 performs point cloud data matching and updates the data (step S26). That is, the three-dimensional data measuring device 140 links the data of the three-dimensional shape of the object acquired in the previous processing and the data of the three-dimensional shape of the object acquired in the current processing based on parts that can be determined to be identical, and updates the data of the three-dimensional shape of the object.

Then, the three-dimensional data measuring device 140 judges whether the processing has been completed for the specified orientations that cover the entire appearance of the object (step S27), and if it is judged not to be completed (No in step S27), it changes the orientation of the object (step S28) and repeats the process from irradiation of the UV pattern, etc. (step S21). The orientation of the object may be changed manually by the operator or by a drive mechanism. If the three-dimensional data measuring device 140 judges that the processing has been completed for the specified orientations (Yes in step S27), it ends the process.

Figure 21 is a flowchart showing an example of the generation of learning data and the learning process. Below, a case where the learning model is generated and updated by the processing device 110 shown in Figure 16 is described, but this is not limited to this, and may be performed by another computer (not shown).

As shown in FIG. 21, the processing device 110 acquires three-dimensional data of a desired object from the object three-dimensional data 1141 (step S101). FIG. 22 is a diagram showing an example of three-dimensional data of an object. By acquiring the three-dimensional data, the posture of the workpiece can be arbitrarily changed and positioned in the virtual space.

Returning to FIG. 21, the processing device 110 then sets various conditions for placing objects in the virtual space (step S102). Objects can be placed in the virtual space using, for example, known image generation software. Conditions such as the number, position, and orientation of objects to be placed can be set so that the image generation software randomly generates the objects, but this is not limiting and the administrator of the processing device 110 may set them as desired.

Next, the processing device 110 places objects in the virtual space according to the set conditions (step S103). Next, the processing device 110 acquires images, positions, and orientations of the placed objects, for example, by capturing the virtual space in which the objects are placed (step S104). Here, the position and orientation of the object are indicated, for example, by three-dimensional coordinates (x, y, z), and the orientation of the object is indicated by a quaternion (qx, qy, qz, qw), which is a quaternion representing the orientation or rotation state of an object.

23 is a diagram showing an example of a captured image of a virtual space in which multiple objects are arranged. As shown in FIG. 23, multiple objects W1a and W1b are arranged in a virtual space at random positions and orientations. In addition, hereinafter, an image of randomly arranged objects may be referred to as a "randomly piled image". Next, the processing device 110 stores the acquired image and the positions and orientations of the arranged objects in the memory circuit 14 (step S105). Furthermore, the processing device 110 repeats steps S102 to S105 a predetermined number of times (step S106). Note that the combination of the image acquired by the above steps and the positions and orientations of the objects arranged, which is stored in the memory circuit 114, may be referred to as "teacher data". By repeating the processes from steps S102 to S105 a predetermined number of times, a sufficient number of teacher data is generated to repeatedly perform the learning process.

The processing device 110 then performs a learning process using the generated teacher data a predetermined number of times to generate or update learning parameters 1142b used as weights in the neural network structure 1142a (step S107). In this way, by placing the object whose three-dimensional data has been acquired in the virtual space, teacher data including a combination of an image of the object and its position and orientation, which is used in the learning process, can be easily generated.

Although the embodiment of the present invention has been described above, the present invention is not limited to the above embodiment, and various modifications are possible without departing from the spirit of the present invention.

As described above, the object 3D data measurement method according to the embodiment includes a first step of irradiating an object coated with a colorless and transparent paint that emits visible light when irradiated with light having a shorter wavelength than visible light in a predetermined pattern, photographing the object, and measuring the 3D shape of the object from the photographed image, and a second step of irradiating the object with visible light, photographing the object, and measuring the texture of the object from the photographed image. This makes it possible to easily measure 3D object data including the 3D shape and texture from the actual object.

In addition, in the first process, the three-dimensional shape of the object is measured by switching between multiple predetermined patterns for one posture of the object. This can improve the accuracy of the measurement of the three-dimensional shape.

In addition, the posture of the object is changed, and the measurement of the object's three-dimensional shape in the first process and the measurement of the object's texture in the second process are repeated, linking and updating the data through point cloud data matching. This makes it possible to obtain the three-dimensional shape of the object that cannot be captured by photographing only from one side.

In addition, in the second step, if the image is captured using a color camera, white light is irradiated and the image is captured, and if the image is captured using a monochrome camera, red light, blue light, and green light are irradiated and the image is captured. This allows for flexible compatibility with both color and monochrome cameras.

In addition, the object 3D data measuring device according to the embodiment emits visible light when irradiated with light having a shorter wavelength than visible light, and the processing circuit executes a first process of irradiating an object coated with a colorless and transparent paint with light having a shorter wavelength than visible light in a predetermined pattern, photographing the object, and measuring the 3D shape of the object from the photographed image, and a second process of irradiating the object with visible light, photographing the object, and measuring the texture of the object from the photographed image. This makes it possible to realize the above-mentioned object 3D data measuring method as a device.

In addition, the learning data generation method according to the embodiment generates images of randomly piled workpieces using image generation software based on the three-dimensional data of the object obtained by the above-mentioned three-dimensional data measurement method of the object, and uses these images as learning data for machine learning. This reduces the cost required for preparing the learning data.

In addition, the learning data generation device according to the embodiment uses image generation software to generate images of randomly piled workpieces based on the three-dimensional data of the object obtained by the above-mentioned method for measuring three-dimensional data of the object, and the processing circuit executes processing to use the images as learning data for machine learning. This allows the above-mentioned learning data generation method to be realized as a device.

Furthermore, the present invention is not limited to the above-mentioned embodiment. The present invention also includes configurations in which the above-mentioned components are appropriately combined. Furthermore, further effects and modifications can be easily derived by those skilled in the art. Therefore, the broader aspects of the present invention are not limited to the above-mentioned embodiment, and various modifications are possible.

110 Processing device, 114 Memory circuit, 1141 Object 3D data, 1142 Machine (deep) learning model, 1142a Neural network structure, 1142b Learning parameters, 115 Processing circuit, 1151 Learning unit, 1152 Data output unit, 120 Projector, 130 Camera, 140 3D data measuring device, 144 Memory circuit, 1441 Object 3D data, 145 Processing circuit, 1451 Object 3D data measuring unit, W Object

Claims (6)

a first step of irradiating a target object, which emits visible light when irradiated with light having a wavelength shorter than that of visible light and is coated with a colorless and transparent paint that is irradiated with light having a wavelength shorter than that of visible light in a predetermined pattern, photographing the target object, and measuring a three-dimensional shape of the target object from the photographed image;
a second step of irradiating the object with visible light, photographing the object, and measuring a texture of the object from the photographed image;
A method for measuring three-dimensional data of an object, comprising: In the first step, a three-dimensional shape of the object is measured by switching among a plurality of the predetermined patterns for one posture of the object.
2. The method for measuring three-dimensional data of an object according to claim 1. changing the posture of the object, repeating the measurement of the three-dimensional shape of the object in the first step and the measurement of the texture of the object in the second step, and linking and updating the data by point cloud data matching;
3. The method for measuring three-dimensional data of an object according to claim 1 or 2. In the second step, when the photographing is performed with a color camera, white light is irradiated and photographed, and when the photographing is performed with a monochrome camera, red light, blue light, and green light are irradiated and photographed, respectively.
The method for measuring three-dimensional data of an object according to any one of claims 1 to 3. Based on the three-dimensional data of the object obtained by the method for measuring three-dimensional data of the object according to claim 1, an image of the randomly piled workpieces is generated by image generation software, and the generated image is used as learning data for machine learning.
Training data generation method. A processing circuit executes a process of generating an image of the randomly piled workpieces using image generation software based on the object three-dimensional data obtained by the object three-dimensional data measuring method according to claim 1, and using the image as learning data for machine learning.
Training data generation device.

Images