JP3028016B2 - 3D image measurement method for cargo - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention is used for observing a box-like cargo such as a corrugated cardboard box by a slit light projection method called a light cutting method or a space coded pattern light projection method. The present invention relates to a method for measuring a three-dimensional image of a cargo.

[0002]

2. Description of the Related Art In a physical distribution process, a box-shaped load is often stacked on a pallet or the like and transported to another place for unloading. In such an operation, the box-shaped cargo regularly stacked by the palletizer can be unloaded by a general-purpose robot or a depalletizer without a visual system. However, an irregular pattern may be loaded by a human interrupting work, or the posture or position of the load may be shifted due to a trouble during transportation. Therefore, a visual system that monitors a three-dimensional pattern of a cargo is desired in order to prevent a state in which the robot cannot perform the deparing operation and increase reliability and flexibility. There have been several examples of such a cargo vision system that have been put to practical use, but none of them are satisfactory.

As general applications, various three-dimensional image measurement methods for measuring the position and shape of an object without contact have been proposed. For example, as a method based on triangulation, a slit light projection method called a light cutting method, There is a space coded pattern light projection method and the like. If this method is applied to the measurement of cargo, it is considered that the result is as shown in FIG. In the slit light projection method, the measurement space of the cargo W
0 to Rn, and each of the measurement areas R0 to R
This is a method of sequentially irradiating slit light every n and capturing an image with the camera 52. By superimposing the captured images of each time, a three-dimensional image having the distance data to the cargo W on the pixel data is obtained. That is, a plane forming a wedge-shaped space,
The three-dimensional coordinates of the point of interest P are obtained from the intersection of the point of interest P with the line of sight of the camera 52. Although the figure shows about ten and several wedge-shaped regions R0 to Rn, it is generally divided into several hundreds.

[0004] Instead of slit light, various pattern lights are radiated in time series to produce a wedge-shaped measurement region R.
An image having distance data divided into 0 to Rn is obtained. The method of irradiating and measuring light in this way is the spatially coded pattern light projection method.

[0005]

However, FIG. 16 (A)
In any of the methods, the three-dimensional coordinates are obtained for all the pixels, the shape is recognized, and the position and orientation of the uppermost cargo W are measured. Take it. Generally, in the unloading operation, it is sufficient to know the plane shape, the position of the center of gravity, the inclination in the plane, the height, and the like of the uppermost load W, and it is not necessary to know the shapes of all the loads W as described above. Therefore, in the above method, time is consumed by useless calculation.

For this reason, the present inventor has proposed a spatial code image of the entire cargo W obtained as shown in FIG. 1B (the spatial code corresponding to the wedge-shaped regions R0 to Rn as the pixel value of each pixel Pc). from an image) including, extracts an image of only the load W ₀ uppermost as shown in FIG. (C), I thought to perform coordinate calculations, and the like only the extracted image. In this case, the image is extracted as follows. That is, the wedge-shaped regions R0 to Rn on the upper surface of each work W are arranged in a straight line parallel to the irradiation direction R of the light from the light projector 51, and each pixel P in the horizontal direction of the screen.
c will have the same spatial code in the same plane. Therefore, by picking out the pixel Pc having the largest spatial code for each column of the screen horizontally, the uppermost image cargo W ₀ as in FIG (C) is considered to be extracted.

However, when the cargo W is a cardboard box, the upper surface often has a curved shape deformed in the height direction as shown in FIG. In this case, the wedge-shaped regions R0 to Rn on the upper surface of the load become lines distorted in a waveform as shown in FIG. 9B, and in the method of selecting the pixel Pc having the maximum spatial code for each column as described above, the same is applied. Only the discrete pixels Pc are left as shown in FIG. 3C, and the top shape of the cargo W cannot be recognized.

An object of the present invention is to provide a three-dimensional image measuring method capable of obtaining an image of the uppermost cargo separately from other cargos by simple and quick processing.
Another object of the present invention is to provide a three-dimensional image measurement method capable of obtaining the position of the center of gravity of the uppermost cargo by simple processing.

[0009]

The three-dimensional image measuring method according to the present invention includes a step of obtaining a space code image of the entire cargo and a step of extracting an image of the uppermost cargo from the space code image.
The spatial code image is obtained as follows. That is, the measurement space of the cargo (W) is divided into wedge-shaped measurement regions (r0 to rn) stacked on each other to determine a spatial code in each measurement region, and a spatial code image including the spatial code in each pixel data. Get. In this case, for example, each wedge-shaped region (r0
To rn), and a spatial code image can be obtained by superimposing the images. The image extraction of the uppermost cargo is performed by leaving only pixels including a space code within a specified value from the space code indicating the wedge-shaped region at the maximum angle for each pixel row in a predetermined direction in the space code image of the entire cargo.

A three-dimensional image measuring method according to a second aspect of the present invention extracts an image of the uppermost cargo by the measuring method of the first aspect, and then obtains a cubic image of a plurality of points on each side of the upper surface of the uppermost cargo from the image data. In this method, original coordinates are calculated, and the position of the center of gravity of the upper surface of the uppermost cargo on the horizontal projection plane is obtained from the obtained three-dimensional coordinates.

[0011]

According to the method of the present invention, the image of the uppermost cargo is extracted by leaving only the pixels including the spatial code within the specified value from the spatial code indicating the wedge-shaped region at the maximum angle. Also, an image of a portion higher than the height lowered by a predetermined value remains. Therefore, even if the top shape of the uppermost cargo is slightly deformed in the height direction, an image showing the shape of the entire top surface can be obtained. In addition, since only the image of the upper surface of the uppermost cargo is extracted as described above, the amount of data required for calculations in various subsequent processes can be reduced, and necessary data relating to the uppermost cargo can be obtained easily and quickly. .

According to the second aspect of the present invention, three-dimensional coordinates of predetermined points on each side are calculated from the image data of the uppermost cargo,
Since the position of the center of gravity of the upper surface of the load with respect to the horizontal projection plane is obtained from the obtained three-dimensional coordinates, even if the line-of-sight direction of the imaging means is different, or even if the shape of the upper surface of the load is vertically deformed,
The position of the center of gravity to be obtained does not change, and the position of the center of gravity of the upper surface of the uppermost cargo can be accurately obtained.

[0013]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS One embodiment of the present invention will be described with reference to FIGS. FIG. 1 is a schematic explanatory view showing the entire processing procedure of image measurement, and FIGS. 2 to 8 are FIGS.
It is an enlarged view of (J). As shown in FIG. 2, the measurement target is the entire cargo W stacked on the pallet 1. The cargo W is made of a cardboard box, and a joining line Wa of a double-opening lid is formed at the center of the upper surface. FIG. 2 (A1) is the same as FIG. 2 (A2).
FIG. 2 is a perspective view (image) of the cargo W and the like viewed from the direction of the line of sight q.

As the image measuring means, a light projector 2 disposed above and behind the pallet 1 and a CCD (solid-state image pickup device) camera 3 disposed above and directly below the center of the pallet 1 are used. In this embodiment, a so-called space-coded pattern light projection method is adopted, and various light patterns are projected on the light projector 2 in a time-series manner, so that the measurement spaces are wedge-shaped measurement areas r0 to r0. rn is used.

FIG. 12 shows a configuration example of a measurement system. The projector 2 is provided with a liquid crystal shutter 4. The liquid crystal shutter 4 forms a thin striped optical shutter array made of liquid crystal, and controls a light transmittance of each stripe on / off by an electric signal to form a desired light pattern. Floodlight 2
The dedicated image processing device 5 is in charge of the control of blinking, the change of the pattern of the liquid crystal shutter 4, and the imaging timing of the camera 3, and performs a predetermined operation according to the input of the control command. The resolution of the camera 3 is, for example, 512 × 512 pixels,
The image memory of the image processing device 5 has 512 × 512 × 8 bits (= 256 Kbytes). Upon receiving the command, the image processing device 5 generates an image in which the space is coded, and transfers the image data to the host workstation 6. The host workstation 6 performs the extraction of the uppermost image described later, the calculation of the three-dimensional coordinate value, the calculation of the center of gravity, the calculation of the height of the upper surface, and the like from the image data. The image processing device 5 has a monitor display 7, and the host workstation 6 has a display 8.

FIG. 14 shows the principle of spatial coding using pattern light. Before describing the embodiments, this principle will be described first. In the figure, the portion where light is shining is "1".
Is set to “0” for the portion that does not hit,
, The rear half is coded as “1” and the front half is coded as “0”. Next, when the pattern B is projected, the last quarter becomes “1” and the next quarter becomes “0”.
The next quarter is coded as “1” and the first quarter is coded as “0”. Similarly, when the pattern C is projected, “1”, “0”, “1”, “0”,.
Is coded.

As described above, the space can be divided into eight wedge-shaped regions using three patterns A, B, and C.
Thus, for each wedge-shaped region, "000"-
A space coded into 3 bits up to “111” is obtained.
The point T in the figure is located in the wedge-shaped area indicated by the code “101”. In practice, including this embodiment, in order to reduce bit errors, a gray code (alternate binary code) pattern shown in FIG. 15 is used.

The binary number of a predetermined number of bits obtained in each wedge-shaped area in this manner is called a "space code". In the slit light projection method, slit light is sequentially projected for each wedge-shaped region, and an appropriate code given to distinguish the wedge-shaped region is also referred to as a “space code”. Images that use these spatial codes as pixel values are called “spatial code images”.
Call.

FIG. 13 shows a basic flow of measurement processing from the projection of light using a gray code pattern light to the acquisition of a distance image. The following ~ for one light pattern
The processing up to is performed. The gray code pattern light is projected into the measurement space. An object is imaged by the camera 3 and the brightness of the input image is binarized. The binarized image is inserted into the bit plane of the multi-level image memory according to the order of the light projection pattern. By repeating these processes for n light patterns, a spatial code image represented by an n-bit gray code is obtained in the image memory. This space code can be converted from a gray code to a binary code, and three-dimensional coordinates can be calculated from the space code image to obtain a distance image.

The calculation of the three-dimensional position can be performed according to the following principle, although the calculation formula is omitted. As shown in FIG. 14, an area obtained by thinly dividing the measurement space into a wedge shape can be regarded as a single plane, and the three-dimensional position of the object surface is uniquely obtained as an intersection between the plane and the line of sight of the camera. . The surface direction of the wedge-shaped region corresponds to the space code of the region, and the equation of the surface can be immediately determined from the value of the space code. On the other hand, the equation of the straight line formed by the camera's line of sight can be determined from the position of the pixel of interest in the image. Therefore, the coordinate value of the measurement point on the surface of the object can be calculated from the camera coordinates of the pixel of interest and the spatial code at that pixel.

Return to the description of the embodiment. In this embodiment, using the measurement system of FIG. 12 described above, first, a spatial code image using gray code pattern light is obtained for the cargo W of FIG. 2 according to the principle described with reference to FIGS. FIG. 3 (A)
Shows a gray-scale image of the work W viewed from above the pallet 1. The grayscale image is not necessarily required to be obtained by this three-dimensional image measurement method, but is shown for reference. 3 (C1) to (C)
3) shows an image obtained by the camera 3 by light projected on each of the wedge-shaped measurement regions r0 to rn as shown in FIG. 2, that is, a light-section image. The hatched portion in the drawing is a portion that is imaged as a bright area when light is applied to the cargo W. Note that, in the spatially coded pattern light projection method employed in this embodiment, the wedge-shaped regions r0 to rn in FIG. 2 are regions divided by the result of the combination of pattern lights, and are shown in FIGS. 3 (C1) to 3 (C3). Does not show images at the time of one light projection. When the slit light projection method is adopted, since slit light is projected for each of the wedge-shaped regions r0 to rn, FIGS. 3 (C1) to (C3) show images at the time of one light projection. .

FIG. 4C shows a case where the light projection angle is φa (the region r3 in FIG. 2 and the space code “3” in FIG. 4), and only the cargo W on the rear side at the bottom is imaged. Figure (C2)
Is a case in which the light projection angle is set to an angle φb larger than the above, and the rear part is imaged as the load W in the upper stage increases. FIG. (C3) shows a case where the light projection angle is further increased to φc, and only the uppermost front cargo W is imaged.

With such a collection of image data, a space code image of the entire cargo shown in FIG. 4 is obtained. In the figure, the numbers attached to each load W indicate the space code, and the thin line indicates the boundary line of the portion where the space code has the same value. It should be noted that, in the figure, the number of divisions is smaller than the actual number of divisions to make the figure easier to see. The hatched area in FIG. 3 indicates a portion where there is no load W or a portion where light from the projector 2 is blocked by another load W and image data as a load surface cannot be obtained.

From the entire space code image, an image of only the uppermost cargo W shown in FIG. 5 is extracted by the following method. That is, from the spatial code image of FIG. 4, for each horizontal pixel row L (L1, L2,...), Only pixels including a spatial code within a specified value from the spatial code indicating the wedge-shaped region at the maximum angle are left. .

For example, in the spatial code image of FIG.
In the horizontal pixel row L4, the spatial code of the pixel row portion of the lowermost cargo W is “4”, the spatial code of the pixel row portion of the second cargo W is “8”, and the uppermost cargo W The spatial code of the pixel column portion is “12”, but the pixel having the maximum spatial code “12” remains. In this case, in the pixel row L4, the spatial code of each pixel of the uppermost cargo W is not actually all “12” but is “11” or “10” in some places. Assuming that the value is set to “2”, “2” is larger than the maximum value “12” of the spatial code.
Only the small space code "10" or more is left.

By repeating such a process for each pixel row L, only the image of the uppermost cargo W is extracted, and the upper surface shape of the uppermost cargo W is, for example, as shown in FIG. Even if it is slightly deformed, an image showing the shape of the entire upper surface can be obtained. However, as a precondition, it is assumed that each of the divided regions r0 to rn in FIG. 2 is a band-shaped portion extending in the horizontal direction on the screen, and the space code is sequentially increased from top to bottom on the screen. In the case of the slit light projection method, the light projected from the light projector 2 is a band-like light extending in the horizontal direction of the screen, and is projected on the screen from top to bottom.

The above extraction method will be described in more detail with reference to FIGS. FIG. 10 shows a space code image of the part F in FIG. In the figure, one mesh portion indicates one pixel Pc, and the number therein indicates a space code value. In the case of an image on a complete horizontal plane, the spatial code has the same value for all pixels in the same horizontal row, and increases by "1" every time the row goes down. Since the top surface shape of the cargo W in FIG. 9 is deformed up and down, the space code of each pixel slightly varies. That is, in the first pixel row L1, the maximum value of the space code is “9”, and there are portions where the space code is “7”. In this case, if the prescribed value is set to “2”, the maximum value is “9”.
Each pixel having a space code of “7” or more, which is smaller than “2”, is left, and pixels having a space code of “5” or “4” are discarded. In the second pixel row L2, since the maximum value of the space code is "10", each pixel having the space code of "8" or more remains. By repeating such processing, all the pixels indicating the upper surface of the uppermost cargo W are left, the pixels of the next lower cargo W (indicated by chain lines) are discarded, and the space of only the uppermost cargo W is discarded. A code image is extracted.

FIG. 11 is a flowchart showing a specific example of this extraction processing. First, in step S1, a row (horizontal column) from which pixel extraction is started is set to the top row of the image,
Set the maximum image extraction value.

Next, the maximum value of the space code in the pixels of one row (one column of pixels) in the set row is searched (S2), and an extracted minimum spatial code value smaller than the maximum value by a specified value is set. (S4). The specified value is input and set in advance to an appropriate value. Thereafter, pixels having a spatial code equal to or greater than the extracted minimum spatial code value are extracted for one row as described above with reference to FIG. 10 (S5).

When the extraction for one line is completed, the line counter is incremented (S7), and the process returns to step S2.

In the processing of the second and subsequent rows, the maximum value of the spatial code is searched for in row unit in step S2, and the maximum value is compared with the maximum image extraction value (S3). , Update the maximum image extraction value. By repeating such processing up to the last line, an appropriate image can be extracted regardless of the arrangement of the uppermost cargo W.

When an extracted image of the uppermost cargo is obtained, the images are grouped as shown in FIG. 6, and the objects to be measured are divided into the cargos W _A , W _B , and W _C. In the grouping, a region in which the space code is close to within a predetermined range between adjacent pixels, in other words, a region in which the wedge-shaped regions r0 to rn are close to each other are defined as one group, and the same label is assigned to pixels in the same group. (Number). In other words, the grouping is performed by performing the labeling process not only to the general connection of pixels having the same value but also to the connection of pixels having adjacent values between adjacent pixels. Each load W _A, W _B, W _C group (area by utilizing the characteristics such as the specific cargo changes on the space code occurring arrangement of each pixel by the groove portion and gaps such as occurring between the edge between the Division) may be performed.

In the uppermost grouping screen, for each group, the contour line LI of each side forming the group is extracted as shown in FIG. 7, and the three-dimensional coordinate data is converted from the line. conversion point C ₁₁ ~C _13, C ₂₁
~C _23, C ₃₁ ~C _33, extracts the C ₄₁ -C _43. The number of conversion points to be extracted is set in advance based on a predetermined reference.

The extracted conversion points C ₁₁ -C ₁₃ , C ₂₁
-C _23, per _{C 31 ~C 33, C 41 ~C} 43, the pixel position in the screen on that point, by the angle of the surface forming the wedge-shaped region corresponding to the space code, the three-dimensional coordinates (world coordinates) Calculate. FIG. 8H shows a three-dimensional coordinate position of each point by a chain line.

The three-dimensional coordinates of each point thus obtained are shown in FIG.
Projecting on a horizontal (X, Y) plane as in (I),
The center of gravity G (X _G , X _G ) and the inclination θ of the measurement object on the X and Y planes are obtained. The data in the Z-axis direction (vertical direction)
As shown in FIG. 8 (J), three sides are extracted from the large data in the Z-axis direction at each point, and the average value (Z _G ) is taken to determine the upper surface position of the load. In the figure, the average value of the Z coordinate of each point of the side 1, the side 2, and the side 4 is defined as the upper surface position. As described above, since the average value of the three sides is set as the upper surface position, the upper surface position can be accurately measured even if the side 3 is deformed as in the example of FIG.

According to this three-dimensional image measurement method, it is possible to recognize the uppermost cargo W, the center of gravity of the upper surface position, and the inclination θ in the horizontal plane in this manner. For this reason, when performing operations such as unloading using a general-purpose robot or a depalletizer, even if the position and posture of the load W are variously different,
Work can be accurately performed by flexibly coping with this. In addition, a small amount of calculation is required, and high-speed operation can be performed with a simple calculation device.

Although the above embodiment employs the spatially coded pattern light projection method in the process of obtaining the space code image, the slit light projection method may be employed.

[0038]

According to the three-dimensional image measuring method of the present invention, since the image of the uppermost cargo is extracted while leaving the pixels of the space code within a specified value from the maximum angle, the image of the uppermost cargo is extracted with another cargo. An image showing the shape of the entire top surface can be obtained even if the top surface shape of the cargo is slightly deformed in the height direction. In addition, since only the image of the upper surface of the uppermost cargo is extracted in this manner, the amount of data required for calculations in various subsequent processes is small, and it is necessary to easily and quickly unload the uppermost cargo. Data can be obtained.

According to a second aspect of the present invention, the image data of the uppermost cargo obtained as described above is converted into three-dimensional coordinates, and then the position of the center of gravity of the upper surface of the cargo with respect to the horizontal projection plane is obtained. The center of gravity of the upper surface of the uppermost cargo can be accurately determined even if the direction is different or the shape of the upper surface of the cargo is vertically deformed. Therefore, even if the way of loading is disturbed, necessary data for facilitating unloading and the like can be obtained.

[Brief description of the drawings]

FIG. 1 is a schematic explanatory view showing the entire processing of a three-dimensional image measurement method according to an embodiment of the present invention.

FIGS. 2A and 2B are a perspective view and a left side view showing a relationship between a measurement target and a measurement unit.

FIG. 3 is an explanatory diagram of a light and shade image viewed from above a pallet and a light cut image at each light projection angle.

FIG. 4 is an explanatory diagram of a space code image.

FIG. 5 is an explanatory diagram of an extracted image of the uppermost cargo.

FIG. 6 is an explanatory diagram of a grouped image at the top.

FIG. 7 is an explanatory diagram of a conversion point extraction process into three-dimensional coordinate data.

FIG. 8 is an explanatory diagram of a state of conversion into three-dimensional coordinates, and a process of creating a center of gravity position and an average height.

FIG. 9 is a perspective view showing a modified example of a load.

FIG. 10 is an explanatory diagram of a space code image corresponding to a portion F in FIG. 9;

FIG. 11 is a flowchart of a pixel extraction process of the uppermost cargo.

FIG. 12 is a block diagram of a system configuration example used for three-dimensional image measurement.

FIG. 13 is an explanatory diagram of a process of converting an image input into a distance image.

FIG. 14 is a diagram illustrating the principle of the space-coded pattern light projection method.

FIG. 15 is an explanatory diagram of gray code pattern light.

FIG. 16 is an explanatory diagram of a proposed example of a three-dimensional image measurement method of a load.

FIG. 17 is an explanatory diagram of each processing state when the cargo has distortion.

[Explanation of symbols]

1: Pallet, 2: Projector, 3: CCD camera, A to C
... light pattern, G ... center of gravity, L ... horizontal pixel row, Pc ...
Pixels, r0 to rn: wedge-shaped measurement area, W: cargo, W _{A to} W
_C ... groups of pixels, C ₁₁ ~C ₄₃ ... conversion point, Z _G ... average value, theta ... slope

──────────────────────────────────────────────────続 き Continuation of the front page (58) Field surveyed (Int. Cl. ⁷ , DB name) G01B 11/00-11/30 102 G06T 7/00

Claims (2)

(57) [Claims] 1. A load measuring space is divided into wedge-shaped measuring regions stacked on each other, a spatial code is determined in each measuring region, and a spatial code image including the spatial code in each pixel data as a load image. The process of obtaining, and for each pixel row in a predetermined direction in the spatial code image, extracting the image of the uppermost cargo, leaving only the pixels including the spatial code within a specified value from the spatial code indicating the wedge-shaped region at the maximum angle. And three-dimensional image measurement method of the cargo including the steps. 2. The method of claim 1, wherein an image of the uppermost cargo is extracted, and three-dimensional coordinates of a plurality of points on each side of the upper surface of the uppermost cargo are calculated from the data of the image. A method for measuring a three-dimensional image of a load, which determines the position of the center of gravity of the uppermost load on the horizontal projection plane from the three-dimensional coordinates.