JP7748864B2 - Image processing device and picking system - Google Patents

Description

The present invention relates to an image processing device and a picking system using the image processing device.

Traditionally, picking robots have been used in many production and manufacturing sites to pick up work objects, transported items, etc. (also referred to as workpieces; hereafter referred to as "workpieces"). In recent years, random picking, which picks bulk workpieces, has become more common. In random picking, bulk workpieces are photographed with a camera before being picked up, and the robot is allowed to recognize the position and orientation of the workpieces from the photographed image (see, for example, Patent Document 1, etc.).

Japanese Patent Application Laid-Open No. 2020-146823

As mentioned above, picking up workpieces, including random picking, requires capturing images of the workpieces. However, at worksites where picking robots operate, the brightness of the worksite can fluctuate due to scattered or reflected light from nearby processes (production lines) and ambient light such as sunlight streaming in through exterior windows. This brightness fluctuation also causes the brightness between images required for picking to fluctuate, reducing the accuracy of workpiece recognition within the images. To mitigate the effects of this ambient light, it is possible to use imaging devices that are resistant to brightness fluctuations or lighting devices that cancel out brightness fluctuations. However, introducing high-performance equipment is costly, and because ambient light differs from worksite to worksite, a system built to suit one worksite cannot be applied to other worksites.

The present invention was developed in consideration of the above problems, and aims to provide a cost-effective and highly versatile image processing device, etc.

According to one aspect of the present invention, an image processing device used to detect a detection target placed under a light source whose luminance fluctuates at a predetermined cycle includes an image acquisition unit that acquires first image information obtained by capturing an image of the detection target under the light source at a first luminance and second image information obtained by capturing an image of the detection target at a second luminance different from the first luminance; a generation unit that generates differential image information representing the difference in luminance between the first image information and the second image information; and an output unit that outputs detection information indicating the detection result of the detection target based on the differential image information.

In an image processing device according to one aspect of the present invention, the image acquisition unit acquires first image information and second image information from an imaging device that captures images at predetermined time intervals, and the generation unit may generate differential image information in which, of the multiple images captured by the imaging device, information relating to the image with the highest brightness is used as the first image information and information relating to the image with the lowest brightness is used as the second image information.

In an image processing device according to one aspect of the present invention, when f1 is the light source frequency, which is the reciprocal of the time between luminance peaks in the light source, and f2 is the imaging frequency, which is the reciprocal of a predetermined time interval in the imaging device, the ratio f1:f2 may be 1:0.5 to 1:4 (excluding 1:1).

In an image processing device according to one aspect of the present invention, if the difference between the first luminance and the second luminance falls below a predetermined threshold, at least one of the light source frequency and the imaging frequency may be changed.

In an image processing device according to one aspect of the present invention, if the difference between the first luminance and the second luminance falls below a predetermined threshold, the timing at which the imaging device captures an image of the detection target may be changed.

In one aspect of the present invention, a picking system including an image processing device and a picking device includes a picking device having a placement unit for a detection target, a picking unit that picks the detection target from the placement unit, and a picking control unit that controls the picking unit based on detection information.

An image processing device according to one aspect of the present invention is an image processing device used to detect a detection target placed under a light source that generates flicker, where the luminance fluctuates at a predetermined cycle. It is characterized by including an image acquisition unit that acquires first image information obtained by capturing an image of the detection target under the light source at a first luminance and second image information obtained by capturing an image of the detection target at a second luminance different from the first luminance; a generation unit that generates differential image information of the difference in luminance between the first image information and the second image information; and an output unit that outputs detection information indicating the detection result of the detection target based on the differential image information. This makes it possible to provide an image processing device, etc., that is cost-effective and highly versatile.

1 is a schematic configuration diagram of a picking system according to one embodiment of the present invention. 1 is a functional diagram of an image processing device according to an aspect of the present invention; 1A and 1B are diagrams illustrating an overview of an image processing device according to an aspect of the present invention. 1A to 1C are diagrams illustrating an overview of an image processing device according to an aspect of the present invention. 1A to 1C are diagrams illustrating an overview of an image processing device according to an aspect of the present invention.

An image processing device according to one embodiment of the present invention will be described in detail below with reference to the drawings. Note that the drawings are for illustrative purposes only, and the image processing device of the present invention is not limited to the one shown. Furthermore, the drawings are schematic, and the size ratios, number, positional relationships, graphs, etc. of the components of the picking system in the drawings are not strictly accurate.

<Picking system>
FIG. 1 is a schematic diagram of a picking system according to one embodiment of the present invention. The picking system 100 includes at least an image processing device 200 and a picking device 30. The picking device 30 includes a picking arm (picking unit) 31 and a placement unit 40, and picks up and grips a workpiece 50 placed on the placement unit 40, or moves the workpiece 50 to another process. While FIG. 1 shows the workpiece 50 as a single unit, the workpiece 50 may be stacked in bulk. Furthermore, the workpiece 50 may be any object that can be picked up by the picking arm 31. Furthermore, the placement unit 40 is not limited to a belt conveyor-like structure as shown in FIG. 1, but may be a box-like structure on which the workpieces 50 are stacked in bulk.

The image processing device 200 acquires images of the workpiece 50 to be detected, captured by the imaging device 20, from the imaging device 20 and performs image recognition processing. The imaging device 20 may be a known camera such as a CCD camera, RGB camera, or stereo camera. The image processing device 200 is also connected to the picking device 30, recognizes the state (orientation, shape, etc.) of the workpiece 50 from the images captured by the imaging device 20, and transmits information regarding the state of the workpiece 50 to the picking device 30. Note that while the image processing device 200 and the picking device 30 are shown separately in Figure 1, the image processing device 200 may also be incorporated into the picking device 30.

In the picking system 100, as shown in FIG. 1, the brightness of the image captured by the imaging device 20 can change due to ambient light 70, including sunlight 60 shining through a window at the work site and light reflected from other processes. Therefore, a picking system according to one aspect of the present invention performs image processing that utilizes the properties of the light source 10 at the work site. As will be described in detail later, the light source 10 at the work site may be a light source whose brightness fluctuates at a predetermined cycle. For example, the light source 10 may be a light source that generates flicker, such as a conventional fluorescent lamp, mercury lamp, incandescent lamp, or LED. In other words, the light source 10 may not have a so-called flickerless function. Note that flicker is caused by frequency fluctuations in the incoming voltage, and is a phenomenon in which light and dark alternate at a cycle of 60 Hz in western Japan and 50 Hz in eastern Japan.

Note that from here on, we will use a light source that flickers as an example of a light source whose brightness fluctuates at a predetermined cycle, but the present invention is not limited to this, and any lighting can be used as long as the speed at which the brightness fluctuates (i.e., the speed at which it changes brightness) can be adjusted.

<Image processing device>
Next, the functional configuration and operation of the image processing device 200 will be described with reference to Figures 2 and 3. As shown in Figure 2, the image processing device 200 includes at least an image acquisition unit 210, an image processing unit 220, and a detection result output unit 230. The image acquisition unit 210 acquires an image captured by the imaging device 20. At this time, the image acquisition unit 210 acquires first image information of the workpiece to be detected captured when the light source 10 is at a first luminance, and second image information of the workpiece to be detected captured when the light source 10 is at a second luminance different from the first luminance.

Here, using FIG. 3, the operation of the image processing device 200 according to one embodiment of the present invention will be described. FIG. 3(a) is a graph showing changes in luminance under the influence of ambient light 70 at the work site of the picking system 100 shown in FIG. 1. In FIG. 3(a), the horizontal axis represents time, the vertical axis represents luminance, and the dashed line 11 represents the change in luminance of the light source 10 over time, while the dashed line 71 represents the change in luminance of the ambient light 70 over time. As indicated by the dashed line 11 in FIG. 3(a), the light source 10 exhibits flicker, in which the luminance fluctuates at a predetermined cycle. In the example shown in FIG. 3(a), the light source 10 alternates between bright and dark at a cycle of T1 [s] (i.e., light source frequency f1 = 1/T1 [Hz]). In contrast, as indicated by the dashed line 71, the ambient light 70 exhibits a more gradual change in luminance over time than the light source 10. The light source 10 and ambient light 70 are superimposed, and the brightness fluctuations at the work site of the picking system 100 are shown in the graph indicated by the solid line 81.

The imaging device 20 captures images of the workpiece 50 under the luminance fluctuations shown by the solid line 81, so images will be captured at different luminances depending on the timing. For example, as shown in Figure 3(a), the luminance of an image captured at time t1 is L1 (first luminance), and the luminance of an image captured at time t0 is L0 (second luminance), allowing images to be captured at different luminances.

The generation unit 221 generates a difference image by subtracting the difference in brightness between the first image and the second image. Here, the properties of the difference image generated from images captured under the first brightness and the second brightness will be explained using Figure 3(b).

In FIG. 3(b), the first image 21 is a high-brightness image captured at a first brightness (L1), which is a relatively high brightness, in FIG. 3(a), and the second image 22 is a low-brightness image captured at a second brightness (L0), which is a relatively low brightness. As shown in FIG. 3(b), noise 72 occurs in the first image 21 and the second image 22 due to the influence of disturbance light 70. Here, because the brightness of the disturbance light 70 varies slowly over time as shown in FIG. 3(a), there is little change in the noise 72 between the first image 21 and the second image 22. Therefore, the difference image 23 of the first image 21 and the second image 22 generated by the generation unit 221 can be an image that does not include noise 72 components.

The image processing unit 220 uses known image recognition technology on the difference image 23 to detect the workpiece 50 contained in the difference image 23. Note that image recognition may be achieved by machine learning or AI (artificial intelligence).

The detection result output unit 230 transmits the detection results from the image processing unit 220 to the pickup device 30. The detection results may include information about the state of the workpiece 50, such as the position and orientation of the workpiece 50. A control unit (not shown) of the pickup device 30 may output a control signal to the picking arm 31 to pick up the workpiece 50, based on the information about the state of the workpiece 50 included in the detection results.

In this way, with an image processing device according to one aspect of the present invention, it is possible to eliminate the effects of ambient light by utilizing fluctuations in the brightness of the light source. Therefore, a low-cost picking system can be realized without requiring high functionality from the light source or imaging device.

The imaging device 20 may be an imaging device that captures images at predetermined time intervals, and the image acquisition unit 210 may acquire multiple images from the imaging device 20. For example, in the example of FIG. 3, the imaging device 20 begins capturing images from time t1 at a period T2 [s] (i.e., imaging frequency f2 = 1/T2 [Hz]), and multiple images are captured at the brightness indicated by the "circles" on the periodically changing curve in the figure. In this case, the generation unit 221 may generate a difference image by using the image with the highest brightness among the multiple images captured by the imaging device 20 as the first image (in the example of the figure, the image at time t1 and brightness L1) and the image with the lowest brightness as the second image (in the example of the figure, the image at time t0 and brightness L0).

In this way, according to the image processing device 200 of one aspect of the present invention, an image with a large difference in brightness may be selected from among the multiple images acquired by the image acquisition unit 210 to generate a difference image. Therefore, the brightness level of the difference image can be increased, thereby improving the detection accuracy of the detection target.

In the above description, an example has been described in which a differential image is generated regardless of the order in which images are captured by the imaging device 20. However, there may be cases in which the state of the detection target changes over time, such as when the workpiece 50 moves on a conveyor belt. In this case, a differential image may be generated between captured images that are adjacent on the time axis.

Figure 4(b) shows a graph of the luminance fluctuation of the difference image 23 obtained when the imaging device 20 captures images at a predetermined time interval and takes the difference between adjacent images on the time axis. Note that Figure 4(a) is the same as Figure 3(a), but for simplicity, the dashed line 11 indicating the luminance fluctuation of the light source 10 is omitted, and only the dashed-dotted line 71 indicating the luminance fluctuation of the ambient light 70 and the solid line 81 indicating the luminance fluctuation at the location where the picking system 100 operates using the light source 10 and ambient light 70 are shown.

When the imaging device 20 captures images at a period T2 (T2 = T1/2 in the example shown in Figure 4(a)), the image acquisition unit 210 acquires multiple first luminance images, which are images captured at a first luminance indicated by a "●" in the figure and have a relatively high luminance, and multiple second luminance images, which are images captured at a second luminance indicated by a "■" in the figure and have a relatively low luminance. The generation unit 221 may generate a difference image from the first luminance image and the second luminance image that are adjacent on the time axis. In this case, image frames F11, F12... of the same luminance are obtained, as shown in Figure 4(b). The image processing unit 220 simply performs recognition processing on the images of each image frame F11, F12...

However, compared to the original image (especially high-brightness images), each image frame F11, F12, etc. in Figure 4(b) emphasizes shot noise and thermal noise, resulting in a poor S/N ratio (signal-to-noise ratio). Note that the S/N ratio here refers to the ratio of noise to the signal representing the detection target. Therefore, as shown in Figure 4(c), the image processing unit 220 may perform brightness averaging between each image frame (between three frames F21 or F22 in the illustrated example). This reduces shot noise and thermal noise and improves the S/N ratio. Note that performing averaging between image frames reduces the number of image frames, which may result in reduced detection accuracy when the detection target is moving, for example. Therefore, the number of image frames to be averaged can be set according to the moving speed of the detection target, etc.

Note that the example in FIG. 4 describes an ideal case in which the difference between the first luminance and the second luminance is greatest when the imaging device 20 captures images at an imaging cycle T2 (=T1/2). However, as shown in FIG. 5(a), depending on the imaging timing, i.e., the relative time lag between the light source cycle T1 of the light source 10 and the imaging cycle T2 of the imaging device 20, the luminance difference between images captured by the imaging device 20 may be very small. In the example in FIG. 5(a), the luminance of images captured by the imaging device 20 at imaging cycle T2 and times t3, t4, t5, and t6 are L3, L4, L5, and L6, respectively, indicating that the difference between the captured image frames is small. In such a case, the luminance of the difference image generated by the generation unit 221 becomes very small, which may make it difficult to detect the target object.

Therefore, according to the image processing device 200 according to one aspect of the present invention, when the light source frequency, which is the reciprocal of the time between luminance peaks in the light source 10, is f1 (=1/T1), and the imaging frequency, which is the reciprocal of the predetermined time interval (T2) in the imaging device 20, is f2 (=1/T2),
f1:f2 = 1:0.5 to 1:4 (excluding 1:1) ... (Equation 1)
The imaging period or the light source period may be set so as to satisfy the above.

For example, Figure 5(b) shows a case where the shooting frequency f2' (= 1/T2') is four times the light source frequency f1. In this case, the luminance of images captured at times t7, t8, t9, and t10 will be L7, L8, L9, and L10, respectively, and a certain degree of luminance difference can be created between some of the captured image frames. Therefore, in the example of Figure 5(b), the image captured at luminance L8 is designated as the first luminance image, and the image captured at luminance L9 is designated as the second image, and a difference image can be generated.

Note that while equation (1) shows the preferred relationship between the imaging frequency and the light source frequency, Figure 5(c) shows the preferred magnification of the imaging frequency to the light source frequency. If the imaging frequency is 0.5 or 1 times the light source frequency, this is not preferred because, as mentioned above, the difference in luminance between the first luminance image and the second luminance image will become small. Therefore, excluding 0.5 and 1 times, if the imaging frequency is 4 times the light source frequency, a detectable difference image can be generated. Of course, it is also possible for the imaging frequency to exceed 4 times the light source frequency, but this would require high functionality from the imaging device 20, so the choice should be made based on a balance with cost.

Note that, as shown in Figure 5(a), when the difference in luminance between captured images is small due to the relationship between the imaging cycle and the light source cycle, i.e., when the difference between the first luminance and the second luminance is below a predetermined threshold, at least one of the light source frequency f1 and the imaging frequency f2 may be changed. For example, by increasing the imaging frequency f2 from the state shown in Figure 5(a), it is possible to transition to the state shown in Figure 5(b). Note that the light source frequency f1 of the light source 10 may be changed using an inverter device or the like.

In the above, we have described a mode in which either the light source frequency f1 or the imaging frequency f2 is changed. However, if the difference between the first luminance and the second luminance falls below a predetermined threshold, the timing at which the imaging device 20 captures an image of the detection target may be changed. For example, from the state shown in FIG. 5(a), imaging by the imaging device 20 may be temporarily stopped, and then imaging may be started again after shifting the imaging timing.

These methods can increase the brightness of the difference image and improve detection accuracy.

Note that the picking system 100, which includes the image processing device 200 according to one aspect of the present invention described above and the picking device 30, may utilize the flicker of the conventionally used light source 10 and set the imaging frequency of the imaging device 20 according to the frequency of the flicker. This makes it possible to realize a low-cost, highly versatile picking system without the need for special lighting equipment or high-performance imaging devices. Furthermore, the light source frequency of the light source 10 may be adjusted relative to the imaging frequency of the imaging device 20. In this case, there is no need to prepare a new imaging device 20 to build the picking system 100, thereby reducing costs.

While the present invention has been described based on various drawings and examples, it should be noted that those skilled in the art would easily be able to make various modifications and alterations based on this disclosure. Therefore, it should be noted that these modifications and alterations are included within the scope of the present invention.

10 Light source 20 Captured image 30 Pickup device 31 Pickup arm 40 Placement unit 50 Workpiece (detection target)
60 Sun 70 Ambient light 100 Picking system 200 Image processing device 210 Image acquisition unit 220 Image processing unit 221 Generation unit 230 Detection result output unit

Claims (4)

An image processing device used to detect a detection target placed under a light source whose luminance fluctuates at a predetermined cycle,
an image acquisition unit that acquires first image information obtained by capturing an image of the detection target under the light source at a first luminance and second image information obtained by capturing an image of the detection target under a second luminance different from the first luminance;
a generating unit that generates differential image information representing a difference in luminance between the first image information and the second image information;
an output unit that outputs detection information indicating a detection result of the detection object based on the differential image information;
Equipped with
the image acquisition unit acquires the first image information and the second image information from an imaging device that captures images at predetermined time intervals;
the generation unit generates difference image information in which information on an image with the highest luminance among a plurality of images captured by the imaging device is the first image information and information on an image with the lowest luminance is the second image information,
When a light source frequency that is the reciprocal of the time between luminance peaks in the light source is f1, and an imaging frequency that is the reciprocal of the predetermined time interval in the imaging device is f2,
f1:f2 = 1:0.5 to 1:4 (excluding 1:1)
Meet the image processing device. When the difference between the first luminance and the second luminance is less than a predetermined threshold, at least one of the light source frequency and the imaging frequency is changed.
2. The image processing device according to claim 1 , wherein: When the difference between the first luminance and the second luminance is below a predetermined threshold, the imaging device changes a timing at which it captures an image of the detection target.
2. The image processing device according to claim 1 , wherein: A picking system comprising the image processing device according to any one of claims 1 to 3 and a picking device,
The picking device is
a placement portion for the detection target;
a picking unit that picks the detection target from the placement unit;
a picking control unit that controls the picking unit based on the detection information;
A picking system comprising: