JP7620441B2 - Workpiece detection device, workpiece detection method, workpiece detection system, and workpiece detection program - Google Patents

Description

The present invention relates to a workpiece detection device, a workpiece detection method, a workpiece detection system, and a workpiece detection program.

Conventionally, multiple workpieces such as plate-shaped members loaded in a loading area are fed by a loading device such as a supply robot to a processing machine (bending machine) such as a press brake that performs bending processes. When loading, a workpiece flipping device (one-piece workpiece pick-up device) is used, such as a magnet floater that uses magnetic force to float the workpiece and separate it from other workpieces, or an air separator that separates the workpiece from other workpieces using air injection pressure.

Typically, a workpiece turning device such as an air separator is fixed to a mounting stand that is installed in a position adjacent to the loading area. For this reason, multiple workpieces loaded onto the loading area are placed in one location, and one workpiece turning device is arranged to accommodate these workpieces.

On the other hand, for example, if the loading area is large, it is expected that multiple workpieces of different types will be placed in multiple locations within the loading area. In such a case, if multiple workpiece turning devices corresponding to each workpiece are placed within the loading area, multiple workpieces and workpiece turning devices will be mixed within the loading area. Note that a workpiece detection device that detects the position of a workpiece placed within the loading area (see, for example, Patent Document 1) uses a monocular camera to capture an image of the workpiece and detect its position.

JP 2018-120388 A

However, in a loading area where multiple workpieces and workpiece turning devices are mixed as described above, part of the workpiece turning device may be positioned on the workpiece. In such cases, part of the workpiece will be hidden in the imaging range of the monocular camera, making it difficult to detect the position of the workpiece accurately and with high precision.

The present invention has been made in consideration of the above circumstances, and aims to provide a work detection device, a work detection method, a work detection system, and a work detection program that can accurately and highly precisely identify the position of a work and detect the work even if part of the work is hidden in the loading area.

The workpiece detection device according to the present invention is a workpiece detection device that detects a workpiece in a loading area, in which a part of a portable auxiliary device that can be freely placed and assists in work on the workpiece is placed on the workpiece, based on an image captured by an imaging device. The workpiece detection device includes an apparatus detection unit that detects the auxiliary device shown in the captured image so that the actual area of the auxiliary device in the loading area can be excluded from the captured image, and a search range is set to the captured image from which the actual area is excluded, and detects an image of the workpiece shown in this search range and a shape that represents the two-dimensional shape of the workpiece stored in advance. The system includes a first detection unit that performs a first match with a plurality of workpiece models consisting of shape data to detect the position of the workpiece in the loading area, a data creation unit that creates new shape data by editing lines that represent contour components of a portion of the actual area that overlaps with an area that overlaps the workpiece, for the shape data of the workpiece model that matches the image of the workpiece whose position has been determined in the first match, and a second detection unit that performs a second match with the workpiece model indicated by the new shape data and the image of the workpiece to identify the actual loading position of the workpiece on the loading area.

In one embodiment of the present invention, the data creation unit performs editing to invalidate the line segments that represent the contour components of the shape data by dividing or erasing them.

In another embodiment of the present invention, the second detection unit performs the second matching by narrowing the search range more than the first matching performed by the first detection unit.

In yet another embodiment of the present invention, the data creation unit and the second detection unit repeatedly create the new shape data and perform the second matching multiple times while gradually narrowing down the search range.

In yet another embodiment of the present invention, the second detection unit identifies a three-dimensional position including the height, size, and rotation angle of the workpiece on the loading area as the actual loading position.

In yet another embodiment of the present invention, the device detection unit detects an indicator that is provided at least on the top surface of the device body of the auxiliary device and is capable of identifying the position and orientation of the auxiliary device, and matches the detected indicator with an indicator indicated by a pre-stored indicator model image to detect the placement position and orientation of the auxiliary device within the loading area, and calculates the actual area.

The work detection method according to the present invention is a method for detecting a work in a loading area, in which a part of a portable auxiliary device that can be freely positioned and that assists in work on the work is placed on the work, based on an image captured by an imaging device, the method including the steps of: detecting the auxiliary device shown in the captured image such that the actual area of the auxiliary device in the loading area can be excluded from the captured image; setting the captured image from which the actual area has been excluded as a search range; and detecting an image of the work shown in this search range and a two-dimensional shape of the work stored in advance. The method includes a step of detecting the position of the workpiece in the loading area by performing a first match with a plurality of workpiece models consisting of shape data representing the workpiece, a step of creating new shape data by editing the line segments representing the contour components of the portion of the actual area that overlaps the area that covers the workpiece for the shape data of the workpiece model that matches the image of the workpiece whose position has been determined in the first match, and a step of performing a second match between the workpiece model represented by the new shape data and the image of the workpiece to identify the actual loading position of the workpiece on the loading area.

In one embodiment of the present invention, the process of creating the shape data involves editing the shape data by dividing or erasing the line segments that represent the contour components.

In another embodiment of the present invention, in the step of identifying the actual loading position, the second matching is performed by narrowing the search range more than the first matching in the step of detecting the location.

In yet another embodiment of the present invention, in the process of creating the shape data and the process of identifying the actual loading position, the process of creating the new two-dimensional shape data and performing the second matching is repeated multiple times in succession while gradually narrowing down the search range.

In yet another embodiment of the present invention, in the step of identifying the actual loading position, a three-dimensional position including the height, size, and rotation angle of the workpiece on the loading area is identified as the actual loading position.

In yet another embodiment of the present invention, the step of detecting the auxiliary device includes detecting an indicator that is provided at least on the top surface of the device body of the auxiliary device and is capable of identifying the position and orientation of the auxiliary device, matching the detected indicator with an indicator indicated by a pre-stored indicator model image to detect the placement position and orientation of the auxiliary device within the loading area, and calculating the actual area.

The work detection system according to the present invention comprises an auxiliary device with portability that can be freely positioned to assist in work on a work loaded in a loading area, an imaging device capable of imaging the loading area, and a work detection device that detects the work in the loading area in which a part of the auxiliary device is placed on the work based on an image captured by the imaging device. The work detection device comprises a device detection unit that detects the auxiliary device shown in the captured image so that the actual area of the auxiliary device in the loading area can be excluded from the captured image, and a search range for the captured image from which the actual area has been excluded, and a work detection unit that detects the work in the search range and a work detection unit that detects the work in the search range from a device detection unit that detects the work in the loading area so that the actual area of the auxiliary device in the loading area can be excluded from the captured image. The system has a first detection unit that detects the position of the workpiece in the loading area by performing a first match with a plurality of workpiece models consisting of shape data representing the stored two-dimensional shape of the workpiece, a data creation unit that creates new shape data by editing lines representing contour components of a portion of the actual area that overlaps with an area that overlaps the workpiece, for the shape data of the workpiece model that matches the image of the workpiece whose position has been determined in the first match, and a second detection unit that identifies the actual loading position of the workpiece on the loading area by performing a second match between the workpiece model indicated by the new shape data and the image of the workpiece.

In one embodiment of the present invention, the system further includes a control device that controls the operation of the auxiliary device with respect to the detected workpiece based on the detection result from the workpiece detection device.

The work detection program of the present invention is a program for detecting a work in a loading area, in which a part of an auxiliary device having a freely positionable portability that assists in work on the work is placed on the work, based on an image captured by an imaging device, and includes a step of causing a computer to detect the auxiliary device shown in the captured image so that the actual area of the auxiliary device in the loading area can be excluded from the captured image, a step of causing a computer to detect the auxiliary device shown in the captured image so that the actual area of the auxiliary device in the loading area can be excluded from the captured image, and ... The method executes the steps of: performing a first match with a plurality of workpiece models consisting of shape data representing two-dimensional shapes to detect the position of the workpiece within the loading area; editing lines representing contour components of a portion of the actual area that overlaps with an area that overlaps the workpiece for the shape data of the workpiece models that match the image of the workpiece whose position has been determined in the first match to create new shape data; and performing a second match between the workpiece model indicated by the new shape data and the image of the workpiece to identify the actual loading position of the workpiece on the loading area.

According to the present invention, the position of the workpiece can be identified accurately and with high precision, and the workpiece can be detected even if part of the workpiece is hidden in the loading area.

1 is a perspective view showing an outline of a configuration example of a workpiece detection system according to an embodiment of the present invention; 2 is a block diagram illustrating a schematic functional configuration of a workpiece detection device in the workpiece detection system. FIG. FIG. 2 is a configuration diagram illustrating a hardware configuration of the workpiece detection device. 4 is a flowchart showing an example of a workpiece detection process in the workpiece detection system. 13 is a flowchart showing an example of a device detection process in the work detection process. 3 is an image diagram conceptually showing an image captured by an imaging device of the workpiece detection system. FIG. FIG. 2 is a perspective view showing an auxiliary device in the workpiece detection system. 11 is a diagram showing an example of an indicator provided on the assisting device. FIG. FIG. 13 is a diagram showing another example of the index. 13 is an image diagram conceptually showing an actual area of the auxiliary device in the captured image. FIG. FIG. 11 is an image diagram for conceptually explaining a first matching using the captured image. 13 is an image diagram for conceptually explaining an image of a workpiece in the captured image and a workpiece model matched within a search range. FIG. 13 is an image diagram for conceptually explaining data creation and second matching using the captured image. FIG.

Below, a workpiece detection device, a workpiece detection method, a workpiece detection system, and a workpiece detection program according to embodiments of the present invention will be described in detail with reference to the attached drawings. However, the following embodiments do not limit the invention according to each claim, and not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention. Note that in the present embodiments, the scale and dimensions of each component may be exaggerated, and some components may be omitted.

Figure 1 is a perspective view that shows a schematic configuration example of a workpiece detection system 100 according to one embodiment of the present invention. As shown in Figure 1, the workpiece detection system 100 according to this embodiment includes a workpiece turning device 10 as an auxiliary device that assists in operations on the workpieces W loaded in a loading area TA on a pallet P (e.g., loading into a bending machine, etc.), and a camera 20 as an imaging device that can capture an image of the loading area TA.

The workpiece detection system 100 also has the functionality of an NC device that controls the entire workpiece detection system 100, as well as the functionality of an image processing device that performs general image processing such as image analysis and image correction based on the image captured by the camera 20, and is equipped with a workpiece detection device 30 that identifies the actual loading position (actual loading position) of the workpiece W on the loading area TA and detects the workpiece.

In this embodiment, the workpiece detection device 30 is configured to include a display 69 as a display device and an input unit 35a consisting of input devices such as a keyboard and a mouse, but is not limited to this as long as it can be replaced by something that has equivalent functions in place of the display 69 and the input unit 35a (for example, a display means or input means that can be used remotely, etc.).

In addition, in this embodiment, for example, processing machines such as bending machines and loading devices such as workpiece holding robots are not shown, but these processing machines and loading devices may be included in the workpiece detection system 100 in a state where various operations can be controlled by the workpiece detection device 30.

In addition, in this embodiment, the work turning device 10 is described as an example of an auxiliary device, but the auxiliary device is not limited to this as long as it can assist in various tasks on the work W. Furthermore, although not shown in the figures, each work turning device 10, camera 20, and work detection device 30 are connected to each other by wire or wirelessly so that they can send and receive data and signals.

The work turning device 10 is portable and can be freely positioned at multiple locations inside and outside the loading area TA on the pallet P so as to correspond to each of the multiple workpieces W placed within the loading area TA. In this embodiment, multiple work turning devices 10 can be placed in the loading area TA so that a portion of each workpiece W is placed on the workpiece, for example, in one-to-one correspondence. The detailed configuration of this work turning device 10 will be described later.

The camera 20 is, for example, an inexpensive and versatile monocular camera (i.e., one camera), and is positioned directly above the center of the loading area TA of the pallet P via a support member such as a camera stand 21 so that the entire loading area TA can be captured within the imaging range. Note that various calibrations of the captured image obtained by the camera 20 for capturing the loading area TA on the pallet P are assumed to have been performed in advance.

Although not shown, lighting equipment having a number of light-emitting diodes (LEDs) or the like is provided at predetermined locations around the pallet P, which irradiates light onto the workpieces W loaded in the loading area TA. When the loading area TA is imaged by the camera 20, the lighting equipment can be used to irradiate the workpieces W with illumination light, allowing the brightness of the captured image to be appropriately adjusted with proper exposure.

On the other hand, the workpiece detection device 30 is functionally configured as follows.
FIG. 2 is a block diagram that shows generally the functional configuration of the work detection device 30 in the work detection system 100, and FIG.

As shown in FIG. 2, image data of the image captured by the camera 20 is input to the workpiece detection device 30. Functionally, the workpiece detection device 30 includes an image acquisition unit 31, an image processing unit 32, a calculation unit 33, a storage unit 34, and an operation unit 35. The detection of the workpiece W is mainly performed by the image processing unit 32, which has a device detection unit 36, a first detection unit 37, a data creation unit 38, and a second detection unit 39.

Here, the image acquisition unit 31 acquires image data of the captured image output from the camera 20. The image processing unit 32 performs various image conversion and image analysis processes (hereinafter, such processes are collectively referred to as "image processing") on the image data acquired by the image acquisition unit 31, including, for example, binarization processing, morphological processing, approximation processing using the Newton method, and matching processing such as template matching.

The image data (raster data) can be converted into numerical data (vector data) that can be expressed by linearly connected line segments by extracting the contour components of each object captured in the loading area TA, such as the work turner 10 and the work W, as feature values (geometric shape data) through image processing by the image processing unit 32, and each matching described below can be performed between vector data unless otherwise specified in order to reduce the processing load.

The device detection unit 36 of the image processing unit 32 detects the work turner 10 that appears in the captured image represented by the image data by calculating the actual area of the work turner 10 within the loading area TA and detecting the work turner 10 so that it can be excluded from the captured image. Details of the detection of the work turner 10 in the work detection system 100 will be described later.

The first detection unit 37 uses the captured image from which the actual area of the work turning device 10 has been excluded by the device detection unit 36 as a search range, and performs matching (first matching) between the image of the work W captured in this search range and a work model consisting of shape data representing the two-dimensional shapes of various work pieces W pre-stored in the memory unit 34. The first detection unit 37 then detects the location of the work piece W, which represents its approximate position within the loading area TA.

The data creation unit 38 creates new shape data by editing the lines representing the contour components of the portion of the actual area of the workpiece turning device 10 that overlaps with the area that covers the workpiece W (see FIG. 6) of the shape data of the workpiece model that matches the image of the workpiece W whose position has been determined by the first detection unit 37. Specifically, the data creation unit 38 edits the lines representing the contour components of the shape data so as to invalidate them by dividing or erasing them.

Then, the second detection unit 39 detects the workpiece W by performing matching (second matching) between the workpiece model indicated by the new shape data created by the data creation unit 38 and the image of the workpiece W, and identifying the actual loading position, which is the actual loading position of the workpiece W on the loading area TA. Specifically, the second detection unit 39 identifies a three-dimensional position including the height of the workpiece W on the loading area TA, the size of the workpiece W, and the rotation angle of the workpiece W as the actual loading position of the workpiece W.

For example, the matching by the second detection unit 39 may be performed with a narrower search range than the matching by the first detection unit 37 (i.e., the search range is not the entire loading area TA, but an enlarged image of the workpiece W and the workpiece model), in order to improve detection accuracy. In addition, the creation of shape data by the data creation unit 38 and the matching by the second detection unit 39 may be performed multiple times in succession while gradually narrowing the search range, in order to further improve detection accuracy.

The calculation unit 33 performs various calculation processes, such as calculations for controlling the operation of the work turning device 10 corresponding to the detected work W and calculations for controlling the entire work detection system 100, based on the detection results from the image processing unit 32, and outputs control to each unit. The memory unit 34 stores various data used by the work detection device 30 in a readable and writable manner, such as CAD data (shape data) that represents the work W and the work turning device 10 in two or three dimensions, various image data including model (work model) images that serve as references for work W of various shapes, and various program data.

The work model image may be stored in multiple types for each work W, and may be multiple images (e.g., images that appear tilted or distorted, etc.) that appear different when captured by the camera 20 depending on, for example, the placement location, loading height, and rotation of the work W actually loaded on the loading area TA, and each may be linked to the above-mentioned shape data (shape data whose dimensions are changed according to each image). The operation unit 35 also accepts operation inputs by the user of the work detection device 30 via the input unit 35a.

As shown in FIG. 3, the workpiece detection device 30 has, as its hardware configuration, for example, a CPU 61, a RAM 62, a ROM 63, a HDD (hard disk drive) 64, and an SSD (solid state drive) 65. The workpiece detection device 30 also has an input I/F (interface) 66, an output I/F (interface) 67, and a communication I/F (interface) 68. Each of the components 61 to 68 is connected to each other by a bus 60.

The CPU 61 controls the entire work detection system 100, including the work detection device 30, by executing various programs stored in the RAM 62, ROM 63, HDD 64, SSD 65, etc., and also realizes the functions of the image processing unit 32 and the calculation unit 33 by executing the work detection program.

The RAM 62 can be used as a working area for the CPU 61. The ROM 63 stores the various programs described above in a readable manner. The HDD 64 and the SSD 65 store the various data described above in a readable and writable manner, and together with the RAM 62 and the ROM 64, realize the functions of the storage unit 34.

The camera 20 is connected to the input I/F 66 to acquire captured images. Thus, the input I/F 66 realizes the function of the image acquisition unit 31. At the same time, a touch panel 69a that functions as the input unit 35a of the operation unit 35 is connected to the input I/F 66, and receives information associated with operation input from the user. Note that input means such as a keyboard and mouse (not shown) can also be connected to the input I/F 66.

The output I/F 67 is connected to a display 69 of the workpiece detection device 30, which has, for example, a built-in touch panel 69a, and various information is output and displayed on the monitor. The workpiece detection device 30 can be connected to a network such as the Internet or external devices (not shown) via the communication I/F 68.

The workpiece detection process in the workpiece detection system 100 including the workpiece detection device 30 configured in this manner is performed, for example, as follows.
Fig. 4 is a flowchart showing an example of work detection processing in the work detection system 100, and Fig. 5 is a flowchart showing an example of device detection processing of the work turning device 10 in the work detection processing. Fig. 6 is an image diagram conceptually showing an image captured by the camera 20 of the work detection system 100, and Fig. 7 is a perspective view generally showing the work turning device 10 in the work detection system 100.

As shown in FIG. 4, when detecting the workpiece W, first, the camera 20 captures an image of the loading area TA where the workpiece W and the workpiece turning device 10 are loaded and arranged (step S100), and outputs image data of the captured image to the workpiece detection device 30. Next, the workpiece detection device 30 acquires image data of the captured image via the image acquisition unit 31, and performs image analysis on the image data as described above (step S101).

As shown in FIG. 6, the captured image 22 captured by the camera 20 installed above the pallet P shows the work W loaded on the loading area TA, as well as the work turning device 10 having the device main body 10A and the turning unit 10B (which will be described later), with, for example, a part of it (part of the turning unit 10B) overlapping a part of the work W.

In general, even if an attempt is made to detect the workpiece W using the captured image 22 showing the workpiece W and the workpiece turning device 10 in this state, the portion of the turning unit 10B that covers the workpiece W obscures the outline and shape of the workpiece W underneath, making it impossible to recognize the relevant portion, and detection of the workpiece W by matching frequently fails (false detection).

Therefore, the workpiece detection device 30 of this embodiment executes a device detection process (step S102) to detect the workpiece turning device 10 in the loading area TA in advance based on the captured image 22 after the image analysis of step S101. The device detection process of step S102 is performed in the device detection unit 36, for example, as follows.

As shown in FIG. 5, in the device detection process, first, markers 14, 15 (see FIG. 6) provided on the work turning device 10 are detected by edge detection process or the like (step S110). Here, the markers 14, 15, etc. are described. The markers 14, 15, etc. indicate the position and orientation of the work turning device 10 in an identifiable manner.

As a premise, as shown in FIG. 7, the workpiece turning device 10 comprises a device main body 10A having, for example, a rectangular outer shape when viewed from above, and a turning unit 10B that is cantilevered with respect to the device main body 10A so as to be movable relative to the device main body 10A and to be in contact with and able to hold the workpiece W.

The turnover unit 10B has a function of turning over and separating the topmost work W of multiple plate-shaped workpieces W loaded in the loading area TA on the pallet P from the other workpieces W, for example by using air injection pressure and suction force. In this way, the work turning device 10 can be used as a one-piece pick-up device for picking up one workpiece W by the turnover unit 10B, but since the detailed configuration of the one-piece pick-up device is publicly known, a description thereof will be omitted here. In the work turning device 10 of this embodiment, the side where the turnover unit 10B is located is defined as the front side of the device.

The device body 10A has a top surface 11 provided on the upper side of the device, a sloped surface 12 that slopes from the top surface 11 at two locations on the left and right sides of the front side of the device, and a pair of side surfaces 13 that connect the top surface 11 (and sloped surface 12) to the left and right sides of the device. Markers 14, 15, and 16 are provided at predetermined locations on the top surface 11, sloped surface 12, and pair of side surfaces 13 of the device body 10A as indicators that can identify the position and direction of the work turning device 10.

Specifically, markers 14 and 16 are provided in multiple locations on the top surface 11 and pair of side surfaces 13 of device body 10A, and marker 15 is provided in multiple locations on the sloped surface 12 of device body 10A. More specifically, marker 14 is provided at two locations on the left and right sides of top surface 11 of device body 10A near the rear side of the device, and marker 16 is provided at locations on pair of side surfaces 13 of device body 10A near the upper side of the device. In other words, marker 16 is provided to the side and directly below marker 14. Also, marker 15 is provided near both ends of the sloped surface 12 at two locations of device body 10A in the direction of inclination.

Figure 8 shows an example of markers 14-16 provided on the work turning device 10, and Figure 9 shows another example of markers 14-16. The markers 14, 15, and 16 provided on the device body 10A of the work turning device 10 can be configured with AR (augmented reality) markers, including, for example, well-known ArUco markers and ChArUco markers.

As shown in FIG. 8, the marker 14 (15, 16) is composed of a film-like or plate-like member that includes at least an overall area 18 that represents the entire marker, and an identification area 19 for identifying the position and orientation of the marker. Note that the identification areas 19 of the markers 14 to 16 are not shown in FIG. 1, FIG. 6, FIG. 7, etc.

The shape of the identification area 19 of the markers 14 to 16 is, for example, a combination of simple rectangles in the example shown in Figure 8, but as shown in Figures 9(a) to 9(g), it is not limited to the shape shown in the figure, and various shapes such as more complex shapes and simpler shapes can be adopted, such as combining multiple rectangles or squares or changing the position of the identification area 19 within the overall area 18, as long as the shape can at least identify the position and direction of the work turning device 10 by itself.

The dimensions of the markers 14-16 (such as the size and shape of the overall area 18 and the identification area 19) can be set according to the imaging performance of the camera 20, such as the number of pixels and imaging resolution, but as long as they can be detected so that the placement position and orientation of the work turning device 10 can be sufficiently determined from the captured image 22 captured from above the loading area TA, it is sufficient to provide at least one on the top surface 11 of the device main body 10A. In other words, when the work turning device 10 is in the correct position, the top surface 11 is most likely to be captured in the captured image.

The markers 16 provided on a pair of side portions 13 of the device body 10A are provided so that the position and orientation of the work turning device 10 (its position and orientation within the loading area TA when it falls over) can be identified even if the work turning device 10 falls over due to some kind of accident (such as a collision with a robot arm of a loading device not shown).

Returning to FIG. 5 again, once the markers 14, 15 are detected in step S110, position information representing the pixel (picture element) positions of the four corners of the entire area 18 of each marker 14, 15 is calculated and stored in the memory unit 34 (step S111). Note that a pixel (picture element) refers to the smallest unit or element that has color information (hue, gradation, etc.) when the captured image 22 is handled by the workpiece detection device 30.

Next, for each marker 14, 15, position information of the model position indicating the position of the marker 14, 15 actually placed on the work turning device 10 is obtained based on various data such as dimensional information of each part of the work turning device 10 indicated by the CAD data stored in the memory unit 34 and image data representing an image of the marker model (step S112).

Then, the device detection unit 36 repeats the following steps S113 to S116 multiple times (e.g., five times). That is, the acquired model position and the stored pixel position are subjected to an approximation process such as Newton's method to calculate the physical three-dimensional positions (x, y, z positions) and rotations (rz) of the markers 14 and 15 within the loading area TA (step S113).

Next, after the three-dimensional positions and rotations are calculated, a marker image is generated that represents the markers 14 and 15 in a state that reflects the physical three-dimensional positions (x, y, z positions) and rotation (rz) based on the calculation results (step S114). The markers 14 and 15 represented by the generated marker image are then searched for, for example, by performing a template matching process (step S115) with various marker model images that differ according to the three-dimensional positions and rotation states that are pre-stored in, for example, the storage unit 34. Based on the matching information (x, y, z positions and rotation rz) of the markers 14 and 15 shown by the marker model images thus searched, the position information representing the pixel positions of the markers 14 and 15 stored in the storage unit 34 is updated (step S116).

After repeating steps S113 to S116 multiple times, it is determined whether the matching information has sufficiently converged (step S117), and if it is determined that the matching information has not sufficiently converged (No in step S117), the process proceeds to step S113 and the subsequent processes are repeated.

On the other hand, if it is determined that sufficient convergence has occurred (Yes in step S117), the actual area R (see FIG. 10) of the work turner 10 within the loading area TA is calculated based on various data stored in the memory unit 34 so that the work turner 10 can be excluded from the captured image 22, and the work turner 10 is detected (step S118). Note that FIG. 10 conceptually illustrates the actual area R of the work turner 10 in the captured image 22. Once the work turner 10 has been detected in this way, the device detection process according to this flowchart is terminated, and the process proceeds to step S103 in FIG. 4.

That is, in step S103, in order to avoid erroneous detection of the workpiece W, the image processing unit 32 extracts the pixel range of the actual area R of the workpiece turning device 10 from the captured image 22 showing the loading area TA, which is the matching search range, as shown in FIG. 10, and excludes the extracted actual area R from the processing target (step S103).

11, the first detection unit 37 sets the entire captured image 22 excluding the real area R (loading area TA) as a search range SR, and performs a first matching (step S104) between the images of each workpiece W captured in this search range SR and multiple workpiece models Wm consisting of shape data representing the two-dimensional shape of the workpiece W pre-stored in the memory unit 34. The shape data of the workpiece models Wm is vector data defined by points (x, y coordinate points), line segments (lines connecting x, y coordinate points), and surfaces (lines closed by lines connecting x, y coordinate points), and specifies the dimensions and shape of the workpiece W, such as length, width, length, and thickness.

As shown in FIG. 12, this first matching allows calculations to be performed by referring to shape data that is linked in advance in a database (DB) in the memory unit 34 to the image of the workpiece W and the workpiece model Wm matched within the search range SR, thereby determining where in the search range SR the workpiece W is located (whether it fits the workpiece model Wm). This makes it possible to grasp the approximate location of the workpiece W within the loading area TA. Note that in the following explanation, including FIG. 12, we will focus on one workpiece W and workpiece turning device 10 out of the multiple workpieces W and workpiece turning devices 10 in the loading area TA, but it goes without saying that the same applies to the other workpieces W and workpiece turning devices 10.

After the first matching, the following steps S105 to S107 are repeated multiple times (for example, four times). That is, once the position of the workpiece W within the loading area TA has been determined, the data creation unit 38 creates a new shape data workpiece model Wmr by editing the line segments representing the contour components of the portion of the actual area R of the workpiece turning device 10 that overlaps with the area 10BR (see FIG. 13) that covers the workpiece W for the shape data of the workpiece model Wm that matched the image of the workpiece W in the first matching (step S105).

That is, as shown in FIG. 13, the data creation unit 38 reads the shape data of the matched work model Wm from the DB, and edits the line segment L and line segments S1 to S4 that represent the contour components of the part that overlaps with the covered area 10BR of the work turning device 10 by, for example, dividing the line segment L into line segments L1 and L2 and erasing the line segments S1 to S4, thereby creating a work model Wmr of new shape data (deformed data) in which the line segments Ld and Sd are invalidated from the work model Wm. Then, the second detection unit 39 performs a second matching (step S106) between the work model Wmr indicated by the deformation data and the image of the work W, and narrows the search range SR (for example, sets the search range to a search range SSR that is smaller than the search range SR) (step S107).

After steps S105 to S107 are repeated multiple times, it is determined whether matching is complete, for example by the workpiece W and the workpiece model Wmr being sufficiently matched down to the smallest detail (step S108). If it is determined that matching is not complete (No in step S108), the process proceeds to step S105 and the subsequent processes are repeated.

On the other hand, if it is determined that matching is complete (Yes in step S108), the second detection unit 39 can obtain accurate matching information (various information such as the loading position coordinates of the workpiece W, loading height, loading dimensions, loading angle, etc.) in the search range SR through calculations by the image processing unit 32 based on the captured image 22 and various data in the memory unit 34, and therefore identifies the actual loading position, which is the actual loading position of the workpiece W on the loading area TA (step S109), detects the workpiece W, and ends the processing according to this flowchart.

In this way, according to the workpiece detection system 100 of this embodiment, even if part of the workpiece W is hidden from the imaging range of the camera 20 by the workpiece turning device 10 arranged at a separate arbitrary location in the loading area TA, the approximate position of the workpiece W in the loading area TA is grasped by matching. Furthermore, the shape data of the matched workpiece model Wm is edited to take into account the hidden part and then matched with the image of the workpiece, for example, multiple times while narrowing the search range. Then, the actual loading position of the workpiece W on the loading area TA is identified by performing calculations based on the result and various data such as CAD data stored in advance. This makes it possible to accurately and highly precisely identify the loading position of the workpiece W in the loading area TA and detect the workpiece W.

Although an embodiment of the present invention has been described above, this embodiment is presented as an example and is not intended to limit the scope of the invention. This new embodiment can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims.

10 Work turning device (auxiliary device)
10A: Device body 10B: Turning unit (movable part)
11 Top surface portion 12 Slope portion 13 Side surface portion 14, 15, 16 Marker (index)
18 Overall area 19 Identification area 20 Camera (imaging device)
21 Camera stand 22 Captured image 30 Workpiece detection device 100 Workpiece detection system P Pallet R Actual area SR, SSR Search range TA Loading area W Workpiece Wm Workpiece model

Claims (15)

A work detection device detects a work in a loading area based on an image captured by an imaging device, the loading area being a loading area on which a work is loaded and a part of an auxiliary device having a freely positionable portability for assisting a task on the work is placed on the work, the work being detected based on an image captured by an imaging device,
a device detection unit that detects the auxiliary device shown in the captured image such that an actual area of the auxiliary device within the loading area can be excluded from the captured image;
a first detection unit that detects the position of the workpiece within the loading area by performing a first matching between a captured image excluding the actual area as a search range and a plurality of workpiece models consisting of shape data representing a two-dimensional shape of the workpiece that is stored in advance;
a data creation unit that creates new shape data by editing line segments that represent contour components of a portion of the actual area that overlaps with an area that covers the workpiece, for shape data of the workpiece model that has been matched in the first matching to the image of the workpiece whose position has been determined;
a second detection unit that performs a second matching between a work model indicated by the new shape data and an image of the work to identify an actual loading position of the work on the loading area. The workpiece detection device according to claim 1 , wherein the data creation unit performs editing to invalidate the line segments representing the contour components of the shape data by dividing or erasing them. The workpiece detection device according to claim 1 or 2, wherein the second detection unit performs the second matching by narrowing a search range more than the first matching performed by the first detection unit. The workpiece detection device according to claim 1 or 2, wherein the data creation unit and the second detection unit repeatedly create the new shape data and perform the second matching multiple times while gradually narrowing down the search range. The workpiece detection device according to any one of claims 1 to 4, wherein the second detection unit identifies a three-dimensional position including a height, a size, and a rotation angle of the workpiece on the loading area as the actual loading position. The workpiece detection device according to any one of claims 1 to 5, wherein the device detection unit detects an index capable of identifying the position and orientation of the auxiliary device, which is provided at least on a top surface portion of a device body of the auxiliary device, matches the detected index with an index indicated by a pre-stored index model image, detects the placement position and placement orientation of the auxiliary device within the loading area, and calculates the actual area. A method for detecting a workpiece in a loading area, in which a workpiece is loaded and a part of an auxiliary device having a freely positionable portability for assisting a task on the workpiece is placed on the workpiece, based on an image captured by an imaging device, the method comprising:
detecting the auxiliary device shown in the captured image such that an actual area of the auxiliary device within the loading area can be excluded from the captured image;
A process of detecting the position of the workpiece within the loading area by performing a first matching between the captured image excluding the actual area as a search range and a plurality of workpiece models consisting of shape data representing a two-dimensional shape of the workpiece that is stored in advance, and
A process of creating new shape data by editing line segments representing contour components of a portion of the actual area that overlaps with an area that covers the workpiece, for the shape data of the workpiece model that has been matched in the first matching to the image of the workpiece whose position has been determined;
A process of performing a second matching between a work model indicated by the new shape data and an image of the work to identify an actual loading position of the work on the loading area. The workpiece detection method according to claim 7 , wherein in the step of creating the shape data, editing is performed to invalidate by dividing or erasing line segments representing the contour components of the shape data. The workpiece detection method according to claim 7 or 8, wherein in the step of specifying the actual loading position, the second matching is performed by narrowing a search range more than the first matching in the step of detecting the presence position. The workpiece detection method according to claim 7 or 8, wherein in the process of creating the shape data and the process of identifying the actual loading position, creating the new shape data and performing the second matching are repeated multiple times in succession while gradually narrowing down the search range. A work detection method according to any one of claims 7 to 10, wherein in the step of identifying the actual loading position, a three-dimensional position including the height, size and rotation angle of the work on the loading area is identified as the actual loading position. The method for detecting a workpiece according to any one of claims 7 to 11, wherein in the step of detecting the auxiliary device, an index capable of identifying a position and orientation of the auxiliary device, which is provided at least on a top surface portion of a main body of the auxiliary device, is detected, the detected index is matched with an index indicated by a pre-stored index model image, the position and orientation of the auxiliary device in the loading area are detected, and the actual area is calculated. a portable auxiliary device that can be freely positioned to assist in work on the workpieces loaded in the loading area;
An imaging device capable of imaging the loading area;
A work detection device that detects the work in the loading area in which a part of the auxiliary device is placed on the work based on an image captured by the imaging device,
The workpiece detection device is
a device detection unit that detects the auxiliary device shown in the captured image such that an actual area of the auxiliary device within the loading area can be excluded from the captured image;
a first detection unit that detects the position of the workpiece within the loading area by performing a first matching between the captured image excluding the actual area as a search range and a plurality of workpiece models consisting of shape data representing a two-dimensional shape of the workpiece that is stored in advance;
a data creation unit that creates new shape data by editing line segments that represent contour components of a portion of the actual area that overlaps with an area that covers the workpiece, for shape data of the workpiece model that has been matched in the first matching to the image of the workpiece whose position has been determined;
A workpiece detection system comprising: a second detection unit that performs a second matching between a workpiece model indicated by the new shape data and an image of the workpiece to identify an actual loading position of the workpiece on the loading area. The workpiece detection system according to claim 13 , further comprising a control device that controls an operation of the auxiliary device with respect to the detected workpiece based on a detection result from the workpiece detection device. A workpiece detection program for detecting a workpiece in a loading area on the basis of an image captured by an imaging device, the loading area being a loading area on which a workpiece is loaded and a part of an auxiliary device having a freely positionable portability for assisting in a task performed on the workpiece is placed on the workpiece, the workpiece being detected based on an image captured by an imaging device,
On the computer,
detecting the auxiliary device shown in the captured image such that an actual area of the auxiliary device within the loading area can be excluded from the captured image;
A process of detecting the position of the workpiece within the loading area by performing a first matching between the captured image excluding the actual area as a search range and a plurality of workpiece models consisting of shape data representing a two-dimensional shape of the workpiece that is stored in advance, and
A process of creating new shape data by editing line segments representing contour components of a portion of the actual area that overlaps with an area that covers the workpiece, for the shape data of the workpiece model that has been matched in the first matching to the image of the workpiece whose position has been determined;
a second matching process between a work model indicated by the new shape data and an image of the work to identify an actual loading position of the work on the loading area.

Images