JP7626640B2 - METHOD FOR DISCLOSURE OF METAL SUBSTANCES, DEVICE FOR DISCLOSURE OF METAL SUBSTANCES, AND METHOD FOR GENERATING TRAINED MODEL - Google Patents

Description

The present invention relates to a method for identifying metal substances, a device for identifying metal substances, and a method for generating a trained model.

There is a known method for determining whether a substance such as a foreign object present in an object to be inspected is a metallic substance (see, for example, Patent Document 1).

JP 2002-62237 A

The inspection method (metallic substance discrimination method) described in Patent Document 1 involves observing the fluorescent X-rays generated when a part (inspection object) is irradiated with X-rays, and qualitatively or quantitatively analyzing the elements contained in the inspection object to determine whether the substance present in the inspection object is metallic or non-metallic, which has the disadvantage of making the process complicated.

The object of the present invention is to provide a metallic substance discrimination method, a metallic substance discrimination device, and a method for generating a trained model that can determine in a simple process whether a foreign object present in an inspection object is metallic or not.

A metallic substance discrimination method according to one aspect of the present invention is a metallic substance discrimination method for discriminating whether or not a foreign substance present in an object to be inspected is a metallic substance, and includes an image acquisition step of acquiring an image of the object to be inspected, a channel image generation step of generating a plurality of channel images by separating and extracting a plurality of channel components from the image to be inspected and converting the image to an image for each of the channel components, and a discrimination step of discriminating whether or not a foreign substance present in the object to be inspected is a metallic substance from the channel images of the object to be inspected, using a trained model that has been deep-learned using the channel images of a metallic substance for training and the channel images of a non-metallic substance for training as training data.

In one aspect of the metal substance discrimination method of the present invention, an imaging step is carried out in which the inspection object is imaged under each of a plurality of lights having different illumination methods, and in the channel image generation step, the plurality of channel images are preferably generated for each image captured under each of the lights.

In the metal material discrimination method according to one aspect of the present invention, it is preferable that one of the multiple illuminations is a coaxial epi-illumination.

A metallic substance discrimination device according to one aspect of the present invention is a metallic substance discrimination device that discriminates whether or not a foreign substance present in an object to be inspected is a metallic substance, and includes an image acquisition means for acquiring an image of the object to be inspected, a channel image generation means for generating a plurality of channel images by separating and extracting a plurality of channel components from the image and converting the image to an image for each of the channel components, and a discrimination means for discriminating whether or not a foreign substance present in the object to be inspected is a metallic substance from the channel images of the object to be inspected, using a trained model that has been deep-learned using the channel images of a metallic substance for training and the channel images of a non-metallic substance for training as teacher data.

A trained model generation method according to one aspect of the present invention is a trained model generation method for determining whether a foreign object present in an inspection object is a metallic substance, and includes a training imaging step of imaging a training metallic substance and a training non-metallic substance, a training channel image generation step of separating and extracting a plurality of channel components from the image captured in the training imaging step and generating a plurality of channel images by converting the image captured in the training imaging step into images for each of the channel components, and a model generation step of performing deep learning on the channel images of the training metallic substance and the channel images of the training non-metallic substance as training data to generate the trained model.

In a method for generating a trained model according to one aspect of the present invention, it is preferable that in the training imaging step, training metallic materials and training non-metallic materials are imaged under each of a plurality of lights having different illumination methods, and in the training channel image generation step, the plurality of channel images are generated for each image captured under each light.

In the method for generating a trained model according to one aspect of the present invention, it is preferable that one of the multiple lights is a coaxial epi-illumination light.

According to one aspect of the present invention, by capturing an image of an object to be inspected, it is possible to determine from the captured image whether a foreign object is a metallic substance, making it possible to determine in a simple process whether a foreign object present in an object to be inspected is a metallic substance without the need for qualitative or quantitative analysis of elements.

FIG. 1 is an explanatory diagram of a metal substance discrimination system according to an embodiment. 1 is an explanatory diagram of a metal substance discrimination method according to an embodiment; An explanatory diagram of a method for generating a trained model according to one embodiment. 1 is an explanatory diagram of a metal substance discrimination method according to an embodiment; FIG. 11 is an explanatory diagram of a metallic substance discrimination system according to a modified example.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
[Device configuration]
In Figure 1, the metal substance discrimination device EA is a device that discriminates whether or not a foreign matter FS (see Figure 2) present in an inspection object WK (see Figure 2) is a metal substance, and is composed of a computer such as a personal computer or a server.
The metallic substance determining apparatus EA comprises a storage means 10, a processing means 20, and an operation means 30, and together with an imaging means 40 and an output means 50 constitute a metallic substance determining system EA1.

The storage means 10 is composed of a memory, a hard disk, etc., and stores various programs that control the metal substance discrimination device EA and the metal substance discrimination system EA1. The storage means 10 also stores a trained model for determining whether or not a metal substance is contained in the inspection target WK.

The processing means 20 is configured with a processor such as a CPU (Central Processing Unit) or a GPU (Central Graphics Processing Unit), and includes an image acquisition means 21, a foreign object extraction means 22, a channel image generation means 23, and a discrimination means 24.
The image acquisition means 21 acquires an image of the inspection object WK from the imaging means 40 .
The foreign matter extraction means 22 detects a foreign matter FS that is not actually present in the inspection object WK from the captured image acquired by the image acquisition means 21, and extracts an image of the foreign matter FS portion from the captured image.
The channel image generating means 23 separates and extracts a plurality of channel components from the captured image, and generates a plurality of channel images RC, GC, and BC (see FIG. 2) by converting the captured image into images for each channel component. For example, if the color space used in the captured image is RGB (red, green, blue), three channel images RC, GC, and BC consisting of R (red), G (green), and B (blue) color channel images are generated from one captured image, and if the color space used in the captured image is CMY (cyan, magenta, yellow), three channel images RC, GC, and BC consisting of C (cyan), M (magenta), and Y (yellow) color channel images are generated from one captured image. Note that the channel components of the captured image are coordinate components of the color space used in the captured image.
The discrimination means 24 uses the trained model stored in the memory means 10 to discriminate, from the channel images RC, GC, and BC, whether or not a foreign matter FS present in the inspection object WK is a metallic substance.

The operation means 30 is composed of a keyboard, an operation panel, a touch panel, a mouse, various switches, a microphone for voice input operation, etc., and can input operation signals from various operations to the metal substance discrimination device EA.

The imaging means 40 is composed of a camera, a camera, a microscope with imaging function, an imaging sensor, etc., and is capable of transmitting the captured image to the processing means 20.

The output means 50 is composed of display devices such as a display or panel, and alarm devices such as indicator lights or speakers, and is configured to output the determination results by the metal substance discrimination device EA on a screen, or by lighting up or sound, etc.

[Trained model]
The trained model is generated by deep learning using the training channel images RC, GC, and BC of metallic substances and the training channel images RC, GC, and BC of non-metallic substances as training data, and is generated by the following procedure shown in Figure 3.

First, metallic and non-metallic substances of various types, compositions, and shapes are prepared for learning, and images of these metallic and non-metallic substances are taken (step ST11). Images may be taken under a single selected illumination method, or under each of a number of selected illumination methods. Examples of illumination methods include transmitted illumination, coaxial epi-illumination, lateral illumination, and variable illumination. In this embodiment, images of one metallic and non-metallic substance are taken under lateral illumination using a ring illumination and under coaxial epi-illumination.

Next, the multiple channel components are separated and extracted from the captured image captured in step ST11, and multiple channel images RC, GC, and BC are generated from the captured image (step ST12). The reason for generating multiple channel images RC, GC, and BC from the captured image is that, since the way metal substances are captured, for example, the way the luster peculiar to metal is captured, differs for each of the channel images RC, GC, and BC, using the channel images RC, GC, and BC as teacher images rather than the captured image itself provides more information available for discrimination and improves discrimination accuracy.
In the present embodiment, R, G, and B channel components are separated and extracted from an image captured under side illumination, and the captured image is converted into color channel images for each of the R, G, and B channel components to generate three channel images RC, GC, and BC. Similarly, three channel images RC, GC, and BC are generated from an image captured under coaxial epi-illumination. That is, a total of six channel images RC, GC, and BC are generated, consisting of the three channel images RC, GC, and BC under side illumination and the three channel images RC, GC, and BC under coaxial epi-illumination. Each of the generated channel images RC, GC, and BC is labeled as metal or non-metal.

After that, deep learning is performed using the channel images RC, GC, and BC of the learning metallic substance and the channel images RC, GC, and BC of the learning non-metallic substance as training data, and a learned model is generated in which the captured image is input and the determination result of whether the foreign matter FS is a metallic substance or not is output (step ST13). In this embodiment, a convolutional neural network is used as the deep learning algorithm.

[Metallic substance identification method]
A metallic material determining method carried out in the following procedure shown in FIG. 4 will be described by taking as an example a metallic material determining system EA1 equipped with the metallic material determining apparatus EA described above.
First, a user of the metal material discrimination system EA1 (hereinafter, simply referred to as "user") inputs a signal to start automatic operation to the metal material discrimination system EA1 via the operation means 30. Next, when the user or a conveying means (not shown), such as an articulated robot or a belt conveyor, places the inspection object WK at a predetermined position, the processing means 20 drives the imaging means 40 to image the inspection object WK as shown in Fig. 2 (step ST21). The imaging is performed under illumination using the same illumination method as that used when the trained model was created or an illumination method with similar characteristics. In the case of this embodiment, imaging is performed under side illumination 60A using a ring illumination and under coaxial epi-illumination 60B (see Fig. 2).

Then, the image acquisition means 21 acquires an image of the inspection object WK from the imaging means 40 (step ST22). Next, the foreign body extraction means 22 determines whether or not a foreign body FS is present in the inspection object WK in the image acquired in step ST22 (step ST23). The method of foreign body detection performed at this time is not particularly limited, and various techniques can be used, such as performing blob analysis after performing binarization processing on the image, performing differential filtering processing based on an image of a good product of the inspection object WK, or using a judgment model based on machine learning. Then, if it is determined in step ST23 that there is no foreign body FS, the process returns to step ST21 and moves on to inspection of the next inspection object.

On the other hand, if it is determined in step ST23 that a foreign object FS is present, the foreign object extraction means 22 extracts an image of the foreign object FS portion from the captured image acquired in step ST22 (step ST24). Next, the channel image generation means 23 generates a plurality of channel images RC, GC, and BC from the captured image of the foreign object FS portion (step ST25). In the case of this embodiment, the channel image generation means 23 generates three channel images RC, GC, and BC of R, G, and B from the captured image under the lateral illumination 60A, and generates three channel images RC, GC, and BC of R, G, and B from the captured image under the coaxial epi-illumination 60B (see FIG. 2). Thereafter, the discrimination means 24 uses the trained model to discriminate whether the foreign object FS is a metal substance from the channel images RC, GC, and BC, and outputs the discrimination result to the output means 50 (step ST26). Next, the output means 50 notifies the user of the determination result by displaying it on a display device, turning on an alarm device, or emitting a sound from the alarm device (step ST27).

[Discrimination example]
As the metal materials for learning, aluminum, copper, lead, tin, zinc, SUS316L (stainless steel), SUS410L (stainless steel), and brass were prepared, and as the nonmetallic materials for learning, polyimide film, zirconia beads, pieces of phenolic resin, and pieces of paper were prepared, and each material was imaged with a digital microscope under lateral illumination by a ring light. In addition, these metal materials and nonmetallic materials were also imaged with a digital microscope under coaxial epi-illumination. Then, six channel images RC, GC, and BC were generated from each of the images taken under lateral illumination and the images taken under coaxial epi-illumination for each material, and each channel image RC, GC, and BC was labeled as metal or nonmetal, and a trained model was created by a convolutional neural network. When the trained model was used to discriminate the inspection target WK for verification, the prediction accuracy was 95% or more, and good results were obtained.

In addition, the lighting method during imaging was limited to side illumination using a ring light, and the metallic and non-metallic substances were captured with a digital microscope to generate three R, G, and B channel images RC, GC, and BC. A trained model was then created from these channel images RC, GC, and BC. When this trained model was used to discriminate the inspection object WK for verification, the prediction accuracy decreased slightly, but it was still above 95%, and good results were obtained.

According to the above embodiment, by capturing an image of the inspection object WK, it is possible to determine whether the foreign matter FS is a metallic substance from the captured image, so there is no need to perform qualitative or quantitative analysis of the elements, and it is possible to determine whether the foreign matter FS present in the inspection object WK is a metallic substance through a simple process.

In addition, multiple channel images RC, GC, and BC are generated for each captured image taken under multiple lighting conditions with different lighting methods, increasing the amount of information available for discrimination and improving discrimination accuracy.

In addition, because one of the multiple lights used for shooting is a coaxial epi-illumination light, the characteristics of the metal are more easily reflected in the captured image, further improving the accuracy of identification.

As described above, the best configurations and methods for implementing the present invention have been disclosed in the above description, but the present invention is not limited thereto. That is, the present invention has been illustrated and described mainly with respect to specific embodiments, but those skilled in the art can make various modifications to the above-described embodiments in terms of shape, material, quantity, and other detailed configurations without departing from the scope of the technical idea and purpose of the present invention. Furthermore, the descriptions limiting the shapes, materials, etc. disclosed above are provided as examples to facilitate understanding of the present invention, and do not limit the present invention. Therefore, descriptions of the names of components that remove some or all of the limitations on the shapes, materials, etc. are included in the present invention.

For example, the memory means 10 may be built into the metal material discrimination device EA, or may be of a type that is externally attached to the metal material discrimination device EA.

5, the processing means 20 may include a model generating means 25 that generates a trained model by deep learning from the captured image acquired by the image acquiring means 21, and the discrimination means 24 may discriminate whether or not the foreign object FS is a metallic substance using the trained model generated by the model generating means 25. That is, the image acquiring means 21 may acquire a captured image of a metallic substance and a non-metallic substance for learning captured by the imaging means 40, the channel image generating means 23 may generate a plurality of channel images RC, GC, and BC from the captured image, and the model generating means 25 may generate a trained model using the plurality of channel images RC, GC, and BC. In this case, an operation of labeling each of the channel images RC, GC, and BC generated from the captured image of the foreign object FS portion as a metal or non-metal may be performed by the operation means 30.
The image acquisition means 21 may acquire the image of the inspection object WK directly from the imaging means 40, or it may not acquire the image directly from the imaging means 40, and may acquire the image, for example, from an image storage means such as a server or database in which the images acquired by the imaging means 40 are stored.
The channel image generating means 23 may generate the multiple channel images RC, GC, BC not from an image from which the foreign matter FS portion has been extracted, but from the image itself captured by the imaging means 40. In this case, the foreign matter extracting means 22 may extract the foreign matter FS portion from the channel images RC, GC, BC, and the discriminating means 24 may discriminate whether or not the foreign matter FS is a metallic substance from the channel images RC, GC, BC from which the foreign matter FS portion has been extracted.
The discrimination means 24 may discriminate whether or not a foreign matter FS present in the inspection object WK is a metallic substance, and may also discriminate the type and name of the metallic substance or non-metallic substance determined to be the foreign matter FS, and output these discrimination results to the output means 50. In this case, a label of metal or non-metal and a label of the type and name of the metallic substance or non-metallic substance may be attached to each of the channel images RC, GC, and BC of the metallic substance and non-metallic substance for learning, to create a trained model.

The operation means 30 may be configured so as to be separable from the metallic substance determining apparatus EA, or may be configured so as not to be separable from the metallic substance determining apparatus EA.
The operation means 30 may or may not be included in the metallic substance discrimination device EA. If it is not included, the metallic substance discrimination system EA1 may be constituted by the metallic substance discrimination device EA, the imaging means 40 and the output means 50, or an operation signal may be input to the metallic substance discrimination device EA from an external input device capable of communicating with the metallic substance discrimination device EA.

The imaging means 40 may be provided in the metallic substance discrimination device EA, may be provided in the metallic substance discrimination system EA1, or may not be provided in the metallic substance discrimination device EA or the metallic substance discrimination system EA1. If it is not provided, the image acquisition means 21 may acquire an image captured by an external imaging device capable of communicating with the metallic substance discrimination device EA or the metallic substance discrimination system EA1.

The output means 50 may be provided in the metallic substance discrimination device EA, may be provided in the metallic substance discrimination system EA1, or may not be provided in the metallic substance discrimination device EA or the metallic substance discrimination system EA1. If it is not provided, the determination result may be output to an external output device capable of communicating with the metallic substance discrimination device EA or the metallic substance discrimination system EA1.

As a deep learning algorithm for creating a trained model, in addition to convolutional neural networks, deep neural networks, recurrent neural networks, etc. can be used, and are not particularly limited.

The type, material, shape, size, etc. of the inspection object WK, the foreign matter FS, the metallic substance, and the non-metallic substance are not particularly limited. For example, the inspection object WK may be an electronic component such as a capacitor or resistor, a battery material such as an anode material or a cathode material, a precision instrument part such as a personal computer or a printer, etc., or may be an assembly of multiple components such as a circuit board or an MLCC (Multi-Layer Ceramic Capacitor), or may be a single component such as a conductive tape or an insulating tape. The foreign matter FS may be mixed into the inspection object WK or may be attached to the inspection object WK. In addition, the metallic substance may be any of iron, aluminum, stainless steel, lead, tin, zinc, gold, silver, copper, nickel, etc., and the non-metallic substance may be any material other than a metal, such as resin, plastic, wood, glass, ceramics, pottery, cloth, paper, fiber, etc.

The means and steps of the present invention are not limited in any way as long as they can perform the operations, functions or steps described for those means and steps, and are in no way limited to the components and steps of a single embodiment shown in the above embodiment. For example, the image acquisition step may be any step that acquires an image of an object to be inspected, and is not limited in any way as long as it is within the scope of the common general technical knowledge at the time of filing (the same applies to other means and steps).

Description of the Related Art EA... Metallic substance discrimination device 10... Storage means 20... Processing means 21... Image acquisition means 22... Foreign substance extraction means 23... Channel image generation means 24... Discrimination means 30... Operation means 40... Imaging means 50... Output means BC... Channel image FS... Foreign substance GC... Channel image RC... Channel image WK... Inspection object

Claims (5)

an imaging step of imaging the inspection object under each of a plurality of illuminations (excluding transmitted illumination) having different illumination methods;
A method for determining whether a foreign object present in an inspection object is a metallic substance, comprising the steps of:
an image acquiring step of acquiring an image of the inspection object;
a channel image generating step of generating a plurality of channel images by separating and extracting the plurality of channel components from the captured image and converting the captured image into images for the respective channel components;
a discrimination process is carried out using a trained model that has been deep-learned using the channel images of a metallic substance for training and the channel images of a non-metallic substance for training as training data, to discriminate, from the channel images of the test object, whether or not a foreign object present in the test object is a metallic substance, and to discriminate the type and name of the metallic substance or non-metallic substance determined to be a foreign object ;
A metal substance discriminating method , wherein the channel image generating step generates the plurality of channel images for each captured image taken under each illumination . 2. The method according to claim 1 , wherein one of said plurality of illuminations is a coaxial epi-illumination. A metallic substance discrimination device for discriminating whether or not a foreign substance present in an inspection object is a metallic substance, comprising:
an imaging means for imaging the inspection object under each of a plurality of illuminations (excluding transmitted illumination) having different illumination methods;
an image acquisition means for acquiring an image of the inspection object;
a channel image generating means for generating a plurality of channel images by separating and extracting a plurality of channel components from the captured image and converting the captured image into images for the respective channel components;
a discrimination means for discriminating whether or not a foreign object present in the inspection object is a metallic substance from the channel image of the inspection object by using a trained model obtained by deep learning using the channel image of a metallic substance for training and the channel image of a non-metallic substance for training as teacher data, and for discriminating the type and name of the metallic substance or non-metallic substance determined to be the foreign object ,
The metallic substance discriminating device is characterized in that the channel image generating means generates the plurality of channel images for each image captured under each illumination . A method for generating a trained model for determining whether a foreign object present in an inspection object is a metal substance, comprising:
a learning imaging step of imaging a learning metal material and a learning non-metal material;
a learning channel image generating step of generating a plurality of channel images by separating and extracting a plurality of channel components from the captured image captured in the learning imaging step and converting the captured image into images for each of the channel components;
A model generation process is carried out in which the channel images of the metallic substances for learning and the channel images of the non-metallic substances for learning are labeled with a metal or non-metal label and a label with the type or name of the metallic substance or the non-metallic substance, and the channel images are used as training data for deep learning to generate the trained model ;
In the learning imaging step, images of a learning metal material and a learning non-metal material are captured under a plurality of different illumination methods (excluding transmitted illumination),
A method for generating a trained model, characterized in that in the training channel image generation process, the multiple channel images are generated for each captured image taken under each lighting . The method for generating a trained model according to claim 4 , wherein one of the plurality of illuminations is a coaxial epi-illumination.

Images