JP7782409B2 - Information processing device, information processing method, and information processing program - Google Patents

Description

This disclosure relates to an information processing device, an information processing method, and an information processing program.

Patent Document 1 discloses a technology that divides a region to be identified into multiple subregions, and compares the reflectance spectrum within each subregion with a pre-prepared reference spectrum to identify the plant species within each subregion. Furthermore, this technology excludes spectral data that does not have a reflectance exceeding a predetermined reflectance from the identification process.

Japanese Patent Application Laid-Open No. 2017-133966

Prior art such as Patent Document 1 proposes technology for identifying images by comparing them with standards. However, no technology has been proposed to date for identifying foreign objects contained in images of materials or parts.

The purpose of this disclosure is to provide an information processing device, information processing method, and information processing program that can acquire images that enable high-precision analysis of materials and parts.

an information processing device according to claim 1, comprising: a division unit that receives an input of an image capturing the state of a material or part and divides the image into meshes; a conversion unit that converts the divided image into spectral information for each divided mesh; a calculation unit that calculates a mesh distribution similarity indicating the degree of similarity for each mesh based on predetermined reference information and the spectral information for each mesh; and a discrimination unit that outputs a portion of the meshes into which the image is divided that corresponds to the similarity based on the mesh distribution similarity, wherein the conversion unit converts the spectral information for each mesh into a predetermined power spectral distribution and a predetermined size distribution, and the size distribution is represented by a graph of particle size and frequency obtained by applying an EM algorithm, which is a predetermined machine learning method, to the power spectral distribution, and the calculation unit calculates the similarity between the power spectral distribution and the size distribution as the mesh distribution similarity, and the discrimination unit discriminates foreign matter in the image for each of the power spectral distribution and the size distribution.

The information processing device described in claim 1 calculates similarity based on the spectral information for each mesh and outputs the portion of the mesh corresponding to the similarity. This makes it possible to obtain images that enable highly accurate analysis of materials and parts. Furthermore, similarity can be determined from different characteristics of the power spectrum and size distribution that represent the characteristics of the material.

According to a second aspect of the present invention, in the information processing device of the first aspect, the reference information is an image average of the image, a predetermined reference image, or a predetermined one-dimensional power spectrum. According to the second aspect of the present invention, the reference information can be specified depending on the situation in which it is available.

Furthermore, in the information processing device described in claim 2, the conversion unit may perform processing within a range corresponding to the reference information on a two-dimensional power spectrum obtained by Fourier transforming the divided image, obtain the distribution of the power spectrum, and apply a predetermined machine learning method to the distribution of the power spectrum to obtain the size distribution.

An information processing method according to a third aspect of the present invention is a computer-implemented process comprising the steps of: inputting an image of a photographed state of a material or part, dividing the image into meshes, converting the divided image into spectral information for each divided mesh, calculating a mesh distribution similarity indicating the degree of similarity for each mesh based on predetermined reference information and the spectral information for each mesh, and outputting a portion of the meshes into which the image has been divided that corresponds to the similarity based on the mesh distribution similarity; in the conversion, the spectral information for each mesh is converted into a predetermined power spectral distribution and a predetermined size distribution, and the size distribution is represented by a graph of particle size and frequency obtained by applying an EM algorithm, which is a predetermined machine learning method, to the power spectral distribution; in calculating the mesh distribution similarity, calculating the similarity between the power spectral distribution and the size distribution as the mesh distribution similarity; and in outputting the portion that corresponds to the similarity, discriminating foreign matter in the image for each of the power spectral distribution and the size distribution .

An information processing program according to a fourth aspect of the present invention causes a computer to execute the following processes: inputting an image of a photographed state of a material or part, dividing the image into meshes, converting the divided image into spectral information for each divided mesh, calculating a mesh distribution similarity indicating the degree of similarity for each mesh based on predetermined reference information and the spectral information for each mesh, and outputting a portion of the meshes into which the image is divided that corresponds to the similarity based on the mesh distribution similarity; in the conversion, the spectral information for each mesh is converted into a predetermined power spectral distribution and a predetermined size distribution, and the size distribution is represented by a graph of particle size and frequency obtained by applying an EM algorithm, which is a predetermined machine learning method, to the power spectral distribution; in calculating the mesh distribution similarity, calculating the similarity between the power spectral distribution and the size distribution as the mesh distribution similarity; and in outputting the portion that corresponds to the similarity, discriminating foreign matter in the image for each of the power spectral distribution and the size distribution .

The technology disclosed herein makes it possible to obtain images that enable highly accurate analysis of materials and parts.

FIG. 1 is a diagram showing the configuration of an information processing system. FIG. 2 is a block diagram showing the hardware configuration of the information processing device. FIG. 3 is a diagram showing an example of dividing an image into meshes. FIG. 4 shows an example of a two-dimensional power spectrum and a one-dimensional power spectrum. FIG. 5 shows an example of the power spectrum distribution and the size distribution. FIG. 6 is a diagram showing an example of the representation of the power spectrum distribution and the size distribution. FIG. 7 is a diagram showing an example of a mask image of the discrimination result. FIG. 8 is a flowchart of an information processing method executed by the information processing device.

This section describes an embodiment of the present invention. In this embodiment, we propose a method for identifying outliers, i.e., parts that are determined to be foreign objects, contained in an image of the state of a material or part, to improve analysis accuracy. By excluding parts of the image that contain foreign objects and using them, we can expect to improve analysis accuracy.

Figure 1 is a diagram showing the configuration of an information processing system 100. As shown in Figure 1, in the information processing system 100, an image capture device 102, a user terminal 104, and an information processing device 110 are connected via a network N such as the Internet.

The imaging device 102 is, for example, a microscope, and outputs the captured image. In this embodiment, the image output is an image of material particles. Note that the image may be any image that captures the surface or cross-section of a material or component, and a camera or other device may be used as the imaging device 102.

The user terminal 104 transmits images captured by the imaging device 102 to the information processing device 110. The user terminal 104 also specifies reference information and outputs the reference information to the information processing device 110. The reference information is specified by specifying whether to use the image average of the image, a specified reference image, or a specified one-dimensional power spectrum. If a reference image or one-dimensional power spectrum is specified, these are included in the reference information. The image average uses the image average of the captured image, which means that no other reference information is specified. In addition, the user terminal 104 transmits the number of mesh divisions, the pixel sizes of the reference side and the analysis side, and the discrimination threshold to the information processing device 110. The discrimination threshold is a similarity threshold for determining a foreign object, and is prepared for both the power spectrum distribution and the size distribution.

Figure 2 is a block diagram showing the hardware configuration of the information processing device 110. As shown in Figure 2, the information processing device 110 has a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, storage 14, an input unit 15, a display unit 16, and a communication interface (I/F) 17. Each component is connected to each other via a bus 19 so that they can communicate with each other.

The CPU 11 is a central processing unit that executes various programs and controls each component. That is, the CPU 11 reads programs from the ROM 12 or storage 14 and executes the programs using the RAM 13 as a work area. The CPU 11 controls the above components and performs various calculation processes in accordance with the programs stored in the ROM 12 or storage 14. In this embodiment, the ROM 12 or storage 14 stores an information processing program.

ROM 12 stores various programs and data. RAM 13 temporarily stores programs or data as a working area. Storage 14 is composed of a storage device such as an HDD (Hard Disk Drive) or SSD (Solid State Drive), and stores various programs, including the operating system, and various data.

The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used to enter various inputs.

The display unit 16 is, for example, a liquid crystal display, and displays various information. The display unit 16 may also function as the input unit 15 by adopting a touch panel system.

The communication interface 17 is an interface for communicating with other devices such as terminals. For this communication, a wired communication standard such as Ethernet (registered trademark) or FDDI, or a wireless communication standard such as 4G, 5G, or Wi-Fi (registered trademark) may be used.

The functional components of the information processing device 110 in Figure 1 will be described below. Functionally, the information processing device 110 includes a memory unit 112, a division unit 120, a conversion unit 122, a calculation unit 124, and a determination unit 126. Each functional component is realized when the CPU 11 reads out an information processing program stored in the ROM 12 or storage 14, expands it in the RAM 13, and executes it.

The memory unit 112 stores images of the state of materials or parts received from the user terminal 104. The memory unit 112 also stores reference information, the number of mesh divisions, the pixel sizes of the reference side and the analysis side, and the discrimination threshold.

The division unit 120 inputs an image and divides it into meshes according to the number of divisions of the mesh. Figure 3 shows an example of dividing an image into meshes. (a1) is an image of material particles, and image (a1) is divided into meshes as shown in (a2).

The conversion unit 122 converts the divided image into spectral information for each divided mesh. First, the conversion unit 122 converts the divided image into a two-dimensional power spectrum for each mesh using a Fourier transform. The conversion unit 122 then converts the two-dimensional power spectrum into a one-dimensional power spectrum, which is used as the power spectrum distribution. The conversion unit 122 also obtains a size distribution from the power spectrum distribution. In this way, the conversion unit 122 converts the spectral information for each mesh into a predetermined power spectrum distribution and a predetermined size distribution. The size distribution will be described later.

Figure 4 shows examples of two-dimensional and one-dimensional power spectra. (b1) is an example of a two-dimensional power spectrum of a divided mesh, showing that the power spectrum spreads from the center, with the center being the origin of spatial frequency. (b2) is an example of a one-dimensional power spectrum (power spectrum distribution) converted from the two-dimensional power spectrum of a mesh, represented as a graph with the vertical axis representing scattering intensity and the horizontal axis representing frequency.

The conversion performed by the conversion unit 122 involves deleting the q range and applying the EM algorithm. Deleting the q range means that when the x-axis range of the power spectrum obtained from the divided image differs from the reference information, the range is adjusted to the shorter x-axis, and the range outside the adjusted x-axis is not used in the analysis. In this way, the conversion unit 122 processes the two-dimensional power spectrum according to the reference information, and obtains the distribution of the power spectrum, which is a one-dimensional power spectrum. By deleting the q range, the distribution of the power spectrum, which is a one-dimensional power spectrum, can be obtained as spectral information from the two-dimensional power spectrum.

The EM algorithm is a method for predicting a Gaussian mixture distribution. The EM algorithm obtains a size distribution from the power spectrum distribution as spectral information. In this way, the conversion unit 122 applies the EM algorithm, a predetermined machine learning method, to the power spectrum distribution to obtain the size distribution.

Figure 5 shows an example of a power spectrum distribution and a size distribution. (c1) is a power spectrum distribution, which, like (b2) above, is represented as a graph with the vertical axis representing scattering intensity and the horizontal axis representing frequency. (c2) is a size distribution, which is represented as a graph with the vertical axis representing particle size and the horizontal axis representing frequency.

The calculation unit 124 calculates the mesh distribution similarity, which indicates the degree of similarity for each mesh, based on predetermined reference information and the spectral information for each mesh. The mesh distribution similarity is the similarity between the power spectrum distribution and the size distribution. The mesh distribution is expressed as an identifiable distribution using color gradations or the like. Figure 6 shows an example of the representation of the power spectrum distribution and the size distribution. (d1) is an example of the power spectrum distribution, and (d2) is an example of the size distribution, which represent the similarity for each divided mesh. In both examples, areas with low numerical values and a strong gradation indicate low similarity. Note that the representation of the power spectrum distribution and the size distribution is just an example, and color patterns or the like may also be used.

The discrimination unit 126 discriminates foreign matter in the image for each of the power spectrum distribution and size distribution based on the similarity of the mesh distribution and a discrimination threshold. Foreign matter in the image is assumed to be foreign matter particles that are outliers of the materials or parts present in the image, or foreign matter particles due to objects other than the material or raw material. The discrimination unit 126 outputs mask images in which portions of the power spectrum distribution and size distribution determined to be foreign matter are masked. Figure 7 shows an example of a mask image of the discrimination result. (e1) is a mask image of the discrimination result for the power spectrum distribution, and (e2) is a mask image of the discrimination result for the size distribution. Discrimination results are obtained for each mesh in the mask image, and meshes determined to be foreign matter are masked in black. Because the power spectrum distribution and size distribution focus on different features, there are differences in the masked portions.

(Flow of Control)
The flow of processing as an information processing method executed by the information processing device 110 of this embodiment will be described with reference to the flowchart in Fig. 8. Fig. 8 is a flowchart of the information processing method executed by the information processing device 110. The processing in the information processing device 110 is executed by the CPU 11 functioning as each unit.

In step S100, the CPU 11 acquires an image of the state of the material or part, as well as reference information.

In step S102, the CPU 11 performs processing according to the reference information to create a similarity standard. If the reference information is an image average, an image average is created from the acquired images and used as the similarity standard. If the reference information is a reference image, the reference image used as the similarity standard is read and spectral information is created. The spectral information can be created by performing processing similar to steps S106 to S110, described below. If the reference information is a one-dimensional power spectrum, the one-dimensional power spectrum used as the similarity standard is read as spectral information.

In step S104, the CPU 11 inputs the image and divides it into meshes according to the number of mesh divisions.

In step S106, the CPU 11 performs a Fourier transform on the divided image for each divided mesh, converting it into a two-dimensional power spectrum.

In step S108, the CPU 11 performs processing on each two-dimensional power spectrum of the mesh within a range corresponding to the reference information, and converts it into a power spectral distribution, which is a one-dimensional power spectrum.

In step S110, the CPU 11 converts the power spectrum distribution of each mesh into a size distribution using a predetermined machine learning method. As a result, the image for each mesh is converted into a power spectrum distribution and a size distribution as spectral information for each mesh.

In step S112, the CPU 11 calculates the mesh distribution similarity, which indicates the degree of similarity for each mesh, based on the similarity criteria created from the reference information in step S102 and the spectral information for each mesh. The mesh distribution similarity is the similarity between the power spectrum distribution and the size distribution.

In step S114, the CPU 11 determines foreign matter in the image for each of the power spectrum distribution and size distribution based on the mesh distribution.

In step S116, the CPU 11 outputs a mask image that identifies foreign matter in the image for each mesh for each of the power spectrum distribution and size distribution.

As described above, the information processing device 110 of this embodiment can acquire images that enable highly accurate analysis of materials and parts.

In the above embodiment, the various processes executed by the CPU 11 after loading software (programs) may be executed by various processors other than the CPU. Examples of processors in this case include PLDs (Programmable Logic Devices) such as FPGAs (Field-Programmable Gate Arrays), whose circuit configuration can be changed after manufacture, and dedicated electrical circuits, such as ASICs (Application Specific Integrated Circuits), which are processors with circuit configurations specifically designed to execute specific processes. Furthermore, each of the above processes may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA). Furthermore, the hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor devices.

Furthermore, in the above embodiment, the information processing program has been described as being pre-stored (installed) on a computer-readable non-transitory recording medium. For example, the information processing program is pre-stored in ROM 12 or storage 14. However, this is not limiting, and each program may be provided in a form recorded on a non-transitory recording medium such as a CD-ROM (Compact Disc Read Only Memory), a DVD-ROM (Digital Versatile Disc Read Only Memory), or a USB (Universal Serial Bus) memory. Furthermore, the information processing program may be downloaded from an external device via a network.

The processing flow described in the above embodiment is an example, and unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

100 Information processing system 102 Image capturing device 104 User terminal 110 Information processing device 112 Storage unit 120 Division unit 122 Conversion unit 124 Calculation unit 126 Discrimination unit

Claims (4)

a division unit that receives an image of the state of a material or part and divides it into meshes;
a conversion unit that converts the divided image into spectral information for each divided mesh;
a calculation unit that calculates a mesh distribution similarity indicating a degree of similarity for each mesh based on predetermined reference information and spectrum information for each mesh;
a discrimination unit that outputs a portion of the meshes obtained by dividing the image according to the similarity of the mesh distribution;
Including,
The conversion unit converts the spectral information for each mesh into a predetermined power spectrum distribution and a predetermined size distribution,
The size distribution is shown as a graph of particle size and frequency obtained by applying an EM algorithm, which is a predetermined machine learning method, to the distribution of the power spectrum;
the calculation unit calculates a similarity between the power spectrum distribution and the size distribution as the similarity between the mesh distributions,
The information processing device , wherein the discrimination unit discriminates foreign matter in the image for each of the power spectrum distribution and the size distribution . The information processing device according to claim 1, wherein the reference information uses an image average of the image, a predetermined reference image, or a predetermined one-dimensional power spectrum. Input an image of the state of the material or part, divide it into meshes,
The divided image is converted into spectral information for each divided mesh;
Calculating a mesh distribution similarity indicating a degree of similarity for each mesh based on predetermined reference information and the spectrum information for each mesh;
outputting a portion of the meshes obtained by dividing the image according to the similarity of the mesh distribution ;
In the conversion, the spectral information for each mesh is converted into a predetermined power spectrum distribution and a predetermined size distribution,
The size distribution is shown as a graph of particle size and frequency obtained by applying an EM algorithm, which is a predetermined machine learning method, to the distribution of the power spectrum;
In the calculation of the mesh distribution similarity, a similarity between the power spectrum distribution and a similarity between the size distribution are calculated as the mesh distribution similarity;
In the output of the portion according to the similarity, a foreign object in the image is identified for each of the power spectrum distribution and the size distribution.
An information processing method in which processing is performed by a computer. Input an image of the state of the material or part, divide it into meshes,
The divided image is converted into spectral information for each divided mesh;
Calculating a mesh distribution similarity indicating a degree of similarity for each mesh based on predetermined reference information and the spectrum information for each mesh;
outputting a portion of the meshes obtained by dividing the image according to the similarity of the mesh distribution ;
In the conversion, the spectral information for each mesh is converted into a predetermined power spectrum distribution and a predetermined size distribution,
The size distribution is shown as a graph of particle size and frequency obtained by applying an EM algorithm, which is a predetermined machine learning method, to the distribution of the power spectrum;
In the calculation of the mesh distribution similarity, a similarity between the power spectrum distribution and a similarity between the size distribution are calculated as the mesh distribution similarity;
In the output of the portion according to the similarity, a foreign object in the image is identified for each of the power spectrum distribution and the size distribution.
An information processing program that causes a computer to execute a process.