JP7633495B2 - Analysis device, analysis method, and program - Google Patents

Description

The present invention relates to an analysis device, an analysis method, and a program.

In the production and development of steel materials, alloying elements such as carbon, manganese, and silicon are actively used. In particular, carbon generates compounds such as cementite through adjustment of the amount added and through thermomechanical treatment, and develops structures such as bainite and pearlite. Manganese also stabilizes the austenite structure. Silicon also promotes the formation of ferrite structures and improves hardenability. The morphological aspects of these structures, such as the phase fraction and grain shape, affect the mechanical and chemical properties of the final product. In addition, structures such as chromium, nickel, niobium, and boron may be added to control the desired properties. In thermomechanical treatment, properties are controlled by using a single or multiple combinations of processes such as hot rolling, cold rolling, annealing, quenching, and tempering. As a result, the final steel material contains a wide variety of microstructures and their distributions, including martensite, bainite, ferrite, pearlite, austenite, and other precipitates. These analyses (structural identification) utilize a variety of analytical instruments, such as optical microscopes (OM), scanning electron microscopes (SEM), electron backscatter diffraction (EBSD), electron probe microanalyzers (EPMA), and transmission electron microscopes (TEM), and use the map data to perform visual structure estimation, quantification of alloy composition from concentration distribution, local orientation analysis, and more.

Among these, a technology has been proposed that estimates the properties of steel material based on images of its microstructure. For example, Patent Document 1 describes a technology in which the material properties of a material are estimated using a neural network model based on scanning electron microscope images of the material, and the estimation result includes information on the degree of certainty of the estimation. Patent Document 2 describes a technology in which image data of sparks generated when grinding steel material is used as input data, and information on the composition of the steel material is used as training data, and a trained model for estimating the composition of the steel material from image data is constructed by machine learning.

JP 2019-12037 A JP 2020-9435 A

The above-mentioned technology for estimating material properties using image processing and machine learning makes it possible to quantify information on the structure, precipitates, inclusions, etc., without relying on visual inspection by an expert as in the past. However, because the above-mentioned technology assumes that images captured using the same method are used as input data, it is difficult to respond to changes in shooting conditions or to select the type of image suitable for identification depending on the structure.

The present invention aims to provide an analysis device, analysis method, and program that can ensure versatility and quantitativeness in estimating the microstructural properties of steel materials using map data and improve usability.

According to one aspect of the present invention, an analysis device is provided that includes a data acquisition unit that acquires map data representing the microstructure of a steel material, and a segmentation unit that classifies elements of the map data into segments using a learning model constructed according to tag information associated with the map data.

According to another aspect of the present invention, an analysis method is provided that includes the steps of acquiring map data representing the microstructure of a steel material, and classifying elements of the map data into segments using a learning model constructed according to tag information associated with the map data.

According to yet another aspect of the present invention, a program is provided for causing a computer to function as an analysis device having a data acquisition unit that acquires map data representing the microstructure of a steel material, and a segmentation unit that classifies elements of the map data into segments using a learning model constructed according to tag information associated with the map data.

According to the above configuration, tag information is associated with map data, and segmentation is performed using a learning model constructed according to the tag information. For example, segmentation can be performed using an optimal learning model depending on changes in the data acquisition method or shooting conditions, ensuring versatility and quantification of the microstructural property estimation of steel materials and improving usability.

1 is a schematic block diagram of an analysis device according to an embodiment of the present invention; 2 is a diagram showing an example of tag information associated with map data in the example of FIG. 1; 5 is a flowchart illustrating an example of a process for updating a model in an analysis device according to an embodiment of the present invention. 10 is a flowchart illustrating an example of a process for correcting input map data in an analysis device according to an embodiment of the present invention.

A preferred embodiment of the present invention will be described in detail below with reference to the accompanying drawings. Note that in this specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals, and redundant explanations will be omitted.

FIG. 1 is a schematic block diagram of an analysis device according to one embodiment of the present invention. As shown in the figure, the analysis device 100 includes a data acquisition unit 101, a segmentation unit 102, a model construction unit 103, a post-processing unit 104, a result output unit 105, a reliability calculation unit 106, a model update unit 107, a data correction instruction unit 108, a data correction unit 109, and a characteristic prediction unit 110. The analysis device 100 is a computer equipped with a processor such as a CPU (Central Processing Unit), a storage device, a communication device, an input/output means, etc., and the above-mentioned functional parts are realized by the processor operating according to a program. The program is stored in a storage device or a removable storage medium and loaded into the analysis device 100. In addition, in the database of the analysis device 100, image data or local analysis data of the microstructure of the steel material is stored as map data 121, the algorithm and parameters of the learning model are stored as model data 122, and the classification results by the segmentation unit 102 (described below) (including the classification results after post-processing by the post-processing unit 104) are stored as result data 123.

The above-described analysis device 100 may be implemented, for example, by a single computer, or may be distributed across multiple computers. Furthermore, at least some of the functional parts of the analysis device 100 may be implemented in one or more server devices, and processing may be performed by transmitting data or processing requests from a client device. Each functional part will be further described below.

The data acquisition unit 101 acquires image data or local analysis data, which are examples of map data 121 that represent the microstructure of steel. The acquired data includes image data such as optical microscope (OM) images, scanning electron microscope (SEM) images, or transmission electron microscope (TEM) images, or local analysis data acquired by an electron probe microanalyzer (EPMA) or electron backscatter diffraction (EBSD) method. Such data have in common that one or more values, such as brightness, crystal orientation, and component composition, are mapped to elements having position information within a specified region, and therefore can be treated in the same way in analysis by the analysis device 100. Such data is also referred to as map data in this specification. In the following description, a collection of combinations of elements and values of the map data 121 may be referred to as an image, but as described above, this expression does not limit the map data 121 to an image.

In this embodiment, the map data 121 acquired by the data acquisition unit 101 is associated with tag information. In the case of image data or local analysis data of steel, for example, a data acquisition method such as OM, SEM, EPMA, or EBSD is associated with each map data 121 as tag information. Furthermore, the steel type classification of the steel (thin plate, thick plate, steel pipe, etc.), composition, heat treatment, etching conditions, or imaging conditions of the image data or local analysis data (such as the type of SEM source) may be associated with each map data 121 as tag information. Furthermore, by combining different types of map data based on the similarity of the tag information, map data having multiple channels, such as RGB images, may be newly created from a database and used for learning. Furthermore, the tag information may include information that can be extracted when the image data or local analysis data is acquired. Specifically, for example, EPMA data can extract information on the component composition of steel when acquired, EBSD can extract the phase fractions of ferrite, austenite, etc., and the average crystal grain size, and SEM and OM can extract the approximate carbon content of steel when acquired.

2 is a diagram showing an example of tag information associated with map data in the example of FIG. 1. The tag information associated with the image file (map data) includes items appropriately selected from the types of measuring equipment, measuring conditions, the composition of each chemical component of the steel, the types of each structure, the fraction of each structure, crystal grain size (average, distribution), heat treatment temperature, and heat treatment speed history (preferably including multiple items). For example, the tag information includes at least one of the steel type classification of the steel, the composition, heat treatment, etching conditions, or the photographing conditions of the image data or local analysis data. In the example shown in FIG. 2, the items include the measuring equipment (SEM), the ferrite fraction indicating the composition, the austenite fraction, the average ferrite grain size, the C composition, the Mn composition, the Si composition, the annealing temperature, and the cooling rate. The tag information items may be, for example, those obtained by records at the time of manufacturing or by separate measurement or analysis, or may be those extracted when the image data or local analysis data is obtained as described above. Note that the items included in the tag information may be modified, such as by adding or deleting, as appropriate if the segmentation described below does not produce good results (e.g., if step S104 in FIG. 3 does not return YES).

The segmentation unit 102 classifies the elements of the map data 121 acquired by the data acquisition unit 101 into segments using a learning model. This process is also referred to as segmentation in this specification. As described above, the map data 121 is data in which values are mapped to elements having location information, and therefore segmentation is a process of classifying elements into segments based on the values mapped to each element and the relationship between values between elements in a specified location relationship. The learning model may use an algorithm such as an FCNN (Fully Convolutional Neural Network) model as described in S. M. Azimi, D. Britz, M. Engstler, M. Fritz and F. Mucklich, "Advanced Steel Microstructural Classification by Deep Learning Methods," Scientific Reports, 8 (2018), 2128, or a CNN hybrid model as described in W. Wang, et al., "Learn to segment single cells with deep distance estimator and deep cell detector," Computers in Biology and Medicine, Vol. 108, May 2019, pp. 133-141. In this embodiment, the classification result by the segmentation unit 102 is given as the probability that each element is classified into a segment, calculated by the learning model.

The model construction unit 103 constructs a learning model used by the segmentation unit 102 according to tag information associated with the map data 121 to be segmented. The model construction unit 103 may select a learning model algorithm that is preset in correspondence with the acquisition method (e.g., OM, SEM, EPMA, or EBSD) of the image data or local analysis data to be segmented, which is indicated by the tag information. Alternatively, the model construction unit 103 may select a learning model algorithm that has been shown to have a high reliability in past analysis, which will be described later, for the acquisition method of the image data or local analysis data to be segmented, which is indicated by the tag information. The model construction unit 103 constructs a model by reading an algorithm or parameters from the model data 122. As will be described later, the model construction unit 103 may perform additional learning or re-learning of the learning model by reading the accumulated image data or local analysis data as teacher data.

The post-processing unit 104 performs post-processing to output the classification results by the segmentation unit 102. Specifically, for example, the post-processing unit 104 performs image labeling and expansion/contraction processing using the OpenCV library. The post-processing unit 104 may perform image correction, for example, using a knowledge base. For example, if there is a segment that is surrounded by other segments and has a size equal to or smaller than a threshold, the segment may be filled in with the other segments. The post-processing unit 104 may also remove noise contained in the image according to a predetermined condition or pattern. Note that the classification results by the segmentation unit 102 may be output as image data even when the input is local analysis data.

The result output unit 105 outputs the classification results by the segmentation unit 102 (including the classification results after post-processing by the post-processing unit 104). Specifically, since the classification results by the segmentation unit 102 are output as images as described above, the result output unit 105 may, for example, output the images to a display or store the image data in a storage device or a removable storage medium. In addition, the result output unit 105 may output the classification results by the segmentation unit 102 to a database as result data 123, and may also output the prediction results of the microstructural properties of the steel material performed by the property prediction unit 110 (described later) based on the accumulated result data 123.

The reliability calculation unit 106 calculates the reliability of the classification result by the segmentation unit 102 (including the classification result after post-processing by the post-processing unit 104). As described above, the classification result by the segmentation unit 102 is given as the probability that each element is classified into a segment. Therefore, even among elements classified into the same segment, the reliability of the classification result differs between an element for which a higher probability (e.g., a probability close to 100%) is calculated and an element for which a lower probability (e.g., a probability close to 50%) is calculated. For example, the reliability calculation unit 106 may identify an area constituted by elements for which the probability of being classified into a segment is lower than a threshold value as an area with low reliability.

The model update unit 107 updates the learning model constructed by the model construction unit 103 according to the reliability calculated by the reliability calculation unit 106. For example, the model update unit 107 may update the model when the area ratio of the area with low reliability in the classification result by the segmentation unit 102 exceeds a threshold. Here, the area ratio is the ratio of the number of elements (e.g., the number of pixels) constituting an area with a reliability lower than a predetermined threshold to the total number of elements (e.g., the total number of pixels) of the image data or local analysis data to be segmented. Specifically, the model update unit 107 may update the learning model by causing the model construction unit 103 to perform additional learning or re-learning of the learning model using image data or local analysis data extracted from the database based on tag information of the image data or local analysis data to be segmented as teacher data. The extracted teacher data is, for example, map data having tag information common or similar to that of the image data or local analysis data to be segmented. Specifically, in the image data or local analysis data of the steel material, data that has a common or similar steel type classification (e.g., thin plate, thick plate, or steel pipe), composition, heat treatment, etching conditions, and imaging conditions (e.g., type of SEM radiation source) is extracted as training data. Therefore, the model update unit 107 extracts tag information that is common or similar to tag information associated with the image data or local analysis data (map data) to be segmented from the database, and performs additional learning or re-learning of the learning model using the map data associated with the extracted tag information. Note that data that is similar to the image data or local analysis data to be segmented may be extracted as training data using, for example, the method described in A. Seko, H. Hayashi, I. Tanaka, "Compositional descriptor-based recommender system for the materials discovery" J. Chem. Phys. Vol. 148 No. 24 (2018), 241719, or indices such as Euclidean distance and cosine similarity.

Alternatively, the model update unit 107 may change the algorithm or network structure of the learning model constructed by the model construction unit 103 based on tag information of the image data or local analysis data to be segmented. In this case, the model construction unit 103 may read parameters already stored as model data 122 for the changed algorithm, or may extract new teacher data using the above-mentioned method and perform additional learning or re-learning. For example, the model update unit 107 may change the algorithm or network structure of the learning model when the reliability is not improved even when teacher data is extracted and additional learning or re-learning of the learning model is performed.

The data correction instruction unit 108 instructs the data acquisition unit 101 to correct the map data 121 acquired according to the reliability calculated by the reliability calculation unit 106. For example, when the area ratio of the area with low reliability in the classification result by the segmentation unit 102 (including the classification result after post-processing by the post-processing unit 104) exceeds a threshold, the data correction instruction unit 108 may instruct the data correction unit 109 described later to correct the map data 121. Here, the update of the learning model by the model update unit 107 and the correction of the map data 121 by the data correction instruction unit 108 may be performed according to, for example, a predetermined priority order. Specifically, the correction of the map data 121 may be instructed when the reliability does not improve even if the learning model is corrected. Alternatively, when areas with low reliability are dispersed in the image, the model update unit 107 may update the learning model, and when areas with low reliability are concentrated, the data correction instruction unit 108 may instruct the map data 121 to be corrected.

The data correction unit 109 corrects the map data 121 acquired by the data acquisition unit 101 in accordance with instructions from the data correction instruction unit 108. The correction of the map data 121 may be performed, for example, by presenting guidelines for operations to correct image data or local analysis data to be subject to segmentation, and having the user manually correct the image data or local analysis data in accordance with the guidelines. Alternatively, the correction of the map data 121 may be performed automatically using a method such as that described in A. Paul, A. Gangopadhyay, A. R. Chintha, D. P. Mukherjee, P. Das and S. Kundu, "Calculation of phase fraction in steel microstructure images using random forest classifier," IET Image Processing, Vol. 12, No. 8, August 2018, pp. 1370-1377, or S. Banerjee et al., "Segmentation of three phase micrograph: an automated approach," CUBE '12: Proceedings of the CUBE International Information Technology Conference, September 2012, pp. 1-4.

The characteristic prediction unit 110 predicts the characteristics of the microstructure of the steel material using the results output to the result output unit 105. Specifically, when a new classification result (including the classification result after post-processing by the post-processing unit 104) is obtained by the segmentation unit 102, the characteristic prediction unit 110 predicts the characteristics of the microstructure of the steel material represented by the new classification result based on the similarity between the new classification result and the result data 123 already stored in the database. The similarity of the result data 123 may be determined, for example, by whether the tag information is common or similar to the image data or local analysis data to be segmented. Specifically, the result data 123 having common or similar steel type classification (thin plate, thick plate, steel pipe, etc.), composition, heat treatment, etching conditions, and shooting conditions (such as the type of SEM radiation source) may be extracted as result data with high similarity. Furthermore, the method described in the above-mentioned publication by A. Seko et al. or indices such as Euclidean distance and cosine similarity may be used to search for result data 123 whose data itself is similar to the image data or local analysis data to be segmented, and the microstructural properties of the steel material may be predicted from the searched result data.

In the above case, the image data or local analysis data of the steel material may be associated with mechanical properties such as tensile strength, hole expansibility, elongation, toughness, and hardness as tag information. In this case, the property prediction unit 110 may predict the mechanical properties of the microstructure of the steel material related to the new classification result based on the similarity between the new classification result by the segmentation unit 102 and the classification result stored in the result data 123, and the mechanical properties associated with the classification result stored in the result data 123. The mechanical properties may be predicted as continuous values or as discrete categories. For the model used to predict the properties, for example, SVM (Support Vector Machine), RF (Random Forest), or MLP (Multi-Layer Perceptron) in the scikit-learn library (https://scikit-learn.org/stable/), or CNN (Convolutional Neural Network) in the Keras library (https://keras.io/), etc. may be used.

Figure 3 is a flowchart showing an example of a process in which a model is updated in an analysis device according to one embodiment of the present invention. In the illustrated example, first, the data acquisition unit 101 acquires image data or local analysis data of the microstructure of the steel material (step S101), and the segmentation unit 102 performs segmentation using the learning model (step S102). Furthermore, the reliability calculation unit 106 calculates the reliability of the classification result by the segmentation unit 102 (step S103). If the reliability is less than a threshold (NO in step S104), the model update unit 107 updates the learning model (step S105), and segmentation is performed again on the same image data or local analysis data using the updated learning model (step S102). The reliability is calculated again (step S103), and if the reliability is equal to or greater than the threshold (YES in step S104), the classification result by the segmentation unit 102 is output (step S106). At this time, the property prediction unit 110 may search for existing result data 123 that is similar to the new classification result (step S107) and predict the properties of the steel material's microstructure from the searched result data (step S108).

Figure 4 is a flowchart showing an example of a process for correcting input map data in an analysis device according to one embodiment of the present invention. In the illustrated example, the process proceeds in the same manner as in the example of Figure 3 above, and if the reliability is less than the threshold (NO in step S104), the data correction instruction unit 108 instructs the data correction unit 109 to correct the image data or local analysis data (step S111). When the input image data or local analysis data is corrected in accordance with the instruction (step S112), the data acquisition unit 101 acquires the corrected image data or local analysis data (step S101), and the process is carried out in the same manner thereafter.

<Example>
Table 1 shows the results of tensile strength characteristic prediction performed by the analysis device according to one embodiment of the present invention. For the tensile strength characteristic prediction, tag information having the composition of carbon, manganese, and silicon of the steel, the ferrite fraction, the martensite and tempered martensite fraction, the pearlite fraction, the austenite fraction, and the fraction of the remaining structure was used. The ferrite fraction, the martensite and tempered martensite fraction, the pearlite fraction, the austenite fraction, and the fraction of the remaining structure are inputted as phase fraction values calculated based on the image segmentation results of the result output unit. In the characteristic prediction unit, a regression model was constructed by using a support vector machine (SVM) to input the tag information and output the tensile strength (MPa). The steel No. shown in Table 1 is a number for managing tag information, and the "teacher data" and "test data" indicate whether the data is the teacher data used to learn the regression model or the test data used to verify the accuracy of the regression model. The results of tensile tests of both the teacher data and the test data are shown to show the versatility of the constructed regression model. The tensile strength was measured in accordance with JIS Z 2241:2011 using a No. 5 test piece of JIS Z 2241:2011. The tensile test piece was taken from the end in the plate width direction at 1/4 of the length. Table 1 shows the results of the strength prediction together with the actual strength measurements, and the average absolute error was 12.0% compared to the prediction results of the test data. The prediction results of the teacher data are shown for comparison, but the accuracy was 8.80%, and it can be confirmed that there is no possibility of overlearning, etc., and that the model is functioning normally. In the study shown in Table 1, the number of teacher data is 20 and the number of test data is 5, but it is expected that higher accuracy will be achieved by increasing the number of data. In addition, for the regression model, by increasing the number of data, it is possible to provide complex models such as a CNN (Convolutional Neural Network) model for learning, and further improvement in accuracy can be expected.

According to one embodiment and examples of the present invention as described above, the identification of the microstructure of steel, which previously required the visual judgment of an expert, can be automated by combining a machine learning algorithm and an image processing algorithm to analyze image data or local analysis data, which not only enables labor saving but also ensures quantitativeness and widens the identification range. In addition, by associating tag information including the data acquisition method (OM, SEM, EPMA, or EBSD) with the image data or local analysis data and performing segmentation using a learning model constructed according to the tag information, segmentation can be performed using an optimal learning model according to changes in the data acquisition method or shooting conditions, ensuring versatility and quantitativeness of the estimation of the microstructure properties of steel materials and improving availability.

The tag information associated with the image data or local analysis data may also include imaging conditions such as the type of SEM source. For example, if imaging conditions differ between the training data of the learning model and the data to be segmented, the accuracy of tissue identification may decrease. However, in this embodiment, for example, if the reliability of the classification results by the segmentation unit 102 decreases, the learning model used for segmentation is updated based on the tag information, so that a decrease in accuracy can be prevented.

The above describes in detail preferred embodiments of the present invention with reference to the attached drawings, but the present invention is not limited to these examples. It is clear that a person skilled in the art of the technical field to which the present invention pertains can conceive of various modified or revised examples within the scope of the technical ideas described in the claims, and it is understood that these also naturally fall within the technical scope of the present invention.

100...analysis device, 101...data acquisition unit, 102...segmentation unit, 103...model construction unit, 104...post-processing unit, 105...result output unit, 106...reliability calculation unit, 107...model update unit, 108...data correction instruction unit, 109...data correction unit, 110...characteristic prediction unit, 121...map data, 122...model data, 123...result data.

Claims (9)

A data acquisition unit that acquires map data representing a microstructure of a steel material;
a segmentation unit that classifies elements of the map data into segments using a learning model constructed according to tag information associated with the map data ;
a reliability calculation unit that calculates the reliability of the classification result by the segmentation unit;
a model update unit that updates the learning model according to the reliability;
Including,
The model update unit performs additional learning or re-learning of the learning model using map data extracted based on the tag information, an analysis device. The analysis device of claim 1, wherein the model update unit extracts tag information that is common or similar to tag information associated with the map data to be segmented, and performs additional learning or re-learning of the learning model using the map data associated with the extracted tag information. The analysis device according to claim 1 , wherein the model update unit changes an algorithm or a network structure of the learning model based on the tag information. a reliability calculation unit that calculates the reliability of the classification result by the segmentation unit;
The analysis device according to claim 1 , further comprising: a data correction instruction unit that instructs correction of the map data in accordance with the reliability. a result output unit that outputs the classification result by the segmentation unit to a database;
The analysis device according to any one of claims 1 to 4, further comprising a characteristic prediction unit that, when a new classification result is obtained by the segmentation unit, predicts the characteristics of the microstructure of the steel material represented by the new classification result based on the similarity between the new classification result and the classification results already stored in the database . The map data includes at least one of image data or local analysis data of the microstructure of the steel material,
The analysis device according to claim 1 , wherein the tag information includes a method for acquiring the image data or the local analysis data. The analysis device according to claim 6 , wherein the tag information includes at least any one of a steel type classification of the steel material, a composition, a heat treatment, an etching condition, or a photographing condition of the image data or the local analysis data. obtaining map data representative of a microstructure of a steel material;
classifying elements of the map data into segments using a learning model constructed in response to tag information associated with the map data ;
a reliability calculation step of calculating a reliability of a classification result obtained by the step of classifying the data into segments;
a model updating step of updating the learning model according to the reliability;
Including,
The model updating step performs additional learning or re-learning of the learning model using map data extracted based on the tag information, in this analysis method. A data acquisition unit that acquires map data representing a microstructure of a steel material;
a segmentation unit that classifies elements of the map data into segments using a learning model constructed according to tag information associated with the map data ;
a reliability calculation unit that calculates the reliability of the classification result by the segmentation unit;
a model update unit that updates the learning model according to the reliability;
Including,
The model update unit is a program for causing a computer to function as an analysis device that performs additional learning or re-learning of the learning model using map data extracted based on the tag information.

Images