JP7844993B2 - Image processing method, image processing apparatus, and computer program - Google Patents

Description

This disclosure relates to image processing methods, image processing devices, and computer programs.

Conventionally, a technique is known in which multiple image processing sequences are generated by combining multiple image processing filters, and then an image processing sequence is determined from among these multiple sequences to obtain the desired image (Patent Document 1). Furthermore, conventionally, a technique is known in which, for object recognition using a neural network type machine learning model, preprocessing such as alignment using feature points in the image is performed, and then the machine learning model is trained using the preprocessed image (Patent Document 2).

Japanese Patent Publication No. 2007-87055 International Publication No. 2017/126482

The technology described in Patent Document 1 uses genetic programming to generate multiple image processing sequences and repeatedly evaluate them generation by generation to obtain a highly-rated image processing sequence. However, since the technology in Patent Document 1 only evaluates image processing sequences that combine multiple image processing filters, it may be difficult to obtain the desired processing results with high accuracy in complex processing that combines image processing sequences with other processes unrelated to image processing.

The technology described in Patent Document 2 involves training and integrating multiple machine learning models corresponding to each of several types of preprocessing. Thus, because the technology in Patent Document 2 requires training and integrating multiple machine learning models corresponding to multiple types of preprocessing, it may become difficult to obtain the optimal preprocessing for achieving the desired processing results.

Therefore, there has been a long-standing need for a technology that can obtain an optimal image processing sequence for training a machine learning model, in a technique that combines image processing using image processing sequences with post-processing using machine learning models.

According to a first embodiment of this disclosure, an image processing method is provided. This image processing method comprises: (a) generating a sequence set consisting of a plurality of image processing sequences, each having a different combination of (i) an image input layer for inputting an image, (ii) at least one of a plurality of image processing layers, and (iii) an image output layer for outputting the image after image processing; (b) processing a plurality of training images with each of the plurality of image processing sequences, and generating an intermediate image group for each of the plurality of image processing sequences, which is a set of a plurality of intermediate images that are images after image processing; (c) inputting the intermediate image group for each of the plurality of image processing sequences into the input layer of a plurality of machine learning models of the neural network type, which have an input layer, an intermediate layer, and an output layer, and performing training of the machine learning model so that the layer output, which is the output of the output layer, approaches a target output that is a predetermined target for each of the plurality of training images; and (d) the corresponding plurality For each machine learning model, the process includes: (a) a step of comparing the layer output with the target output to calculate an output evaluation value indicating the degree of similarity between the layer output and the target output; and (b) a step of identifying the machine learning model from which the maximum output evaluation value, which is the highest output evaluation value among the multiple output evaluation values calculated for each of the multiple machine learning models, was calculated, and the image processing sequence used to generate the intermediate image group input to the machine learning model from which the maximum output evaluation value was calculated, if the maximum output evaluation value is less than the threshold, a step of generating a new sequence set by replacing one or more of the generated image processing sequences that have low performance evaluation values regarding the prediction accuracy of the machine learning model with an image processing sequence composed of a new combination of the sequence elements; and steps (b) to (e) are executed again on the new sequence set.

According to a second embodiment of this disclosure, an image processing apparatus is provided. This image processing apparatus includes: a sequence generation unit that generates a sequence set consisting of a plurality of image processing sequences, each having a different combination of (i) an image input layer for inputting an image, (ii) at least one of a plurality of image processing layers, and (iii) an image output layer for outputting the image after image processing; an image processing unit that processes a plurality of training images using each of the plurality of image processing sequences and generates an intermediate image group for each of the plurality of image processing sequences, which is a set of a plurality of intermediate images that are images after image processing; and inputs the intermediate image group for each of the plurality of image processing sequences into the input layer of a plurality of machine learning models of the neural network type, which has an input layer, an intermediate layer, and an output layer, so that the layer output, which is the output of the output layer, approaches a target output that is predetermined for each of the plurality of training images. The system comprises a learning unit that performs learning of the model, and an evaluation unit that, for each of the multiple corresponding machine learning models, compares the layer output with the target output and calculates an output evaluation value indicating the degree of similarity between the layer output and the target output. The evaluation unit, if the maximum output evaluation value (the highest output evaluation value among the multiple output evaluation values calculated for each of the multiple machine learning models) is higher than a predetermined threshold, identifies the machine learning model from which the maximum output evaluation value was calculated and the image processing sequence used to generate the intermediate image group input to the machine learning model. If the maximum output evaluation value is less than the threshold, the sequence generation unit generates a new sequence set by replacing one or more image processing sequences with low performance evaluation values regarding the prediction accuracy of the machine learning model with image processing sequences composed of new combinations of sequence elements.

According to a third embodiment of this disclosure, a computer program is provided. This computer program has the following functions: (a) a function to generate a sequence set consisting of a plurality of image processing sequences, each having a different combination of (i) an image input layer for inputting an image, (ii) at least one of a plurality of image processing layers, and (iii) an image output layer for outputting the image after image processing, wherein the sequence elements are: (b) a function to process a plurality of training images using each of the plurality of image processing sequences, and to generate a group of intermediate images, which is a set of intermediate images, which are the images after image processing, for each of the plurality of image processing sequences; (c) a function to input the group of intermediate images for each of the plurality of image processing sequences into the input layer of a plurality of machine learning models of the neural network type, which have an input layer, an intermediate layer, and an output layer, and to perform training of the machine learning model so that the layer output, which is the output of the output layer, approaches a target output that is a predetermined target for each of the plurality of training images; and (d) the corresponding plurality of machines For each machine learning model, the computer is instructed to perform the following functions: (a) a function that compares the layer output with the target output to calculate an output evaluation value indicating the degree of similarity between the layer output and the target output; (e) a function that identifies the machine learning model from which the maximum output evaluation value, which is the highest of the multiple output evaluation values calculated for each of the multiple machine learning models, is higher than a predetermined threshold, and identifies the machine learning model from which the maximum output evaluation value was calculated and the image processing sequence used to generate the intermediate image group input to the machine learning model from which the maximum output evaluation value was calculated; and (a) a function that generates a new sequence set by replacing one or more of the generated image processing sequences that have low performance evaluation values regarding the prediction accuracy of the machine learning model with an image processing sequence composed of a new combination of the sequence elements, and functions (b) to (e) are executed again on the new sequence set.

A diagram illustrating the generation system of an embodiment. A diagram illustrating memory storage devices. A diagram showing an example of training data. A diagram to explain the gene table. A diagram used to explain a population of a single generation. A diagram showing the image processing sequence of the tree structure generated by the gene translation region. A diagram used to explain machine learning models. A flowchart showing the processes performed by the generation device. A first flowchart showing the pre-processing steps of step S10. A second flowchart showing the pre-processing steps of step S10. A diagram illustrating the pre-treatment process. A diagram illustrating the Type 1 evaluation process. A schematic diagram illustrating the process of generating an individual. A flowchart showing the utilization processing steps in step S50 of Figure 8.

A. Embodiments:
Figure 1 is a diagram illustrating a generation system 10 of an embodiment. The generation system 10 comprises a generation device 20 as an image processing device and an imaging device 200. The generation device 20 and the imaging device 200 can transmit and receive data via wired or wireless means. The imaging device 200 is a camera that captures an image of an object and acquires an image. The image acquired by the imaging device 200 is transmitted to the generation device 20.

The generation device 20 has a preprocessing sequence generation function, a machine learning model generation function, and a result output function. The preprocessing sequence function generates a specific image processing sequence DSq to obtain intermediate images from images to be input to the machine learning model in order to improve the prediction accuracy of the machine learning model. The machine learning model generation function trains a neural network type machine learning model by inputting multiple intermediate images and generates a trained machine learning model. The result output function obtains intermediate images by image processing the image using the specific image processing sequence DSq, inputs the intermediate images into the trained machine learning model generated by the machine learning model generation function, and obtains the output of the output layer. In this embodiment, the generation device 20 automatically generates a machine learning model to achieve a desired objective based on the trained image LM. For example, if the desired objective is to inspect whether an object is good or defective from an image of the object, the specific image processing sequence DSq is used to image process the image of the object to generate intermediate images, and the generated intermediate images are input into the trained machine learning model to determine whether the object is good or defective. Furthermore, the generation device 20 of this embodiment automatically generates an optimal specific image processing sequence DSq for generating intermediate images to be input for training from training images, in order to generate a machine learning model with high prediction accuracy.

The preprocessing sequence generation function generates multiple image processing sequences (Sq) using genetic programming, and identifies a specific image processing sequence (DSq) from among the generated sequences (Sq) to obtain the optimal intermediate image after image processing for training. In this embodiment, the preprocessing sequence generation function, machine learning model generation function, and result output function are all mounted on a single generation device (20), but this is not limited to that. In other embodiments, the preprocessing sequence generation function, machine learning model generation function, and result output function may be mounted on separate devices or reside in the cloud.

The generation device 20 is an electronic computer, such as a personal computer. The generation device 20 comprises a processor 25, a storage device 70, an input unit 150, and a display unit 160. The processor 25 controls the operation of the generation device 20 by executing various programs stored in the storage device 70. The detailed functions of the processor 25 will be described later. The storage device 70 is composed of memory such as RAM or ROM. The storage device 70 stores various programs for realizing the functions of the processor 25. Details of the storage device 70 will be described later. The input unit 150 is an interface for receiving information from the outside. For example, the input unit 150 accepts input of captured images from the imaging device 200, or input of images generated by the user using other devices. The display unit 160 displays various information. The display unit 160 is, for example, a liquid crystal monitor.

The processor 25 functions as a sequence generation unit 30, an image processing unit 40, a learning and evaluation unit 50, and a display control unit 60 by executing various programs in the storage device 70. Some of the functions performed by the processor 25 may be implemented by hardware circuits. Here, in this disclosure, "processor" is a term that encompasses CPUs and GPUs.

The sequence generation unit 30 structurally represents the gene sequence indicated by individual IV in a tree-like structure, thereby generating a sequence set composed of multiple image processing sequences Sq with different combinations of sequence elements. The image processing sequence Sq is represented by a combination of multiple sequence elements. These sequence elements include (i) an image input layer, which is a node for inputting an image; (ii) an image processing layer, which is an intermediate node between the image input layer and the image output layer, acting as an image processing filter; and (iii) an image output layer, which is a node for outputting an intermediate image MM, which is the final image after image processing.

The sequence generation unit 30 comprises an individual generation unit 32 and a gene translation unit 34. The individual generation unit 32 generates individual IV, which is composed of multiple genes arranged in one or two dimensions. Each gene constituting individual IV is a numerical value representing an image input layer in, an image processing layer type, and in this embodiment, an image processing filter Ft. Different image input layers in and different image processing filters Ft are assigned to each numerical range. In other words, individual IV is generated by arranging genes corresponding to the image input layer in and the image processing layer Ft, respectively. The individual generation unit 32 generates a predetermined number of individual IV for each generation. In this embodiment, the individual generation unit 32 generates individual IV with a predetermined gene length by arranging genes in one dimension. In each generation, the combination of sequence elements of multiple image processing sequences Sq is different. For the very first generation, the individual generation unit 32 generates a population 84 consisting of multiple individual IVs by randomly assigning numerical values from 0 to 1 (up to three decimal places) as genes to each gene locus of individual IV. For subsequent generations, the individual generation unit 32 generates new individual IVs from the previous parent generation's individual IVs using at least one method, such as crossover and mutation, thereby generating a population 84 consisting of the newly generated individual IVs and individual IVs maintained in the parent generation, i.e., individuals IVs that have been replicated from the parent generation's individual IVs.

The gene translation unit 34 translates each gene of individual IV by referring to the gene table 82, which is stored in the memory device 70 (described later), and generates an image processing sequence Sq represented as a tree structure. The gene table 82 is a table that defines the corresponding image processing filters for each numerical range of the numerical values that constitute the genes, and the connection relationships between each sequence element.

The image processing unit 40 processes the image input to the image input layer using the image processing sequence Sq generated by the gene translation unit 34 and the specific image processing sequence DSq stored in the memory device 70, and outputs the processed intermediate image from the image output layer.

The learning and evaluation unit 50 learns and evaluates the neural network type machine learning model 88, which will be described later, stored in the memory device 70. The learning and evaluation unit 50 comprises a learning unit 52 and an evaluation unit 54. The learning unit 52 performs learning of the machine learning model 88 using intermediate images MM generated by image processing the learning images LM of the learning data 74 (described later) using the image processing sequence Sq. The evaluation unit 54 performs evaluation processing on the learned machine learning model 88. The evaluation unit 54 can perform a first-type evaluation process using the intermediate output output from the intermediate layer of the machine learning model 88, and a second-type evaluation process using the layer output output from the output layer. Furthermore, the evaluation unit 54 generates prediction results based on the layer output output by inputting images into the machine learning model 88. Details of the evaluation processing performed by the evaluation unit 54 will be described later.

The display control unit 60 displays various information on the display unit 160. For example, the display control unit 60 displays the processed image on the display unit 160, which is processed by a specific image processing sequence DSq or an image processing sequence Sq represented by individual IV.

Figure 2 is a diagram illustrating the storage device 70. The storage device 70 stores training data 74, image filter group 80, gene table 82, population 84, intermediate image group 86, machine learning model 88, specific data 90, and various programs 95. The specific data 90 has a specific image processing sequence DSq and model parameters 94. The model parameters 94 are parameters of a trained machine learning model 88 that has been determined by the evaluation unit 54 to satisfy predetermined criteria. The model parameters 94 are parameters that are updated by the training of the machine learning model 88, such as weights and biases determined by training. The machine learning model 88 having these model parameters 94 is also called a trained specific model 88. The specific image processing sequence DSq is the image processing sequence Sq used to generate the training data 74 used to train the trained specific model 88. The specific image processing sequence DSq is represented, for example, by layer identifiers that identify multiple image processing filters, which are multiple image processing layers, and by the order of these layer identifiers, i.e., their connection relationships. The various programs 95 are programs executed by the processor 25.

Figure 3 shows an example of training data 74. Training data 74 consists of training images LM and target outputs GT, which are ground truth data associated with the training images. Target outputs GT are predetermined target outputs for each of the multiple training images LM. Training images LM are, for example, images of circuit boards, which are industrial products. Target outputs GT are, for example, data showing the inspection results of manufactured industrial products. In this embodiment, this is either label 0, which indicates a good product, or label 1, which indicates a defective product. That is, in this embodiment, target outputs GT are either "0" or "1" and are discrete values. Labels are also called classes. Multiple training images LM are prepared. Training image LM1 is an image of a good circuit board captured by the imaging device 200, and is associated with label 0, which indicates a good product. Training image LM2 is an image of a circuit board with a defect DA present, captured by the imaging device 200, and is associated with label 1, which indicates a defective product.

Each training image LM of the training data 74 is processed by an image processing sequence Sq represented by individual IV generated by the sequence generation unit 30. This generates intermediate images MM from the training images LM. For example, intermediate image MM1 is generated from training image LM1, and intermediate image MM2 is generated from training image LM2. The intermediate images MM and their associated labels are input as a set to the machine learning model 88 for training. To improve the prediction accuracy of the machine learning model 88, the intermediate images MM processed by the image processing sequence Sq are used for training the machine learning model 88. For example, the image processing sequence Sq may perform image processing such that the difference between the defective area DA and the normal area excluding the defective area DA is more emphasized. In other embodiments, the target output GT does not have to be a discrete value. For example, the target output GT may be an image. For example, the target output GT may be a binarized image where the defective area DA is a black image and the area excluding the defective area DA is a white image.

The image filter group 80 shown in Figure 2 is a collection of multiple image filters that are candidates for sequence elements in the image processing sequence Sq. Examples of image filters include mean filters, maximum value filters, minimum value filters, binarization filters, moving average filters, Laplacian filters, Gaussian filters, Sobel filters, gamma correction filters, and filters that combine two images into one. Note that even if image processing filters are of the same type, if their kernel size or coefficients differ, they are stored in the image filter group 80 as different image processing filters.

Gene Table 82 is a table showing the relationship between the connection of sequence elements that are candidates for the image processing sequence Sq, and the correspondence between the numerical values represented by the genes and the types of image processing filters that correspond to them.

Figure 4 is a diagram illustrating the gene table 82. For ease of understanding, it is assumed that the image filter group 80 consists of five image processing filters FtA to FtE. In reality, more than five image processing filters are stored in the image processing filter group. The gene table 82 defines the number of inputs, outputs, and occurrence range for each filter type, which indicates the type of image processing filter. The filter type, as a sequence element type, is an identifier that identifies each image processing filter in the image filter group 80 and the image input layer. In Figure 4, the image input layer is represented by the filter type "in". The number of inputs indicates the number of sequence elements connected to the input side of the image processing filter or image input layer. When the number of inputs is "2", data output from two sequence elements is input to the image processing filter. The number of outputs indicates the number of sequence elements connected to the output side of the image processing filter. In this embodiment, the number of outputs is always "1", and the image processing layer and image input layer are connected to one output-side sequence element.

The appearance range of gene table 82 defines the range of the gene's numerical value VG. In this embodiment, the numerical value from 0 to 1 is divided into six equal parts so that each image processing filter appears randomly with equal probability in the image processing sequence Sq. For example, if the gene's numerical value VG is between 0 and 0.167, the image processing filter FtA is assigned to this gene.

The population 84 shown in Figure 2 is a collection of multiple individuals IV in one generation generated by the individual generation unit 32. Figure 5 is a diagram illustrating the population 84 of one generation G. In this embodiment, an example is described in which five individuals IV1 to IV5 are generated in one generation G. Each individual IV1 to IV5 has multiple gene loci arranged one-dimensionally. The gene lengths of each individual IV1 to IV5 are the same. Each individual IV1 to IV5 is constructed by sequentially arranging genes in gene loci No. 1 to No. 10. The individual generation unit 32 places genes represented by numerical values from 0 to 1 to three decimal places in each gene locus. In other words, in this embodiment, the values that a gene can take at each gene locus are numerical values between 0 and 1 to three decimal places.

Before further explaining Figure 2, we will use Figure 6 to explain the process of generating the image processing sequence Sq by the gene translation unit 34. Figure 6 shows the image processing sequence Sq of the tree structure generated by the gene translation unit 34. The image processing sequence Sq shown in Figure 6 is image processing sequence Sq1, which was generated based on individual IV1 shown in Figure 5.

The gene translation unit 34 generates a tree-structured image processing sequence Sq using individual IV, the gene table 82, and the image filter group 80, starting from the image output layer (out). First, the gene translation unit 34 sets the image output layer (out) as the terminal node and connects the sequence elements in the order of the genes in individual IV. The connection of the sequence elements is performed according to predetermined connection rules. In this embodiment, the image processing sequence Sq is generated according to the rules of prioritizing the placement from the terminal node side in the order of the gene sequence, and prioritizing the connection destination on the left side over the connection destination on the right side, as shown in Figure 6. In other words, the image processing layers or image input layers (in) are placed from the image output layer (out) side in the order of the gene sequence, and if the input side is divided into multiple parts, the left side is prioritized until the left sequence becomes the image input layer (in). In this embodiment, the number of inputs to the image output layer (out) is pre-set to "1".

The gene translation unit 34 refers to the numerical value VG of gene No. 1 of individual VI1, "0.534," and the gene table 82 to determine that "0.534" falls within the range of occurrence of image processing filter FtD. This connects image processing filter FtD to the input side of the image output layer 'out'. Next, the gene translation unit 34 refers to the gene table 82 to determine that the number of inputs to image processing filter FtD is "2," and places two sequence elements on the input side of image processing filter FtD. Specifically, the gene translation unit 34 places the sequence element corresponding to gene No. 2 of individual VI1 at the left input connection point shown in Figure 6. That is, the gene translation unit 34 refers to the numerical value VG of gene No. 2, "0.292," and the gene table 82 to determine that "0.292" falls within the range of occurrence of image processing filter FtB. This places image processing filter FtB at the left input connection point of image processing filter FtD.

Next, the gene translation unit 34 refers to the image processing filter FtB represented by gene No. 2 and the gene table 82, and identifies that the number of inputs to image processing filter FtB is "1," thereby placing one sequence element on the input side of image processing filter FtB. Specifically, the gene translation unit 34 refers to the numerical value VG of gene No. 3, "0.462," and the gene table 82, and identifies that "0.462" falls within the range of appearance of image processing filter FtC. This places image processing filter FtC at the input connection destination of image processing filter FtB indicated by gene No. 3.

Next, the gene translation unit 34 refers to the image processing filter FtC represented by gene No. 3 and the gene table 82, and identifies that the number of inputs to image processing filter FtC is "1". Therefore, it places one sequence element on the input side of image processing filter FtC. Specifically, the gene translation unit 34 refers to the numerical value VG of gene No. 4, "0.856", and the gene table 82, and identifies that "0.856" falls within the range of appearance of image input layer in. This places image input layer in at the input connection destination of image processing filter FtC indicated by gene No. 3. This completes the generation of the sequence on the left side of image processing sequence Sq in Figure 6.

Next, the gene translation unit 34 places the sequence element corresponding to gene No. 5 of individual VI1 at the input connection destination on the right side, as shown in Figure 6. In other words, the gene translation unit 34 refers to the numerical value VG of gene No. 5, "0.138," and the gene table 82 to determine that "0.138" falls within the range of appearance of image processing filter FtA. This places image processing filter FtA at the input connection destination on the right side of image processing filter FtD.

Furthermore, the gene translation unit 34 refers to the image processing filter FtA represented by gene No. 5 and the gene table 82, and identifies that the number of inputs to the image processing filter FtA is "1," thereby placing one sequence element on the input side of the image processing filter FtA. Specifically, the gene translation unit 34 refers to the numerical value VG of gene No. 6, "0.932," and the gene table 82, and identifies that "0.932" is within the range of appearance of the image input layer in. As a result, the image input layer in is placed at the input connection destination of the image processing filter FtA indicated by gene No. 5. With this, the generation of the sequence on the right side of the image processing sequence Sq in Figure 6 is completed. As described above, if all input sequence elements, which are input connection destinations connected to output sequence elements, become image input layers in, the gene translation unit 34 terminates the generation process of the image processing sequence Sq, even if not all genes of individual IV1 are used. Furthermore, when using all the genes of individual IV1, if the gene translation unit 34 finds that not all input sequence elements, which are the input connection destinations, are image input layer in, it sets the remaining unconnected input sequence elements to image input layer in, thereby terminating the image processing sequence Sq generation process.

The gene translation unit 34 terminates the image processing sequence Sq generation process even if not all genes of individual IV1 are used, if all input sequence elements become image input layer in.

The image processing sequence Sq1, represented by individual VI1, has two image input layers in. An image input to one image input layer in is processed by image processing filter FtC, then by image processing filter FtB, generating a first processed image. An image input to the other image input layer in is processed by image processing filter FtA to generate a second processed image. The first and second processed images are then processed by image processing filter FtD, and an intermediate image MM is output by the image output layer out.

As shown in Figure 2, the intermediate image group 86 is a collection of intermediate images MM generated by processing each training image LM of the training data 74 with multiple image processing sequences Sq represented by each individual IV. Each intermediate image MM is associated with the image processing sequence Sq used for image processing and the label of the original training image LM.

The machine learning model 88 is a neural network type machine learning model as described above. In this embodiment, the machine learning model 88 is, for example, a convolutional neural network type. Figure 7 is a diagram illustrating the machine learning model 88. The machine learning model 88 comprises an input layer L1, an intermediate layer L2, and an output layer L3. The input layer L1 is a layer into which an image to be predicted, such as an intermediate image MM, is input. The intermediate layer L2 has at least one layer L22 to L2X. Each layer L22 to L2X of the intermediate layer L2 is composed of multiple nodes Nd, and the first node output Cx, which is output from each node Nd constituting the intermediate layer L2 on the input layer L1 side, is input to the next node Nd of the intermediate layer L2 and outputs a second node output Cx. The node output Cx is a value that indicates the feature quantity extracted from the image input to the input layer L1. In Figure 7, for ease of understanding, the node outputs Cx from each node Nd in layer L22 are shown using the codes Cxa to Cxx. Furthermore, the set of node outputs Cx from multiple nodes Nd in a specific layer of the hidden layer L2, and the set of node outputs Cx from all nodes Nd in all layers L22 to L2X constituting the hidden layer L2, are also called the intermediate output C. The intermediate output C is an X-dimensional vector formed by the arrangement of node outputs Cx from X nodes Nd, and represents the image features input to the machine learning model 88.

The output layer L3 has multiple nodes FNd. The output layer L3 uses the output from the intermediate layer L2 to output a layer output FC indicating the prediction result. The layer output FC is a vector composed of the judgment values FCa and FCb output from each node FNd. The judgment values FCa and FCb are usually normalized values using the softmax function. It can also be said that the judgment values FCa and FCb represent probabilities. The evaluation unit 54 shown in Figure 1 generates a prediction result corresponding to the node FNd1 and FNd2 that output the maximum value among the judgment values FCa and FCb output from each node FNd1 and FNd2 of the output layer L3. For example, if the judgment value FCa output from node FNd1 is the maximum value, the evaluation unit 54 outputs result information indicating "good product" as the prediction result.

Figure 8 is a flowchart showing the process executed by the generation device 20. This process includes the steps of: step S10, which involves generating a specific image processing sequence DSq and training the machine learning model 88; and step S50, which involves utilizing the specific image processing sequence DSq, executed after step S10. Step S10 is performed by the pre-processing sequence generation function and machine learning model generation function of the generation device 20. Step S250 is performed by the result output function of the generation device 20. Step S10 is also referred to as the pre-processing step.

Figure 9 is a first flowchart showing the pre-treatment process in step S10. Figure 10 is a second flowchart showing the pre-treatment process in step S10. Figure 11 is a diagram illustrating the pre-treatment process.

As shown in Figure 9, first, in step S101, the storage device 70 stores a plurality of image processing filters as an image filter group 80. Step S101 is executed when the user inputs a plurality of candidate image processing filters to be used in a specific image processing sequence DSq to the generation device 20. Then, in step S102, the storage device 70 stores training data 74, which consists of a plurality of training images LM associated with the target output GT, which is the correct answer data. Step S102 is executed when the user inputs a plurality of training images LM associated with the target output GT to the generation device 20. Note that the order of steps S101 and S102 is not limited to this.

Next, in step S103, the individual generation unit 32 generates a population 84 consisting of multiple individuals _IVn for the current generation _Gn . The population 84 is composed of multiple individuals _IVn1 to _IVn5 . The generated population 84 is stored in the storage device 70. Next, in step S104, the gene translation unit 34 selects one of the individuals _IVn1 to _IVn5 and translates the selected individual _IVn by referring to the gene table 82 to generate an image processing sequence Sq.

Next, in step S105, the image processing unit 40 processes the training image LM using the image processing sequence Sq generated in step S104 to generate an intermediate image MM. Then, in step S106, the image processing unit 40 outputs the generated intermediate image MM to the storage device 70. The storage device 70 then stores the intermediate image MM as an intermediate image group 86. The intermediate image MM is associated with a sequence identifier for identifying the image processing sequence Sq used in the image processing.

Next, in step S107, the image processing unit 40 determines whether or not the processing in steps S104 to S106 has been performed for all individuals IV of the current generation G _n . If the determination in step S107 is "No", the processing from step S104 onwards is executed again. In this case, in step S104, the gene translation unit 34 selects one of the remaining individuals IV _n for which the processing in steps S104 to S106 has not been completed and generates an image processing sequence Sq. On the other hand, if the determination in step S107 is "Yes", the processing flow shown in Figure 10 is executed. As shown in Figure 11, the learning image LM is image processed with each image processing sequence Sq corresponding to each individual IV _n 1 to IV _n 5 of the current generation _G n to generate an intermediate image MM. In Figure 11, the intermediate images MM generated by each image processing sequence Sq for individuals IV _n 1 to IV _n 5 are shown as intermediate images MM_1 to MM_5. For each individual _IVn1 to _IVn5 in the current generation _Gn , when the processing in steps S104 to S107 is executed, intermediate images MM_1 to MM_5 are stored in the memory device 70 as the individual group 84. Multiple intermediate images MM_1 to MM_5 are generated, each corresponding to multiple training images LM. The set of multiple intermediate images MM generated by one image processing sequence Sq is also called the intermediate image group GMM. As described above, the process in steps S104 to S107 is a process of image processing multiple training images LM using multiple image processing sequences Sq, and generating an intermediate image group GMM, which is a set of multiple intermediate images MM that are the images after image processing, for each of the multiple image processing sequences Sq.

As shown in Figure 10, in step S108, the learning evaluation unit 50 inputs the intermediate image group GMM, generated using the image processing sequence Sq represented by one individual _IVn1 selected from individuals _IVn1 to _IVn5 of the current generation _Gn , into the input layer L1 of the machine learning model 88 before training. Also in step S108, the learning evaluation unit 50 extracts the intermediate output C output from each node Nd of the intermediate layer L2 of the machine learning model 88. The extracted intermediate output C of each node Nd is stored in the storage device 70. Also in step S108, the learning evaluation unit 50 extracts the layer output FC output from each node FNd of the output layer L3 of the machine learning model 88. The extracted layer output FC of each node FNd is stored in the storage device 70. The parameters such as weights and biases of the machine learning model 88 before training can use either predetermined initial values, random values, or values generated by training in the past.

In step S109, the evaluation unit 54 performs a first-type evaluation process using the intermediate output C. Figure 12 is a diagram illustrating the first-type evaluation process. The evaluation unit 54 performs the first-type evaluation process using the intermediate output C for each learning image LM obtained by inputting a plurality of intermediate images MM generated by an image processing sequence Sq represented by a single individual IV into the machine learning model 88. Each intermediate output C is associated with the target output GT of the original learning image LM. The evaluation unit 54 groups the plurality of intermediate outputs C obtained in step S108 according to the corresponding target output GT. In this embodiment, as shown in Figure 12, the evaluation unit 54 groups the plurality of intermediate outputs C into group G0 of label 0, where the target output GT indicates a "good product", and group G1 of label 1, where the target output GT indicates a "defective product". Note that in Figure 12, for ease of understanding, the grouping is shown assuming that the intermediate output C is a two-dimensional vector composed of two node outputs Cx, but in reality, the intermediate output C is represented by an X-dimensional vector, which is greater than 2. The evaluation unit 54 calculates an intermediate evaluation value S _F , which is expressed by the degree of variation σa and σb of the intermediate output C within groups G0 and G1, and the degree of variation β between groups G0 and G1. The representative values of the intermediate output C for each group G0 and G1 are the representative values Gr0 and Gr1. In this embodiment, the representative values are the average values. Specifically, the evaluation unit 54 calculates the intermediate evaluation value S _F using the following formula (1).

In equation (1) above, the numerator on the right side is the sum of the variances representing the degree of variation within each group of intermediate output C. Also, in equation (1) above, the denominator on the right side is the variance representing the degree of variation β between the mean values Gr0 and Gr1 of each group of intermediate output C. In other words, the numerator on the right side of equation (1) above can also be said to be the sum of the degrees of variation σa and σb in Figure 12. It can also be said that the numerator on the right side of equation (1) above is the degree of variation from the mean values Gr0 and Gr1. The groups G0 and G1 for each target output GT of intermediate output C, which are features of the intermediate image MM, are separated from each other, and the smaller the variance within groups G0 and G1, the more accurately the target output GT corresponding to the intermediate image MM is output. In other words, the smaller the intermediate evaluation value S _F in equation (1) above, the more preferable the intermediate image MM is for the machine learning model 88 to use for training to improve prediction accuracy. In other words, the interim evaluation value _S₀F is an indicator of the prediction accuracy of the machine learning model 88, and the smaller the interim evaluation value _S₀F , the higher the prediction accuracy tends to be.

As shown in Figure 10, after step S109, the evaluation unit 54 performs a second type of evaluation process using the layer output FC. The evaluation unit 54 performs a first type of evaluation process using the layer output FC for each learning image LM obtained by inputting multiple intermediate images MM generated by the image processing sequence Sq represented by a single individual IV into the machine learning model 88. The evaluation unit 54 compares the layer output FC output in step S108 with the target output GT associated with the learning image LM from which the layer output FC was derived, and calculates an output evaluation value _S0 that indicates the degree of similarity between the layer output FC and the target output GT. The output evaluation value _S0 is an index value related to the prediction accuracy of the machine learning model 88, and the larger the output evaluation value _S0 , the higher the prediction accuracy tends to be. In this embodiment, the layer output FC and the target output GT are each represented by two-dimensional vectors. Specifically, the layer output FC and target output GT are two-dimensional vectors composed of a value representing the probability of label 0, which indicates a good product, and a value representing the probability of label 1, which indicates a defective product. The layer output FC uses the judgment values FCa and FCb shown in Figure 7 as its elements. The target output GT is a vector where the probability of the class indicated by the target output GT is "1" and the probability of all other classes is "0". The layer output FC and target output GT are vectors arranged in order of the probabilities for each class.

The evaluation unit 54 calculates the output evaluation value _S0 using the following formula (2). In this embodiment, the output evaluation value _S0 is a value relating to the Euclidean distance, and specifically, it is the average value of the Euclidean distance between each target output GT and each corresponding layer output FC. The degree of similarity between the layer output FC and the target output GT can be easily evaluated using the output evaluation value _S0 relating to the Euclidean distance.

The output evaluation value _S0 is not limited to Euclidean distance or values related to Euclidean distance; other index values that represent the degree of similarity can be used. For example, the output evaluation value _S0 may be cosine similarity.

The intermediate evaluation value S _F calculated in step S109 and the output evaluation value S ₀ calculated in step S110 are stored in the storage device 70 in association with the corresponding individual IV. Note that the order of processing between step S109 and step S110 is not limited to the above.

As shown in Figure 10, following steps S109 and S110, in step S111, the evaluation unit 54 determines whether or not the evaluation processes of steps S109 and S110 have been performed for all individual IVs in the current generation Gn. If the determination in step S111 is "No", the learning evaluation unit 50 selects one of the remaining individual IVs in the current generation _Gn that have not undergone the processes of steps S109 and S110, and extracts the intermediate image group _GMM generated by the image processing sequence Sq represented by the selected individual IV from the storage device 70. Then, the processes of steps S108 to S110 are executed again on the extracted intermediate image group GMM. The machine learning model 88 to which the intermediate image group GMM is input in step S108, which is executed again, is the machine learning model 88 before training. Multiple machine learning models 88 are prepared according to multiple individual _IVs 1 to ₅ , and they are all the same neural network type algorithm. In Figure 11, the codes "88_1" to "88_5" are used to distinguish between the machine learning models 88 corresponding to multiple individuals _IVn1 to _IVn5 .

On the other hand, if a "Yes" determination is made in step S111, the evaluation unit 54 determines in step S112 whether the termination condition has been met. The termination condition is that the maximum output evaluation value _S0 max, which is the highest output evaluation value _S0 among the multiple output evaluation values S0 calculated for each of the multiple machine learning models 88 corresponding to the multiple individuals IV1 to IV5, is higher than a predetermined threshold. If the output evaluation value _{S0 max} is higher than the threshold, the learning evaluation unit 50 identifies the image processing sequence Sq used to generate the intermediate image group GMM input to the machine learning model 88 from which the maximum output evaluation value _S0 max was calculated, as a specific image processing sequence DSq in step S115. The identified specific image processing sequence DSq is stored in the storage device 70. If the output evaluation value _S0 max is higher than the threshold, the learning evaluation unit 50 identifies the machine learning model 88 that was used to calculate the maximum output evaluation value _S0 max in step S116. The identified machine learning model 88 is stored in the storage device 70. Specifically, the learning evaluation unit 50 stores the parameters of the machine learning model 88 that was used to calculate the value as model parameters 94 in the storage device 70.

If it is determined in step S112 that the termination condition is not met, the learning unit 52 performs a learning process in step S113, which involves updating the parameters of the multiple machine learning models 88_1 to 88_5. The learning process in step S113 is performed by inputting the intermediate image group GMM for each of the multiple image processing sequences Sq into the input layer L1 of the corresponding multiple machine learning models 88_1 to 88_5, and updating the parameters of the machine learning model 88 so that the layer output FC of the output layer L3 approaches the target output GT defined for each of the multiple learning images LM. The learning unit 52 performs the above learning using, for example, backpropagation. The intermediate image MM input in a single learning process may be the entire intermediate image group GMM or a part of the intermediate image group GMM. For example, the learning unit 52 may perform the learning process using mini-batch learning.

Next, in step S114, the learning unit 52 determines whether the learning process in step S113 has been repeated the specified number of times. If the determination in step S114 is "No", the processes from step S108 to step S113 are repeatedly executed until the specified number of learning times is reached, unless the termination condition is met. In other words, the learning unit 52 completes the learning process by repeating the learning of the machine learning model 88 a predetermined number of times so that the layer output FC approaches the target output GT. The specified number of learning times is N, where "N" is an integer of 2 or more. In this embodiment, "N" is set to 500. This allows for further improvement of prediction accuracy by repeatedly performing learning on the machine learning model N times using the intermediate image group GMM generated by the same image processing sequence Sq.

On the other hand, if the determination is "Yes" in step S114, the individual generation unit 32 executes step S103 again, as shown in Figure 9. Step S103, which is executed again, replaces the individuals IV m from IV _m 1 to IM _m 5 of the parent generation G _m generated in step S103 of the previous processing routine, whose performance evaluation value EV is up to the lowest J, with new individuals _{IV n .} "J" is an integer of 1 or more. In other words, the individual generation unit 32 duplicates the individuals IV whose performance evaluation value EV does not fall within the lowest J, and generates new individuals IV by performing at least one of crossover or mutation on the individuals IV whose performance evaluation value EV does not fall within the lowest J. This makes it easy to generate a new image processing sequence Sq with a different combination of sequence elements from the image processing sequence Sq generated by the previous routine.

The performance evaluation value EV is an index value that shows a positive correlation with the prediction accuracy of the machine learning model 88. In this embodiment, the performance evaluation value EV is calculated by a function that includes the output evaluation value _S0 and the intermediate evaluation value _SF . For example, the performance evaluation value EV is calculated by the following equation (3).
EV=A×S ₀ +B×S _F +C (3)
Here, A is a number greater than or equal to 0, B is a number less than or equal to 0, and C is the constant term.

In equation (3) above, if "A" is set to a number greater than 0, the performance evaluation value EV shows a positive correlation with the output evaluation value S _0. Also, in equation (3) above, by setting at least one of "A" and "B" to a value other than "0", the performance evaluation value EV is expressed using at least one of the output evaluation value S ₀ and the intermediate evaluation value S _F. Furthermore, in equation (3) above, if both "A" and "B" are set to "0", the constant term "C" is set to a value that shows a positive correlation with the prediction accuracy of the machine learning model 88.

Furthermore, in the above equation (3), by setting "B" and "C" to "0", the performance evaluation value EV may be the output evaluation value S ₀ , or may be represented by the output evaluation value S ₀ .

Figure 13 schematically shows the process in step S103 in which the individual generation unit 32 generates current generation G _n individuals _{IV n 1} to IM _n 5 from parent generation G _m individuals IV _m 1 to IM m 5. The current generation G _n individuals IV _n 1 to IV _n 2 shown in Figure 13 are generated by duplicating individuals IV _m 1 to IV _m 2 from the parent generation G _m individuals IV _m 1 to IV _m 5 generated in step S103 of the previous routine. Also, the current generation G _n individual IV _n 3 shown in Figure 13 is an individual generated by crossing parent generation G _m individuals IV _m 1 and IV _m 2. Furthermore, individual _IVn4 of the current generation G _n shown in Figure 13 is an individual generated by crossing individual _IVm2 and individual _IVm3 of the parent generation G _m . Also, individual _IVn5 of the current generation G _n shown in Figure 13 is individual IV generated by mutating individual _IVm2 of the parent generation G _m . In this way, the individual generation unit 32 generates a new individual IV using at least one of crossing and mutation, thereby generating an image processing sequence Sq composed of a different combination of sequence elements than the previous generation G. In this way, if the maximum output evaluation value _S0 max of the image processing sequence Sq, which has been repeatedly trained for a specified number of training iterations, is less than the threshold, one or more image processing sequences Sq with a low performance evaluation value EV among the multiple generated image processing sequences Sq are replaced with an image processing sequence composed of a new combination of sequence elements, thereby generating a new sequence set for the next generation G _n . The new sequence set for the next-generation G _n is executed as the current-generation sequence set, and the following routines from step S104 onwards are executed.

In the processing from step S104 onward for a new sequence set, the same image processing sequence Sq that constitutes the sequence set used in the previous routine may be processed as follows. That is, the intermediate evaluation value S F and output evaluation value S ₀ may be calculated using the intermediate evaluation value S _F , output evaluation value S ₀ , and performance evaluation value EV calculated for the image processing sequence Sq that constitutes the previous sequence set, without performing any subsequent processing to calculate them. This eliminates the need to recalculate the intermediate evaluation value S _F , output evaluation value S _{0 ,} and performance evaluation value EV, thereby improving the processing efficiency of the pre-processing steps performed by the image processing method.

Figure 14 is a flowchart showing the utilization processing step of step S50 in Figure 8. In the utilization processing step, first, in step S501, the generation device 20 stores the specific image processing sequence DSq received by the input unit 150 in the storage device 70. The specific image processing sequence DSq is represented by the image processing filters that constitute the sequence elements and the order in which the sequence elements are arranged. If the utilization processing step is performed by the same generation device 20 that performed the pre-processing step, the specific image processing sequence DSq is already stored in the storage device 70, so step S501 can be omitted. Also, in step S502, the generation device 20 stores the trained specific model 88, which is the machine learning model 88 identified in step S116 of Figure 10, in the storage device 70. Specifically, in step S502, the trained specific model 88, which is a machine learning model 88 having model parameters 94, is stored in the storage device 70. Furthermore, if the utilization processing step is performed by the same generation device 20 that executed the pre-processing step, step S502 can be omitted because the machine learning model 88 with the model parameters 94 is already stored in the storage device 70.

Next, in step S503, the image processing unit 40 acquires an image that is the target of the utilization processing step by reading it from the storage device 70 via the input unit 150. The image acquired in step S503 is, for example, an image of the same type of object as the training image LM. Note that if there are multiple target images, the image processing unit 40 may acquire an image by reading one of the multiple images pre-stored in the storage device 70, or it may acquire the target image from the imaging device 200 each time it is performed to execute the image processing described later.

Next, in step S504, the image processing unit 40 obtains an intermediate image MM from the image acquired in step S503 by performing image processing using a specific image processing sequence DSq. That is, it applies image processing filters stored in the image filter group 80 of the storage device 70 to the image in the order indicated by the specific image processing sequence DSq. The intermediate image MM is generated when step S504 is executed.

Next, in step S505, the evaluation unit 54 inputs the intermediate image MM to the trained specific model 88. Then, in step S506, the evaluation unit 54 outputs the prediction result based on the layer output FC output from the output layer L3 of the trained specific model 88. In this embodiment, the evaluation unit 54 outputs the class corresponding to the node FNd1 and FNd2 that output the maximum judgment value FCa and FCb from each node FNd1 and FNd2 of the layer output FC as the prediction result.

Next, in step S507, the evaluation unit 54 determines whether to continue the user processing process. For example, the evaluation unit 54 determines to continue the user processing process if information indicating the continuation of the user processing process is input from the user, or if there are still unprocessed images that are the target of the user processing process remaining in the storage device 70. If the user processing process is to be continued, the processing from step S503 onwards is executed again. On the other hand, the evaluation unit 54 terminates the user processing process if information indicating the termination of the user processing process is input from the user, or if there are no unprocessed images that are the target of the user processing process remaining in the storage device 70.

According to the above embodiment, if the maximum output evaluation value _S0 max among the multiple output evaluation values _S0 calculated for each machine learning model 88_1 to 88_5 is less than a threshold, as shown in Figure 13, one or more image processing sequences with low performance evaluation values EV among the multiple image processing sequences are replaced with new image processing sequences to generate a new sequence set. This makes it easy to identify the optimal image processing sequence Sq for improving the prediction performance of the machine learning model 88. Furthermore, according to the above embodiment, the intermediate image MM processed by the image processing sequence Sq is input to the machine learning model 88 for learning and image prediction. In other words, even in the complex process of inputting the intermediate image MM after image processing of an image to the machine learning model 88, the prediction accuracy of the machine learning model 88 can be improved. Furthermore, according to the above embodiment, as shown in Figures 9 and 10, the generation of candidate image processing sequences Sq to obtain the optimal intermediate image MM for input to the machine learning model 88, and the evaluation using the output evaluation value obtained by inputting the intermediate image MM generated by the generated image processing sequence Sq to the machine learning model 88 are executed alternately. Therefore, compared to the case where all candidate image processing sequences Sq are generated and then the machine learning model 88 is evaluated according to all image processing sequences Sq, the processing time and computational cost required to identify the trained specific model 88 and the specific image processing sequence DSq can be reduced. Furthermore, according to the above embodiment, even if the number of image processing layers used in the image processing sequence Sq is increased, the generation of the image processing sequence Sq and the evaluation of the machine learning model 88 are performed alternately. This makes it possible to generate the optimal specific image processing sequence DSq for improving prediction accuracy while suppressing an increase in the training time of the machine learning model 88.

Furthermore, according to the above embodiment, as shown in Figure 13, the individual generation unit 32 generates a new image processing sequence Sq by replacing individuals IV _m up to the Jth individual, where the performance evaluation value EV is as follows, with a new individual IV _n . In other words, the performance evaluation value EV can be used to identify image processing sequences Sq up to the Jth individual that are unsuitable for improving the prediction performance of the machine learning model 88. As a result, unsuitable image processing sequences Sq can be replaced with new image processing sequences Sq.

B. Other embodiments:
B-1. Other Embodiment 1:
In the above embodiment, the image processing of the learning image LM by multiple image processing sequences Sq shown by multiple individuals IV belonging to one generation G, the learning process of multiple machine learning models 88, and the calculation of intermediate evaluation values S _F and output evaluation values S _O were each performed sequentially by a single generation device 20. However, these processes may be performed in parallel using multiple generation devices 20.

B-2. Another Embodiment 2:
In the above embodiment, the image processing sequence Sq was represented by individual IV and gene table 82, but is not limited to this. For example, the image processing sequence Sq may be represented by a plurality of sequence elements and a table defining the connection relationships of each sequence element. Also, in the above embodiment, the sequence generation unit 30 generated a plurality of image processing sequences Sq using a genetic algorithm, but the generation method is not limited to this. A plurality of image processing sequences Sq may be generated by randomly arranging a plurality of sequence elements.

B-3. Other Embodiment 3:
In the above embodiment, various algorithms can be used as the neural network type machine learning model 88. For example, the machine learning model 88 may be a vector neural network type model that uses vector neurons, and may be a model called a capsule network. A vector neuron is a neuron whose input and output are vectors. A capsule network is a machine learning model in which vector neurons called capsules are used as nodes in the network.

C. Other forms:
This disclosure is not limited to the embodiments described above, and can be implemented in various forms without departing from its spirit. For example, this disclosure can also be implemented in the following forms (aspects). The technical features in the embodiments described above that correspond to the technical features in each of the forms described below can be replaced or combined as appropriate in order to solve some or all of the problems of this disclosure, or to achieve some or all of the effects of this disclosure. Furthermore, if such technical features are not described as essential in this specification, they can be deleted as appropriate.

(1) According to a first embodiment of the present disclosure, an image processing method is provided. This image processing method comprises: (a) generating a sequence set comprising a plurality of image processing sequences, each having a different combination of (i) an image input layer for inputting an image, (ii) at least one of a plurality of image processing layers, and (iii) an image output layer for outputting the image after image processing; (b) processing a plurality of training images with each of the plurality of image processing sequences, and generating a group of intermediate images, which is a set of intermediate images, for each of the plurality of image processing sequences; (c) inputting the group of intermediate images for each of the plurality of image processing sequences into the input layer of a corresponding plurality of neural network type machine learning models having an input layer, an intermediate layer, and an output layer, and performing training on the machine learning model so that the layer output, which is the output of the output layer, approaches a target output that is a predetermined target for each of the plurality of training images; and (d) the corresponding plurality The process includes: (a) For each of the machine learning models, a step of comparing the layer output with the target output to calculate an output evaluation value indicating the degree of similarity between the layer output and the target output; (e) If the maximum output evaluation value, which is the highest of the multiple output evaluation values calculated for each of the multiple machine learning models, is higher than a predetermined threshold, a step of identifying the machine learning model from which the maximum output evaluation value was calculated and the image processing sequence used to generate the intermediate image group input to the machine learning model from which the calculation was made; and if the maximum output evaluation value is less than the threshold, a step of generating a new sequence set by replacing one or more of the generated multiple image processing sequences that have a low performance evaluation value regarding the prediction accuracy of the machine learning model with an image processing sequence composed of a new combination of the sequence elements; and steps (b) to (e) are executed again on the new sequence set. In this configuration, if the maximum output evaluation value among the multiple output evaluation values calculated for each machine learning model falls below a threshold, one or more image processing sequences with low performance evaluation values are replaced with new image processing sequences to generate a new sequence set. This makes it easy to identify the optimal image processing sequence for improving the predictive performance of the machine learning model.

(2) In the above embodiment, in step (d), the layer output and the target output are each represented by vectors, and the output evaluation value calculated in step (d) may be a value relating to the Euclidean distance. According to this embodiment, the degree of similarity between the layer output and the target output can be evaluated using a value relating to the Euclidean distance.

(3) In the above embodiment, step (c) may be performed by inputting the intermediate image group into the input layer and repeating the learning process of the machine learning model so that the layer output approaches the target output a predetermined N times (where N is an integer of 2 or more). According to this embodiment, the prediction accuracy can be further improved by performing N repeated learning processes on the machine learning model.

(4) In the above embodiment, the target output is represented by discrete values, and the image processing method further includes (f) a step in step (c) in which a plurality of intermediate outputs representing feature quantities output from the intermediate layer corresponding to each of the plurality of training images are obtained, the plurality of intermediate outputs are grouped according to the corresponding target output, and an intermediate evaluation value is calculated represented by the degree of variation of the intermediate outputs within the group and the degree of variation between the groups, wherein the performance evaluation value may be calculated by a function including the output evaluation value and the intermediate evaluation value. According to this embodiment, the performance evaluation value calculated by a function including the output evaluation value and the intermediate evaluation value can be used to identify image processing sequences unsuitable for improving the predictive performance of the machine learning model. As a result, unsuitable image processing sequences can be replaced with new image processing sequences.

(5) In the above embodiment, in step (a), the image processing sequence is represented by an individual having a plurality of genes corresponding to the image input layer and the image processing layer, respectively, and a gene table defining the connection relationships of the sequence elements in the image processing sequence. Step (a) may generate the image processing sequence composed of a new combination of the sequence elements by newly generating the individual using at least one of crossover and mutation. According to this embodiment, an image processing sequence composed of a new combination of sequence elements using at least one of crossover and mutation can be easily generated.

(6) In the above configuration, in step (a) or step (e), for the same image processing sequence that constitutes the sequence set used in the previous routine, the output evaluation value or performance evaluation value for the image processing sequence that constitutes the sequence set used in the previous routine may be used without executing steps (b) through (d). This configuration eliminates the need to recalculate the output evaluation value or performance evaluation value, thereby improving the processing efficiency of the image processing method.

(7) According to a second embodiment of the present disclosure, an image processing apparatus is provided. This image processing apparatus includes a sequence generation unit that generates a sequence set consisting of a plurality of image processing sequences, each having a different combination of (i) an image input layer for inputting an image, (ii) at least one of a plurality of image processing layers, and (iii) an image output layer for outputting the image after image processing, as sequence elements; an image processing unit that processes a plurality of training images using each of the plurality of image processing sequences and generates an intermediate image group for each of the plurality of image processing sequences, which is a set of a plurality of intermediate images that are images after image processing; and inputs the intermediate image group for each of the plurality of image processing sequences into the input layer of a plurality of machine learning models of the neural network type having an input layer, an intermediate layer and an output layer, so that the layer output, which is the output of the output layer, approaches a target output that is predetermined for each of the plurality of training images. The system comprises a learning unit that performs learning of the machine learning model, and an evaluation unit that, for each of the multiple machine learning models, compares the layer output with the target output and calculates an output evaluation value indicating the degree of similarity between the layer output and the target output. The evaluation unit, if the maximum output evaluation value, which is the highest of the multiple output evaluation values calculated for each of the multiple machine learning models, is higher than a predetermined threshold, identifies the machine learning model from which the maximum output evaluation value was calculated and the image processing sequence used to generate the intermediate image group input to the machine learning model that calculated the maximum output evaluation value. The sequence generation unit, if the maximum output evaluation value is less than the threshold, generates a new sequence set by replacing one or more of the generated image processing sequences that have low performance evaluation values regarding the prediction accuracy of the machine learning model with an image processing sequence composed of a new combination of sequence elements. This makes it easy to identify the optimal image processing sequence for improving the predictive performance of machine learning models.

(8) According to a third embodiment of the present disclosure, a computer program is provided. The computer program has the function of generating a sequence set consisting of a plurality of image processing sequences, each having a different combination of (i) an image input layer for inputting an image, (ii) at least one of a plurality of image processing layers, and (iii) an image output layer for outputting the image after image processing, as sequence elements; (b) a function to process a plurality of training images using each of the plurality of image processing sequences, and to generate a group of intermediate images, which is a set of intermediate images, which are images after image processing, for each of the plurality of image processing sequences; (c) a function to input the group of intermediate images for each of the plurality of image processing sequences into the input layer of a plurality of machine learning models of the neural network type, which have an input layer, an intermediate layer, and an output layer, and to perform training of the machine learning model so that the layer output, which is the output of the output layer, approaches a target output that is a predetermined target for each of the plurality of training images; and (d) the corresponding plurality of machines Function (a) includes a function to calculate an output evaluation value indicating the degree of similarity between the layer output and the target output by comparing the layer output and the target output for each machine learning model, and a function to identify the machine learning model from which the maximum output evaluation value, which is the highest of the multiple output evaluation values calculated for each of the multiple machine learning models, is calculated, and the computer is made to execute the following functions: (a) If the maximum output evaluation value is less than the threshold, the computer is made to execute the following functions: (b) If the maximum output evaluation value is less than the threshold, the computer is made to execute the following functions: (a) If the maximum output evaluation value is less than the threshold, the computer is made to execute the following functions: (b) If the computer is made to execute the maximum output evaluation value, which is the highest of the multiple output evaluation values calculated for each of the multiple machine learning models, the computer is made to execute the maximum output evaluation value and the computer is made to execute the image processing sequence used to generate the intermediate image group input to the machine learning model from which the maximum output evaluation value is calculated, and (b) If the computer is made to execute the maximum output evaluation value, it is made to execute the maximum output evaluation value. In this configuration, if the maximum output evaluation value among the multiple output evaluation values calculated for each machine learning model falls below a threshold, one or more image processing sequences with low performance evaluation values are replaced with new image processing sequences to generate a new sequence set. This makes it easy to identify the optimal image processing sequence for improving the predictive performance of the machine learning model.

Furthermore, in the above configuration, if there are no input-side sequence elements unconnected to the output-side sequence elements, step (a) may terminate the image processing sequence generation process corresponding to the individual even if not all of the multiple genes constituting the individual are used as the image processing sequence. This configuration allows for the generation of an image processing sequence even if not all of the multiple genes are used. Furthermore, in the above configuration, if all of the multiple genes constituting the individual are used as the image processing sequence, and there are input-side sequence elements unconnected to the output-side sequence elements, step (a) may terminate the image processing sequence generation process corresponding to the individual by setting the image input layer to the unconnected input-side sequence elements. This ensures reliable generation of an image processing sequence using the individual.

This disclosure can also be implemented in various forms other than those described above. For example, it can be implemented in the form of a non-transitory storage medium on which a computer program is recorded.

10...Generation system, 20...Generation device, 25...Processor, 30...Sequence generation unit, 32...Individual generation unit, 34...Gene translation unit, 40...Image processing unit, 50...Learning and evaluation unit, 52...Learning unit, 54...Evaluation unit, 60...Display control unit, 70...Storage device, 74...Learning data, 80...Image filter group, 82...Gene table, 84...Individual population, 86...Intermediate image group, 88, 88_1 to 88_5...Machine learning model, 90...Specific data, 94...Model parameters, 95...Various programs, 15 0...Input unit, 160...Display unit, 200...Imaging device, DA...Defective area, Ft...Image processing layer, FtA~FtE...Image processing filters, G...Generation, GT...Target output, LM...Training image, MM...Intermediate image, Sq, Sq1...Image processing sequence, VI1~VI5...Individual, out...Image output layer, Cx...Node output, DSq...Specific image processing sequence, FC...Layer output, FCa, FCb...Judgment value, FNd, FNd1, FNd2...Node, Ft, FtA~FtE...Image processing filters, IV, IV _m , IV _m 1 to IV _m 5, IV _n , IV _n 1 to IV _n 5...individual, L1...input layer, L2...hidden layer, L22, L2X...layer, L3...output layer, LM...training image, Nd...node, in...image input layer

Claims (10)

An image processing method,
(a) A step of generating a sequence set comprising a plurality of image processing sequences, each having a different combination of (i) an image input layer for inputting an image, (ii) at least one of a plurality of image processing layers, and (iii) an image output layer for outputting the image after image processing, wherein (a) the sequence elements include:
(b) A step of processing multiple training images by each of the multiple image processing sequences, and for each of the multiple image processing sequences, generating a set of intermediate images which is a collection of multiple intermediate images which are images after image processing,
(c) A step of inputting the group of intermediate images for each of the plurality of image processing sequences into the input layer of a plurality of machine learning models of the neural network type, which has an input layer, an intermediate layer and an output layer, and training the machine learning model so that the layer output, which is the output of the output layer, approaches a predetermined target output for each of the plurality of training images,
(d) For each of the corresponding machine learning models, a step of comparing the layer output with the target output to calculate an output evaluation value that indicates the degree of similarity between the layer output and the target output,
(e) If the maximum output evaluation value, which is the highest of the multiple output evaluation values calculated for each of the multiple machine learning models, is higher than a predetermined threshold, the process includes identifying the machine learning model from which the maximum output evaluation value was calculated and the image processing sequence used to generate the intermediate image group input to the machine learning model from which the calculation was made.
Step (a) includes, if the maximum output evaluation value is less than the threshold, a step of generating a new sequence set by replacing one or more of the generated image processing sequences that have a low performance evaluation value regarding the prediction accuracy of the machine learning model with an image processing sequence composed of a new combination of sequence elements,
Steps (b) through (e) are performed again for the new sequence set ,
In step (d) above, the layer output and the target output are each represented by vectors,
An image processing method wherein the output evaluation value calculated in step (d) is a value relating to Euclidean distance . An image processing method,
(a) A step of generating a sequence set comprising a plurality of image processing sequences, each having a different combination of (i) an image input layer for inputting an image, (ii) at least one of a plurality of image processing layers, and (iii) an image output layer for outputting the image after image processing, wherein (a) the sequence elements include:
(b) A step of processing multiple training images by each of the multiple image processing sequences, and for each of the multiple image processing sequences, generating a set of intermediate images which is a collection of multiple intermediate images which are images after image processing,
(c) A step of inputting the group of intermediate images for each of the plurality of image processing sequences into the input layer of a plurality of machine learning models of the neural network type, which has an input layer, an intermediate layer and an output layer, and training the machine learning model so that the layer output, which is the output of the output layer, approaches a predetermined target output for each of the plurality of training images,
(d) For each of the corresponding machine learning models, a step of comparing the layer output with the target output to calculate an output evaluation value that indicates the degree of similarity between the layer output and the target output,
(e) If the maximum output evaluation value, which is the highest of the multiple output evaluation values calculated for each of the multiple machine learning models, is higher than a predetermined threshold, the process includes identifying the machine learning model from which the maximum output evaluation value was calculated and the image processing sequence used to generate the intermediate image group input to the machine learning model from which the calculation was made.
Step (a) includes, if the maximum output evaluation value is less than the threshold, a step of generating a new sequence set by replacing one or more of the generated image processing sequences that have a low performance evaluation value regarding the prediction accuracy of the machine learning model with an image processing sequence composed of a new combination of sequence elements,
Steps (b) through (e) are performed again for the new sequence set,
Step (c) is performed by inputting the intermediate image group into the input layer and repeating the training of the machine learning model so that the layer output approaches the target output a predetermined N times (where N is an integer of 2 or more) .
The aforementioned target output is expressed as a discrete value,
The aforementioned image processing method further includes:
(f) The process includes a step in step (c) in which a plurality of intermediate outputs representing feature quantities output from the intermediate layer corresponding to each of the plurality of training images are obtained, the plurality of intermediate outputs are grouped according to the corresponding target output, and an intermediate evaluation value is calculated represented by the degree of variation of the intermediate outputs within the group and the degree of variation between the groups,
An image processing method wherein the performance evaluation value is calculated by a function that includes the output evaluation value and the intermediate evaluation value . An image processing method according to claim 1 or claim 2 ,
In step (a), the image processing sequence is represented by an individual having a plurality of genes corresponding to the image input layer and the image processing layer, respectively, and a gene table defining the connection relationships of the sequence elements in the image processing sequence.
The image processing method comprises step (a) generating the image processing sequence composed of a new combination of the sequence elements by generating a new individual using at least one of crossover and mutation. An image processing method,
(a) A step of generating a sequence set comprising a plurality of image processing sequences, each having a different combination of (i) an image input layer for inputting an image, (ii) at least one of a plurality of image processing layers, and (iii) an image output layer for outputting the image after image processing, wherein (a) the sequence elements include:
(b) A step of processing multiple training images by each of the multiple image processing sequences, and for each of the multiple image processing sequences, generating a set of intermediate images which is a collection of multiple intermediate images which are images after image processing,
(c) A step of inputting the group of intermediate images for each of the plurality of image processing sequences into the input layer of a plurality of machine learning models of the neural network type, which has an input layer, an intermediate layer and an output layer, and training the machine learning model so that the layer output, which is the output of the output layer, approaches a predetermined target output for each of the plurality of training images,
(d) For each of the corresponding machine learning models, a step of comparing the layer output with the target output to calculate an output evaluation value that indicates the degree of similarity between the layer output and the target output,
(e) If the maximum output evaluation value, which is the highest of the multiple output evaluation values calculated for each of the multiple machine learning models, is higher than a predetermined threshold, the process includes identifying the machine learning model from which the maximum output evaluation value was calculated and the image processing sequence used to generate the intermediate image group input to the machine learning model from which the calculation was made.
Step (a) includes, if the maximum output evaluation value is less than the threshold, a step of generating a new sequence set by replacing one or more of the generated image processing sequences that have a low performance evaluation value regarding the prediction accuracy of the machine learning model with an image processing sequence composed of a new combination of sequence elements,
Steps (b) through (e) are performed again for the new sequence set,
An image processing method in which, in step (a) or step (e), for the same image processing sequence that constitutes the sequence set used in the routine up to the previous time, the output evaluation value or the performance evaluation value for the image processing sequence that constitutes the sequence set used in the routine up to the previous time is used without executing steps (b) to (d). An image processing device,
A sequence generation unit generates a sequence set comprising a plurality of image processing sequences, each having a different combination of the following sequence elements: (i) an image input layer for inputting an image, (ii) at least one of a plurality of image processing layers, and (iii) an image output layer for outputting the image after image processing;
An image processing unit that processes multiple training images using each of the multiple image processing sequences and generates an intermediate image group, which is a collection of multiple intermediate images that are images after image processing, for each of the multiple image processing sequences,
A learning unit inputs the group of intermediate images for each of the multiple image processing sequences into the input layer of a group of corresponding neural network-type machine learning models having an input layer, an intermediate layer, and an output layer, and performs training on the machine learning model so that the layer output, which is the output of the output layer, approaches a predetermined target output for each of the multiple learning images.
For each of the corresponding machine learning models, the system includes an evaluation unit that compares the layer output with the target output and calculates an output evaluation value indicating the degree of similarity between the layer output and the target output.
The evaluation unit, if the maximum output evaluation value, which is the highest of the multiple output evaluation values calculated for each of the multiple machine learning models, is higher than a predetermined threshold, identifies the machine learning model from which the maximum output evaluation value was calculated and the image processing sequence used to generate the intermediate image group input to the machine learning model from which the calculation was made.
If the maximum output evaluation value is less than the threshold, the sequence generation unit generates a new sequence set by replacing one or more of the generated image processing sequences that have a low performance evaluation value regarding the prediction accuracy of the machine learning model with an image processing sequence composed of a new combination of sequence elements .
The layer output and the target output are each represented by vectors.
The output evaluation value calculated by the evaluation unit is a value relating to the Euclidean distance.
Image processing device. An image processing device,
A sequence generation unit generates a sequence set comprising a plurality of image processing sequences, each having a different combination of the following sequence elements: (i) an image input layer for inputting an image, (ii) at least one of a plurality of image processing layers, and (iii) an image output layer for outputting the image after image processing;
An image processing unit that processes multiple training images using each of the multiple image processing sequences and generates an intermediate image group, which is a collection of multiple intermediate images that are images after image processing, for each of the multiple image processing sequences,
A learning unit inputs the group of intermediate images for each of the multiple image processing sequences into the input layer of a group of corresponding neural network-type machine learning models having an input layer, an intermediate layer, and an output layer, and performs training on the machine learning model so that the layer output, which is the output of the output layer, approaches a predetermined target output for each of the multiple learning images.
For each of the corresponding machine learning models, the system includes an evaluation unit that compares the layer output with the target output and calculates an output evaluation value indicating the degree of similarity between the layer output and the target output.
The evaluation unit, if the maximum output evaluation value, which is the highest of the multiple output evaluation values calculated for each of the multiple machine learning models, is higher than a predetermined threshold, identifies the machine learning model from which the maximum output evaluation value was calculated and the image processing sequence used to generate the intermediate image group input to the machine learning model from which the calculation was made.
If the maximum output evaluation value is less than the threshold, the sequence generation unit generates a new sequence set by replacing one or more of the generated image processing sequences that have a low performance evaluation value regarding the prediction accuracy of the machine learning model with an image processing sequence composed of a new combination of sequence elements.
The learning unit inputs the intermediate image group into the input layer and performs training of the machine learning model for a predetermined N times (where N is an integer of 2 or more) so that the layer output approaches the target output.
The aforementioned target output is expressed as a discrete value,
The evaluation unit acquires a plurality of intermediate outputs representing the feature quantities output from the intermediate layer corresponding to each of the plurality of training images, groups the plurality of intermediate outputs according to the corresponding target output, and calculates an intermediate evaluation value represented by the degree of variation of the intermediate outputs within the group and the degree of variation between the groups.
The image processing apparatus wherein the performance evaluation value is calculated by a function that includes the output evaluation value and the intermediate evaluation value. An image processing device,
A sequence generation unit generates a sequence set comprising a plurality of image processing sequences, each having a different combination of the following sequence elements: (i) an image input layer for inputting an image, (ii) at least one of a plurality of image processing layers, and (iii) an image output layer for outputting the image after image processing;
An image processing unit that processes multiple training images using each of the multiple image processing sequences and generates an intermediate image group, which is a collection of multiple intermediate images that are images after image processing, for each of the multiple image processing sequences,
A learning unit inputs the group of intermediate images for each of the multiple image processing sequences into the input layer of a group of corresponding neural network-type machine learning models having an input layer, an intermediate layer, and an output layer, and performs training on the machine learning model so that the layer output, which is the output of the output layer, approaches a predetermined target output for each of the multiple learning images.
For each of the corresponding machine learning models, the system includes an evaluation unit that compares the layer output with the target output and calculates an output evaluation value indicating the degree of similarity between the layer output and the target output.
The evaluation unit, if the maximum output evaluation value, which is the highest of the multiple output evaluation values calculated for each of the multiple machine learning models, is higher than a predetermined threshold, identifies the machine learning model from which the maximum output evaluation value was calculated and the image processing sequence used to generate the intermediate image group input to the machine learning model from which the calculation was made.
If the maximum output evaluation value is less than the threshold, the sequence generation unit generates a new sequence set by replacing one or more of the generated image processing sequences that have a low performance evaluation value regarding the prediction accuracy of the machine learning model with an image processing sequence composed of a new combination of sequence elements.
An image processing apparatus in which, for the same image processing sequence that constitutes the sequence set used in the previous routine, the output evaluation value or the performance evaluation value for the image processing sequence that constitutes the sequence set used in the previous routine is used. It is a computer program,
(a) A function to generate a sequence set comprising multiple image processing sequences, each having a combination of (i) an image input layer for inputting an image, (ii) at least one of a plurality of image processing layers, and (iii) an image output layer for outputting the image after image processing, wherein the sequence elements are:
(b) A function that processes multiple training images using each of the multiple image processing sequences, and generates a set of intermediate images, which is a collection of multiple intermediate images, for each of the multiple image processing sequences,
(c) A function that inputs the group of intermediate images for each of the multiple image processing sequences into the input layer of a corresponding group of neural network-type machine learning models having an input layer, an intermediate layer and an output layer, and performs training of the machine learning model so that the layer output, which is the output of the output layer, approaches a predetermined target output for each of the multiple training images,
(d) A function to compare the layer output with the target output for each of the corresponding machine learning models and calculate an output evaluation value indicating the degree of similarity between the layer output and the target output,
(e) If the maximum output evaluation value, which is the highest of the multiple output evaluation values calculated for each of the multiple machine learning models, is higher than a predetermined threshold, the computer is made to execute a function to identify the machine learning model from which the maximum output evaluation value was calculated and the image processing sequence used to generate the intermediate image group input to the machine learning model from which the calculation was made.
Function (a) includes, if the maximum output evaluation value is less than the threshold, a function that generates a new sequence set by replacing one or more of the generated image processing sequences that have a low performance evaluation value regarding the prediction accuracy of the machine learning model with an image processing sequence composed of a new combination of sequence elements,
Functions (b) through (e) are executed again for the new sequence set ,
In the function (d) described above, the layer output and the target output are each represented by vectors,
A computer program in which the output evaluation value calculated in the function (d) is a value relating to Euclidean distance . It is a computer program,
(a) A function to generate a sequence set comprising multiple image processing sequences, each having a combination of (i) an image input layer for inputting an image, (ii) at least one of a plurality of image processing layers, and (iii) an image output layer for outputting the image after image processing, wherein the sequence elements are:
(b) A function that processes multiple training images using each of the multiple image processing sequences, and generates a set of intermediate images, which is a collection of multiple intermediate images, for each of the multiple image processing sequences,
(c) A function that inputs the group of intermediate images for each of the multiple image processing sequences into the input layer of a corresponding group of neural network-type machine learning models having an input layer, an intermediate layer and an output layer, and performs training of the machine learning model so that the layer output, which is the output of the output layer, approaches a predetermined target output for each of the multiple training images,
(d) A function to compare the layer output with the target output for each of the corresponding machine learning models and calculate an output evaluation value indicating the degree of similarity between the layer output and the target output,
(e) If the maximum output evaluation value, which is the highest of the multiple output evaluation values calculated for each of the multiple machine learning models, is higher than a predetermined threshold, the computer is made to execute a function to identify the machine learning model from which the maximum output evaluation value was calculated and the image processing sequence used to generate the intermediate image group input to the machine learning model from which the calculation was made.
Function (a) includes, if the maximum output evaluation value is less than the threshold, a function that generates a new sequence set by replacing one or more of the generated image processing sequences that have a low performance evaluation value regarding the prediction accuracy of the machine learning model with an image processing sequence composed of a new combination of sequence elements,
Functions (b) through (e) are executed again for the new sequence set,
The function (c) is performed by inputting the intermediate image group into the input layer and repeating the training of the machine learning model so that the layer output approaches the target output a predetermined N times (where N is an integer of 2 or more).
The aforementioned target output is expressed as a discrete value,
The aforementioned computer program, further,
(f) Function (c) includes a function that acquires a plurality of intermediate outputs representing feature quantities output from the intermediate layer corresponding to each of the plurality of training images, groups the plurality of intermediate outputs according to the corresponding target output, and calculates an intermediate evaluation value represented by the degree of variation of the intermediate outputs within the group and the degree of variation between the groups.
The performance evaluation value is calculated by a computer program using a function that includes the output evaluation value and the intermediate evaluation value. It is a computer program,
(a) A function to generate a sequence set comprising multiple image processing sequences, each having a combination of (i) an image input layer for inputting an image, (ii) at least one of a plurality of image processing layers, and (iii) an image output layer for outputting the image after image processing, wherein the sequence elements are:
(b) A function that processes multiple training images using each of the multiple image processing sequences, and generates a set of intermediate images, which is a collection of multiple intermediate images, for each of the multiple image processing sequences,
(c) A function that inputs the group of intermediate images for each of the multiple image processing sequences into the input layer of a corresponding group of neural network-type machine learning models having an input layer, an intermediate layer and an output layer, and performs training of the machine learning model so that the layer output, which is the output of the output layer, approaches a predetermined target output for each of the multiple training images,
(d) A function to compare the layer output with the target output for each of the corresponding machine learning models and calculate an output evaluation value indicating the degree of similarity between the layer output and the target output,
(e) If the maximum output evaluation value, which is the highest of the multiple output evaluation values calculated for each of the multiple machine learning models, is higher than a predetermined threshold, the computer is made to execute a function to identify the machine learning model from which the maximum output evaluation value was calculated and the image processing sequence used to generate the intermediate image group input to the machine learning model from which the calculation was made.
Function (a) includes, if the maximum output evaluation value is less than the threshold, a function that generates a new sequence set by replacing one or more of the generated image processing sequences that have a low performance evaluation value regarding the prediction accuracy of the machine learning model with an image processing sequence composed of a new combination of sequence elements,
Functions (b) through (e) are executed again for the new sequence set,
A computer program that, in function (a) or function (e), uses the output evaluation value or performance evaluation value for the image processing sequence that constitutes the sequence set used in the previous routine, without executing function (b) to function (d).