JP7627564B2 - IMAGE PROCESSING DEVICE, IMAGE PROCESSING METHOD, LEARNING MODEL GENERATION DEVICE, LEARNING MODEL PRODUCTION METHOD, LEARNED MODEL, AND PROGRAM - Google Patents

Description

The present invention relates to an image processing device and related technology, and in particular to technology for detecting abnormalities in an object using a captured image of the object.

There is technology that uses a convolutional neural network (CNN) to analyze images.

In a convolutional neural network, a convolutional layer, a pooling layer, etc. are generally provided after an input layer, and a fully connected layer is provided at the end (see Patent Document 1, etc.). Also, as shown in Patent Document 1 (see paragraph 0040), etc., a global average pooling process (a process of calculating the average value of all channels for each channel in a feature map) may be performed before the fully connected layer.

JP 2020-8896 A

By the way, it is possible to use a convolutional neural network to detect the presence or absence of an abnormality (such as a scratch) in an object based on a photographed image of the object.

For example, it is possible to apply a convolutional neural network such as that described in Patent Document 1 to the detection of abnormalities in objects.

However, when the global average pooling process described above is performed, the pixel values of multiple pixels, including both abnormal and normal pixels, are averaged. Therefore, there is a risk that small abnormalities (such as scratches) in the image may be overlooked.

Therefore, the objective of this invention is to provide technology that can prevent small abnormalities in images from being overlooked and detect abnormalities more accurately.

In order to solve the above problem, the image processing device of the present invention includes a control unit that detects abnormalities in a captured image of an object based on a learning model machine-learned using teacher data that inputs each captured image and outputs information about abnormalities in each of the captured images, wherein the learning model is a model machine-learned using a value obtained by generating a second feature map by integrating pixel values at the same position and/or nearby positions of each channel in a first feature map, which is an intermediate feature map in a convolutional neural network and has a plurality of channels, and performing a global max pooling process on the second feature map, and the control unit detects abnormalities in the object by inputting the captured image of the object to the learning model.

The first feature map may be generated by an intermediate layer including multiple convolutional layers and multiple pooling layers arranged after an input layer in the convolutional neural network.

The convolutional neural network may not have a fully connected layer after a pooling layer that performs the global max pooling process on the second feature map.

The learning model may be a machine-learned model that is generated by comparing the maximum value obtained by performing the global max pooling process on the second feature map, or a transformed value obtained by transforming the maximum value without multiplying the maximum value by a learning parameter, with correct answer data.

The learning model may be a machine-learned model obtained by performing the global max pooling process on the second feature map having two or more channels, converting each of the multiple maximum values obtained for each channel without averaging the multiple maximum values, and comparing the converted values with ground truth data.

The process of generating the second feature map may include integrating pixel values at the same pixel position in multiple channels of the first feature map using a 1×1 convolution filter that weights and adds them together, and calculating pixel values at each pixel position in the channels of the second feature map.

The process of generating the second feature map may include a process of integrating pixel values at the same pixel position in multiple channels of the provisional feature map after the convolution process for each of the multiple channels of the first feature map by weighted addition, and calculating the pixel value of each pixel position in the channel of the second feature map.

The process of generating the second feature map may include a process of calculating pixel values at each pixel position in one channel of the second feature map a predetermined number of times using a convolution filter that weights and adds pixel values obtained by integrating pixel values at the same position and/or adjacent positions in multiple channels of the first feature map, thereby generating a predetermined number of channels in the second feature map.

The second feature map may have two or more channels that are fewer than the number of channels of the first feature map, and the learning model may be a model that is machine-learned using values obtained by performing the global max pooling process on at least a portion of the two or more channels.

The second feature map may be a feature map having a single channel, and the learning model may be a model trained by machine learning using values obtained by performing the global max pooling process on the single channel.

The control unit may display the information visualized based on the second feature map on a specified display unit.

In order to solve the above problem, an image processing method according to the present invention includes a step of: a) detecting an abnormality in a photographed image of an object based on a learning model machine-learned using teacher data in which each photographed image is input and information regarding an abnormality in each of the photographed images is output, wherein the learning model is a model machine-learned using a value obtained by generating a second feature map by integrating pixel values at the same position and/or nearby positions of each channel in a first feature map, which is an intermediate feature map in a convolutional neural network and has a plurality of channels, and performing a global max pooling process on the second feature map, and wherein in step a), the photographed image of the object is input to the learning model to detect an abnormality in the object.

In order to solve the above problem, the present invention provides a program for causing a computer to execute the steps of: a) detecting an abnormality in a photographed image of an object based on a learning model machine-learned using teacher data in which each photographed image is input and information regarding an abnormality in each of the photographed images is output , the learning model being a model machine-learned using a value obtained by generating a second feature map by integrating pixel values at the same position and/or nearby positions of each channel in a first feature map, which is an intermediate feature map in a convolutional neural network and has a plurality of channels, and performing a global max pooling process on the second feature map, and wherein in step a), an abnormality in the object is detected by inputting the photographed image of the object to the learning model.

To solve the above problem, the learning model generation device of the present invention is a learning model generation device that inputs a captured image and generates a learning model that outputs information about anomalies in the captured image, and is characterized by having a control unit that generates a second feature map by integrating pixel values at the same position and/or nearby positions of each channel in a first feature map (an intermediate feature map in a convolutional neural network of the learning model and having multiple channels) based on the first feature map, and performs machine learning in the learning model using values obtained by performing a global max pooling process on the second feature map.

To solve the above problem, the learning model production method according to the present invention is a learning model production method for inputting a captured image and producing a learning model that outputs information about anomalies in the captured image, and is characterized by comprising the steps of: a) generating a second feature map by integrating pixel values at the same position and/or nearby positions of each channel in a first feature map (an intermediate feature map in a convolutional neural network of the learning model and having a plurality of channels) based on the first feature map, and performing machine learning in the learning model using values obtained by performing a global max pooling process on the second feature map.

In order to solve the above problem, the trained model of the present invention is a trained model that has been machine-learned to output information regarding an abnormality in an inference target image in response to an input of an inference target image, and is characterized in that the trained model is for causing a computer to execute the following processes: a process of generating a first feature map for the inference target image, the first feature map being an intermediate feature map in a convolutional neural network and having a plurality of channels; a process of generating a second feature map by integrating pixel values at the same position and/or nearby positions of each channel in the first feature map based on the first feature map; and a process of outputting information regarding the abnormality based on a value obtained by performing a global max pooling process on the second feature map.

To solve the above problem, the program of the present invention is characterized in that it is a program for causing a computer to execute the steps of: a) generating a second feature map by integrating pixel values at the same position and/or nearby positions of each channel in a first feature map, which is an intermediate feature map having multiple channels in a convolutional neural network of a learning model that inputs a captured image and outputs information about anomalies in the captured image, and performing machine learning in the learning model using values obtained by performing a global max pooling process on the second feature map.

The present invention makes it possible to prevent small abnormalities in an image from being overlooked and to detect abnormalities more accurately.

FIG. 1 is a schematic diagram showing an image processing system. FIG. 2 is a conceptual diagram showing the processing in the learning stage of a learning model. FIG. 1 is a conceptual diagram showing the processing at the inference stage based on a learning model. 10 is a flowchart showing the process of the controller in a learning stage. 13 is a flowchart showing the process of the controller in the inference stage. FIG. 2 is a diagram showing a learning model according to the first embodiment; FIG. 2 is a conceptual diagram illustrating an example of a configuration of a feature extraction layer. FIG. 13 is a diagram showing how a second feature map is generated using a convolution filter of size 1×1. FIG. 13 is a diagram showing how a second feature map is generated using a convolution filter of size N×N. FIG. 13 is a diagram showing how a second feature map is generated using a convolution filter of size N×N. FIG. 1 illustrates a convolution process within a channel. FIG. 13 is a diagram showing how output values are generated by GMP processing on a second feature map. 13 is a diagram showing how output values are generated by GMP processing and normalization processing on a second feature map. FIG. A diagram showing GMP processing for one channel of the second feature map. FIG. 11 is a diagram showing a learning model according to the second embodiment. FIG. 2 shows how a second feature map 250 having C2 channels is generated using C2 convolution filters of size 1×1. FIG. 2 shows how a second feature map 250 having C2 channels is generated using C2 convolution filters of size N×N. FIG. 13 is a diagram showing how output values are generated by GMP processing or the like for a second feature map having two channels. FIG. 13 is a diagram showing how output values are generated by GMP processing or the like for a second feature map having two channels. FIG. 13 is a diagram showing a learning model according to the third embodiment. A figure showing how GMP processing etc. is applied to some of the five channels of the second feature map to generate output values. FIG. 13 is a diagram showing a screen for displaying a determination result regarding the presence or absence of an abnormality, etc. FIG. 13 is a diagram showing another screen for displaying the determination result regarding the presence or absence of an abnormality, etc. FIG. 4 is a diagram illustrating an example of an input image. FIG. 13 is a diagram showing a learning model according to a comparative example.

The following describes an embodiment of the present invention with reference to the drawings.

1. First embodiment
<1-1. System Overview>
Fig. 1 is a schematic diagram showing an image processing system 1. As shown in Fig. 1, the image processing system 1 includes an image capture device 20 that captures an image of an object, and an image processing device 30 that processes the captured image. The image processing device 30 is also called an abnormality detection device (or an appearance inspection device) because it is a device that detects an abnormality in an object (e.g., a mechanical part) based on a captured image of the object (a device that inspects the appearance of the object). Similarly, the image processing system 1 is also called an abnormality detection system (or an appearance inspection system).

The image processing device 30 includes a learning model 410 (see FIG. 2). For example, a neural network model consisting of multiple layers is used as the learning model 410. Then, weighting coefficients (learning parameters) between multiple layers (input layer, multiple intermediate layers, output layer) in the neural network model are adjusted by a predetermined machine learning method (deep learning, etc.).

The learning model 410 is a model that takes a photographed image of an object (for example, a part such as a gear or screw) as input and outputs information about an abnormality in the photographed image (an abnormality of the object). In the machine learning (learning stage) of the learning model 410, for example, the presence or absence of an abnormality in each photographed image (for example, a minute scratch on the surface of the object) is used as a label (correct answer data). In other words, a combination (dataset) of a photographed image of the object and the presence or absence of an abnormality in the photographed image is used as teacher data (labeled data) in the machine learning of the learning model 410. The presence or absence of an abnormality may be the presence or absence of one type of abnormality, or may be the presence or absence of each of multiple types of abnormalities (for example, "minute scratches," "minute tears," "minute stains," etc.).

The learning model 410 after being trained by machine learning is also referred to as a trained model 420. Specifically, the learning parameters of the learning model 410 (learner) are adjusted using a predetermined machine learning method, and a trained learning model 410 (trained model 420) is generated (see FIG. 2).

In this embodiment, the image processing device 30 executes both the learning process of the learning model 410 (learning stage processing) and the inference process based on the learning model 410 (inference stage processing).

Figure 2 is a conceptual diagram showing the learning stage processing of the learning model 410. As shown in Figure 2, in the learning stage of the learning model 410, the image processing device 30 generates the learning model 410 using teacher data (a large number of teacher data) that takes each captured image as input and outputs information about anomalies in each captured image (such as the presence or absence of anomalies in the object). Note that the image processing device 30 is also a device that executes the learning stage processing (generation processing of the learning model 410), that is, a processing device that generates the learning model 410 (420), and is therefore also referred to as a learning model generation device.

Figure 3 is a conceptual diagram showing the processing of the inference stage based on the learning model 410. As shown in Figure 3, in the inference stage, the image processing device 30 detects abnormalities in the captured image of the object based on the learning model 410 that has been machine-learned using the teacher data described above. Specifically, the image processing device 30 inputs a captured image of the object (an object to be determined for the presence or absence of an abnormality) to the learning model 410, thereby determining the presence or absence of an abnormality in the object.

<1-2. Image processing device 30>
As shown in FIG. 1, the image processing device 30 includes a controller 31 (also referred to as a control unit), a storage unit 32, a communication unit 34, and an operation unit 35.

The controller 31 is a control device that is built into the image processing device 30 and controls the operation of the image processing device 30.

The controller 31 is configured as a computer system including one or more hardware processors (e.g., a central processing unit (CPU) and a graphics processing unit (GPU)). The controller 31 realizes various processes by executing a predetermined software program (hereinafter also simply referred to as a program) stored in a storage unit (a non-volatile storage unit such as a ROM and/or a hard disk) 32 in the CPU, etc. The program (more specifically, a group of program modules) may be recorded on a portable recording medium such as a USB memory, read from the recording medium, and installed in the image processing device 30. Alternatively, the program may be downloaded via a communication network, etc., and installed in the image processing device 30.

Specifically, the controller 31 executes processing related to the learning stage of the learning model 410 (see FIG. 2). Furthermore, the controller 31 executes processing related to the inference stage using the learning model 410 (see FIG. 3).

The storage unit 32 is configured with a storage device such as a hard disk drive (HDD) and/or a solid state drive (SSD). The storage unit 32 stores the learning model 410 (including learning parameters and programs related to the learning model) (and thus the learned model 420) and the like.

The communication unit 34 is capable of performing network communication via a network. In this network communication, various protocols such as TCP/IP (Transmission Control Protocol/Internet Protocol) are used. By using this network communication, the image processing device 30 can send and receive various data (teacher data (photographed image data, correct answer data, etc.)) to and from a desired destination (for example, the photographing device 20 or an information storage device (not shown)).

The operation unit 35 includes an operation input unit 35a that accepts operation input to the image processing device 30, and a display unit 35b that displays and outputs various information. A mouse and a keyboard are used as the operation input unit 35a, and a display (such as a liquid crystal display) is used as the display unit 35b. A touch panel that functions as both a part of the operation input unit 35a and a part of the display unit 35b may also be provided.

<1-3. Learning model according to this embodiment>
6 is a diagram showing a learning model 410 according to the first embodiment. The learning model 410 is constructed using a convolutional neural network (CNN) 200.

As shown in FIG. 6, the learning model 410 has a hierarchical structure in which multiple layers (layers) are hierarchically connected. Specifically, the learning model 410 includes an input layer 310, a feature extraction layer 320, a convolutional layer 340, a pooling layer 360, and an output layer 370. In particular, the learning model 410 includes a pooling layer 360 that performs GMP processing (described below) immediately after the convolutional layer 340, and does not include a fully connected layer after the pooling layer 360 (on the output side).

<Input Layer 310>
The input layer 310 is a layer that accepts an input image 210. Here, the input image 210 is a photographed image of an object (for example, a part such as a gear, or a finished product). For example, a color image (3 channels) having a pixel array (rectangular pixel array) with a width (horizontal) of W0 pixels and a height (vertical) of H0 pixels is input as the input image 210. In other words, the input image 210 is generated as voxel data of W0×H0×C0 (where C0=3).

Figure 24 is a schematic diagram showing an example of an input image 210. In the input image 210 in Figure 24, there is a tiny "flaw" (scratch) in the lower left part of the image (see the circle with the dashed line). Such an input image 210 is input to the input layer 310.

<Feature Extraction Layer 320>
6, the convolutional neural network 200 of the learning model 410 includes a feature extraction layer 320 following the input layer 310. The first feature map 230 is generated by processing the input image 210 through the feature extraction layer 320.

The feature extraction layer 320 is an intermediate layer that includes multiple convolutional layers and multiple pooling layers (max pooling and/or average pooling, etc.). In this convolutional neural network 200, multiple intermediate layers are provided between the input layer 310 and the output layer 370. The feature extraction layer 320 is a part of the multiple intermediate layers. As the feature extraction layer 320, for example, a part (feature extraction part) of various convolutional neural network configurations (VGG, ResNet, etc.) can be used.

For example, the feature extraction layer (13 convolutional layers and 5 pooling layers) (see FIG. 7) provided just before the fully connected layer (3 layers) in VGG16 is provided as feature extraction layer 320 in convolutional neural network 200. In FIG. 7, a part of the configuration of VGG16 (having 13 convolutional layers, 5 pooling layers, and 3 fully connected layers) (the feature extraction part (just before the fully connected layer)) is illustrated as an example of feature extraction layer 320. Note that activation functions and the like are omitted as appropriate in FIG. 7.

Alternatively, all (or a part) of the feature extraction layer provided in other convolutional neural networks such as ResNet (Residual Network) (residual network) may be provided as the feature extraction layer 320 in the convolutional neural network 200. ResNet is a convolutional neural network that includes adding up residuals between layers. The feature extraction layer in ResNet is composed of multiple residual blocks, etc., which are composed of a combination of a convolutional layer, an activation function, and a skip connection (shortcut connection). Here, in a general convolutional neural network, a fully connected layer or the like is provided after the feature extraction layer as a layer (also called a classification layer) that performs classification processing based on the features extracted in the feature extraction layer. All (or a part) of the feature extraction layer provided immediately before such a fully connected layer may be provided as the feature extraction layer 320 in the convolutional neural network 200.

The first feature map 230 is an intermediate feature map in the convolutional neural network 200 of the learning model 410. The first feature map 230 is generated as a feature map having multiple channels. The first feature map 230 is generated as three-dimensional array data (voxel data of W1×H1×C1) having C1 channels each composed of two-dimensional array data of a pixel array (rectangular pixel array) of width W1 pixels and height H1 pixels. The number of channels C1 of the first feature map 230 is, for example, 512. However, it is not limited to this and may be 256, 1024, etc.

Here, the feature extraction layer 320 is constructed by repeatedly arranging one or more convolution layers and one pooling layer. In each convolution layer, features in the image are extracted by a filter that performs convolution processing. In addition, in each pooling layer, a process (such as maximum pooling process) is performed to extract the maximum pixel value for each small pixel range (for example, a 2 x 2 pixel range), and the pixel size is reduced (for example, by 1/2 in both the vertical and horizontal directions) (the amount of information is condensed).

Then, the first feature map 230 is generated by performing the processing (convolution processing and pooling processing) by the feature extraction layer 320 on the input image 210. In this way, the first feature map 230 is generated by an intermediate layer including multiple convolution layers and multiple pooling layers provided after the input layer 310 in the convolutional neural network 200. According to this, image features (microscopic scratches, etc.) in the input image 210 are extracted in a state where their rough positions are maintained in the two-dimensional image of each channel in the first feature map 230. Note that in the pooling layer included in the feature extraction layer 320, a maximum pooling process (or an average pooling process) or the like is performed. In order to more appropriately avoid missing detection of microscopic scratches, etc., it is preferable to perform a maximum pooling process.

<Convolutional Layer 340>
Furthermore, in the learning model 410 in the first embodiment, the second feature map 250 is generated by performing a convolution process on the first feature map 230. In other words, the convolution layer 340 is provided next to the feature extraction layer 320, and the second feature map 250 is generated based on the first feature map 230. According to such a process, image features (such as minute scratches) in the input image 210 are extracted in a state where their rough positions are still maintained in the two-dimensional image of each channel in the second feature map 250. The convolution layer 340 is also expressed as being a part of the intermediate layer in the convolutional neural network 200.

The second feature map 250 is generated by integrating (by weighting, etc.) pixel values at the same position and/or nearby positions of each channel in the first feature map 230. The second feature map 250 is generated, for example, as three-dimensional array data (W2×H2×C2 voxel data) having C2 channels each made up of two-dimensional array data of pixel arrays (rectangular pixel arrays) (each) of width W2 pixels and height H2 pixels. The second feature map 250 is generated by using C2 filters (described below) of size N×N×C1.

However, in the first embodiment, the case where the second feature map 250 has a single channel (i.e., when C2 = 1) is exemplified. Also, the case where W2 = W1 and H2 = H1 is mainly exemplified. Note that the case where the second feature map 250 has multiple channels will be described in the second embodiment.

The values W0, H0, W1, H1, W2, H2, C0, C1, and C2 can be set to appropriate values depending on the configuration of the feature extraction layer 320. For example, W0=H0=512, W1=H1=16, W2=H2=16, C0=3, C1=512, and C2=1.

Figures 8 and 9 show how a second feature map 250 (here, a second feature map 250 having a single channel) is generated by a convolution process on a first feature map 230.

In the convolution process for the first feature map 230, for example, a filter (341 or 342, etc.) as shown in FIG. 8 or FIG. 9 is used to generate the second feature map 250.

Figure 9 shows how the second feature map 250 is generated by a convolution filter 342 with a size of NxN (for example, N=3). On the other hand, Figure 8 shows how the second feature map 250 is generated by a convolution filter (kernel) 341 with a size of 1x1. Note that the convolution filter 341 in Figure 8 can also be expressed as one aspect of the convolution filter 342 in Figure 9 (when N=1). However, for convenience, the explanation will be divided into two cases (when N is 1 and when N is 2 or more).

The size in the depth direction of both convolution filters 341 and 342 is the value C1 (the number of channels in the first feature map 230). In other words, filter 341 has a size (three-dimensional size) of 1x1xC1, and filter 342 has a size (three-dimensional size) of NxNxC1. Note that the value N is not limited to "3" and may be another value such as "5".

First, referring to FIG. 8, we will explain the process of generating the second feature map 250 using a convolution filter 341 with a size of 1×1 (plane size).

The convolution filter 341 in FIG. 8 has a size of 1×1×C1 in detail. The convolution filter 341 is a filter that performs weighted addition of pixel values at the same pixel position in multiple (C1) channels of the first feature map 230. Specifically, the convolution filter 341 is configured as a collection of C1 weighting coefficients (weighting coefficients for each channel) for the (i, j) pixel of each channel of the first feature map 230 in the channel direction. Note that the channel direction is the direction that connects the channels (the depth direction in FIG. 8) and is also referred to as the inter-channel direction, etc.

Then, a value D2(i,j) is obtained by applying filter 341 to a total of C1 pixel values D1(i,j,h) (h=1,...,C1) present at pixel positions (i,j) of each of the multiple channels of first feature map 230. Here, h is the channel number. Specifically, first, the product is calculated by multiplying pixel value D1(i,j,h) of the (i,j) pixel of the h-th channel of first feature map 230 by the corresponding weighting coefficient W(h) in convolution filter 341. Then, value D2(i,j) is calculated as the sum (sum of products) of the products for the C1 channels (see equation (1)).

The sum-of-products value D2(i,j) is calculated as the pixel value of the (i,j) pixel of the second feature map 250 (the first channel of the second feature map 250). In other words, the pixel values of the same position (i,j) of each channel of the first feature map 230 are integrated to calculate the pixel value of the (i,j) pixel of the second feature map 250. In short, the weighted average (or a constant multiple thereof) of the pixel values of the same position (i,j) of each channel of the first feature map 230 is calculated as the pixel value of the (i,j) pixel of the second feature map 250. The value D2(i,j) of the kth channel of the second feature map 250 is also expressed as the value D2(i,j,k). For example, the value D2(i,j) of the first channel of the second feature map 250 is also expressed as the value D2(i,j,1) (k=1 in the first embodiment).

In this way, a process is executed to calculate the pixel value of a certain pixel position (i, j) of the second feature map 250 (specifically, its first channel (k=1)) based on the convolution filter 341 and the pixel values of a certain pixel position (i, j) in the multiple channels of the first feature map 230. As a result, pixel values (multiple pixel values) at the same position in the multiple channels of the first feature map 230 are condensed into the pixel value of the corresponding pixel position in the second feature map 250.

Then, this process is similarly performed for each of the pixels at all other pixel positions in the first feature map 230. This causes the pixel values of all other pixel positions in the second feature map 250 (specifically, its single channel) to be calculated.

This type of convolution process condenses pixel information from each position (same position) across multiple channels into the pixel value for that position within a single new channel.

Next, with reference to FIG. 9, we will explain the process of generating the second feature map 250 using a convolution filter 342 with a size (plane size) of (N×N).

The convolution filter 342 in FIG. 9 specifically has a size (three-dimensional size) of N×N×C1. The convolution filter 342 is a filter that weights and adds pixel values at the same pixel position and its neighboring (surrounding) pixel positions in multiple (C1) channels of the first feature map 230.

The convolution filter 342 is configured as a collection of C1 weighting coefficients (weighting coefficients for each channel) for (N x N) pixels in a neighborhood centered on the (i, j) pixel of each channel of the first feature map 230 in the channel direction (depth direction in FIG. 9). Each weighting coefficient group for each channel is composed of (N x N) (e.g., 3 x 3) weighting coefficients (see also FIG. 11). Furthermore, the weighting coefficient group for each channel is composed of independent values for each channel. In other words, the values of the weighting coefficient group for a certain channel are fundamentally different from the values of the weighting coefficient groups for other channels.

Then, a value D2(i,j) is obtained by applying a filter 342 of the same size to a voxel range of size NxNxC1 (a total of (NxNxC1) pixel values that exist in the three-dimensional region indicated by the dashed rectangular parallelepiped in Figure 9) centered on each pixel position (i,j) of the multiple channels of the first feature map 230.

This filter processing can be divided into processing within a channel and processing between channels (see FIG. 10). Like FIG. 9, FIG. 10 is a diagram showing the generation process of the second feature map 250 using a convolution filter 342 of size (planar size) (N×N). However, the three-dimensional feature map and three-dimensional filter in FIG. 9 are broken down into channels (as a planar collection) in FIG. 10. The left-right direction in FIG. 10 corresponds to the depth direction (channel direction) in FIG. 9.

Specifically, using the convolution filter 342, first, intra-channel weighting processing is performed on the pixel values of (N×N) pixels (including the central (i, j) pixel) in the vicinity of the (i, j) pixel in the h-th channel of the first feature map 230. In other words, (planar) convolution processing is performed on each of the multiple channels of the first feature map 230. As a result, a provisional feature map (also called a provisional feature map) 245 is generated, as shown near the center in the horizontal direction of FIG. 10. The pixel values of the pixels in the vicinity range (dashed area) of the position (i, j) of the h-th channel of the first feature map 230 in FIG. 10 are convoluted to obtain the pixel value D3(i, j, h) of the pixel at the position (i, j) of the provisional feature map 245. The pixel value D3(i, j, h) is the pixel value of the (i, j) pixel in the h-th channel of the provisional feature map 245 (see also FIG. 10).

Then, a value D2(i,j) is obtained by summing C1 pixel values D3(i,j,h) (where h is the channel number, h=1,..,C1) for C1 channels of the provisional feature map 245 (see equation (2)). In other words, integration processing (summing processing) between channels is performed.

Such a value D2(i, j) is calculated as the pixel value of the (i, j) pixel in the second feature map 250 (first channel).

Here, the value D3(i,j,h) can also be expressed as the product of the in-plane component D4(i,j,h) and the inter-plane component W4(h). Taking this into consideration, the above formula (2) can be transformed into the following formula (3).

In other words, planar convolution processing is performed in each channel using the in-plane components D4(i,j,h) of the convolution filter 342 to obtain the values D4(i,j,h) of the provisional feature map 245. Furthermore, the pixel values D4(i,j,h) (where h is the channel number, h=1,..,C1) are multiplied by the corresponding weighting coefficient W4(h) (inter-plane component) in the convolution filter 342 to obtain the product. The products for the C1 channels are then summed up to obtain a value D2(i,j) (see equation (3)).

That is, pixel values D4(i,j,h) obtained by convolving pixel values in the same neighborhood centered on the same pixel position (i,j) in multiple channels of the provisional feature map 245 are weighted and added together to be calculated as the pixel value of the (i,j) pixel in the second feature map 250. In short, the weighted average (or a constant multiple of it) of the pixel values D4(i,j,h) is calculated as the pixel value of the (i,j) pixel in the second feature map 250.

Note that here, the convolution process is performed on the first feature map 230 after padding (zero padding, etc.) is performed to fill the outer peripheral region of the peripheral pixels of each channel of the first feature map 230 with an appropriate value (for example, zero (or the same value as the peripheral pixels)). This makes it possible to prevent the pixel size (W x H) of the feature map (second feature map 250) after the convolution process from becoming small. Such convolution process involving padding can realize a situation where W2 = W1 and H2 = H1. However, this is not limited to W2 = W1 and H2 = H1. For example, the planar size (W2 x H2) of the second feature map 250 may be smaller than the planar size (W1 x H1) of the first feature map 230. Note that in FIG. 11, for the sake of simplicity, the case of W2 = W1 = 6 and H2 = H1 = 6 is shown.

In this way, a process is executed to calculate the pixel value of a corresponding pixel position (i, j) in the second feature map 250 (specifically, the kth channel) based on the convolution filter 342 and the pixel values of a pixel position (i, j) and its neighboring positions in the multiple channels of the first feature map 230. As a result, pixel values (multiple pixel values) in the same range in the multiple channels of the first feature map 230 are condensed into the pixel value of the corresponding pixel position in the second feature map 250.

Then, this process is similarly performed for the pixels at all other pixel positions of each channel of the first feature map 230. This causes the pixel values of all other pixel positions of the second feature map 250 (specifically, its kth channel (k=1)) to be calculated.

This type of convolution process condenses pixel information near each position (near the same position) across multiple channels into the pixel value for that position within a single new channel.

As described above (as shown in FIG. 8 or FIG. 9), the second feature map 250 is generated by integrating pixel values at the same position and/or nearby positions in each channel in the first feature map 230. Specifically, a process of calculating pixel values at all pixel positions in the first channel of the second feature map 250 (a process of calculating pixel values in a channel) is executed once. This calculation process is performed using a convolution filter 341 or 342. The convolution filter 341 or 342 is a filter that performs weighted addition of pixel values at the same position and/or nearby positions in multiple (C1) channels of the first feature map 230. In this way, a predetermined number (1) of channels in the second feature map 250 are generated.

It is preferable that each value in the second feature map 250 is further normalized and corrected (updated) using an activation function or the like.

Pooling Layer 360 and Output Layer 370
Next, a global max pooling process is performed on the second feature map 250, and a score (output value) 270 of the learning model 410 is calculated (see FIG. 6). In other words, a pooling layer 360 is provided next to the convolution layer 340, and a score 270 (output value of the output layer 370) of the learning model 410 is generated. The pooling layer 360 is a layer that performs a global max pooling (GMP) process (hereinafter also referred to as a GMP process) on the second feature map 250. The pooling layer 360 is also referred to as a GMP layer. The pooling layer (GMP layer) 360, like the feature extraction layer 320 and the convolution layer 340, is also expressed as being a part of the intermediate layer in the convolution neural network 200.

Max Pooling is a process that extracts the maximum value (maximum pixel value) of multiple pixels corresponding to the kernel (filter) size as a feature value (output value). Max Pooling often extracts the maximum value of multiple pixels (e.g., four pixels) corresponding to a filter size (e.g., 2x2 size) that is generally smaller than the channel size.

In contrast, Global Max Pooling (GMP) processing is a type of Max Pooling processing, and more specifically, it is a Max Pooling processing that targets the "entire channel" (see FIG. 14). GMP processing (global max pooling processing) is a Max Pooling processing that extracts the maximum value from multiple pixels (all pixels in a channel) (e.g., 256 pixels) corresponding to a filter size (e.g., W2×H2=16×16) that is the same as the channel size.

GMP processing (global max pooling processing) is processing that extracts (for each channel) the maximum pixel value for each channel of the feature map to be processed (here, the second feature map 250). When GMP processing is performed on a feature map with a single channel, a single value (maximum value) is output. On the other hand, when GMP processing is performed on a feature map with multiple (e.g., two) channels, multiple values (maximum values for each channel) (for example, two values, the maximum value in the first channel and the maximum value in the second channel) are output.

When the GMP process is performed on the second feature map 250 having a single channel, the maximum value (max value) of the multiple (W2×H2) pixel values in the two-dimensional array data (W2×H2) of the single channel is output as the processing result of the GMP process (see FIG. 14). Note that FIG. 14 is a diagram showing the GMP process for one channel of the second feature map. In FIG. 14, for the sake of simplicity, the second feature map 250 is illustrated as having a size of 6×6. In the case of FIG. 14, that is, when W2=H2=6, the maximum value of 6*6=36 pixel values is output as the processing result of the GMP process for the second feature map 250 (first channel). Alternatively, for example, when W2=H2=16, the maximum value of 16*16=256 pixel values is output as the processing result of the GMP process for the second feature map 250 (first channel).

If each value in the second feature map 250 has already been normalized, the value resulting from the GMP processing (a value between 0 and 1) is output as is as the output value 270 of the convolutional neural network 200 (output layer 370) (see FIG. 12).

On the other hand, if the values in the second feature map 250 have not yet been normalized (for example, if each value is one of 8-bit values (0 to 255)), the processing result of the GMP processing is normalized using a predetermined normalization function (such as an activation function) (see FIG. 13). As a result, the processing result of the GMP processing is normalized to a value between 0 and 1. Note that the activation function may be a sigmoid function, a ReLU (Rectified Linear Unit) function, or the like.

The learning model 410 has the configuration described above, and when a photographed image (input image) 210 of an object is input, an output value indicating information regarding abnormalities in the object (presence or absence of abnormality, etc.) is output. As described below, in the learning stage, the normalized output value (output value of the learning model 410) is compared with the correct answer data, and machine learning of the learning model 410 is performed to reduce the difference between the two (difference value, etc.). In the inference stage, a determination is made as to whether the object is normal or abnormal based on the normalized output value (output value of the learning model 410).

Here, in the GMP layer 360, GMP processing is performed on the second feature map 250. According to this, in the learning model 410 (420) after learning, if there is even one pixel that is considered abnormal (a value close to "1") among the multiple (W2 x H2) pixel values of the second feature map 250, the value of that pixel (abnormal pixel) is output to the output value 270. Therefore, it is possible to prevent abnormal pixels from being overlooked.

<1-4. Learning stage processing>
Next, the learning stage processing of the learning model 410 will be described with reference to FIG.

Figure 4 is a flowchart showing the processing of the controller 31 in the learning stage. Figure 4 is also a diagram showing a method for generating a trained model. In this application, generating a trained model 420 means manufacturing (producing) the trained model 420, and the "method for generating a trained model" means the "method for manufacturing a trained model."

As shown in FIG. 4, in step S11, the controller 31 first acquires training data that takes each captured image as input and outputs information regarding abnormalities in each captured image. For example, training data (multiple training data) is acquired in which information indicating whether or not a "microscopic scratch" is present on the object in each captured image (e.g., abnormal "1"/normal "0") is associated with each captured image as correct answer data. In particular, it is preferable to prepare a very large amount of training data.

Then, in step S12, the controller 31 uses the training data acquired in step S11 to machine-train the learning model 410 as described above.

More specifically, as shown in FIG. 12 (and FIG. 13), the output value (normalized output value (value between "0" and "1")) from the learning model 410 is compared with the correct answer data (for example, "0" for normality and "1" for an abnormality). Then, machine learning of the learning model 410 is performed so as to reduce the difference between the two (such as a differential value). In other words, machine learning of the learning model 410 is performed using the value (output value 270) obtained by performing GMP processing on a single channel of the second feature map 250, and the learned model 420 is generated.

Specifically, the learning parameters of the learning model 410 (weighting coefficients and bias values of each filter (kernel), etc.) are optimized using a backpropagation method or the like. More specifically, the weighting coefficients and bias values of each filter (kernel) in the feature extraction layer 320, and the weighting coefficients and bias values of each filter (kernel) in the convolution layer 340, etc. are optimized.

By this process, a learning model 410 is generated that has learned the relationship between each captured image and information about anomalies in the captured image (e.g., the presence or absence of "microscopic scratches") (step S13). In other words, a trained learning model 410 (trained model 420) is generated.

As a result, as described below, it is possible to use the trained model 420 to estimate the presence or absence of abnormalities (such as "tiny scratches") in the subject (object) in a captured image based on the captured image of the object to be judged.

<1-5. Processing in the inference stage>
Next, the processing of the inference stage using the trained model 420 will be described with reference to FIG.

Figure 5 is a flowchart showing the processing of the controller 31 in the inference stage. Figure 5 also shows an anomaly detection method (such as an image processing method for anomaly detection) that uses a trained model.

First, in step S31, the controller 31 acquires a photographed image of the object. Specifically, the controller 31 acquires the photographed image (also referred to as the determination target image) captured by the photographing device 20 from the photographing device 20 via a network or the like.

In the next step S32, the controller 31 inputs the captured image acquired in step S31 into the learning model 410 (trained model 420). Then, the controller 31 acquires an output result from the trained model 420. The output result is an estimation result (inference result) regarding the presence or absence of an abnormality in the target object, etc.

When a value equal to or greater than the threshold value TH1 ("0.5") is output as the normalized output result (output value), it is determined that an abnormality exists in the object in the captured image. A normalized output result (output value) closer to "1" (abnormal) indicates that the possibility of an abnormality is high (in short, that an abnormality exists). Conversely, an output result closer to "0" (normal) indicates that the possibility of an abnormality is low (that an abnormality does not exist). In addition, the controller 31 also outputs the numerical value itself (or the converted value) related to the output result as the reliability of the determination result.

For example, when the output result value from the trained model 420 is "0.9", it is determined that a microscopic flaw exists and the reliability (reliability regarding the presence or absence of anomalies in the entire input image 210) is determined to be 90%. On the other hand, when the output result value from the trained model 420 is "0.2", it is determined that a microscopic flaw does not exist and the reliability is determined to be 80% (= (1-0.2) * 100 (%)).

Then, in step S33, the controller 31 displays a screen 510 on the display unit 35b, which shows the determination result regarding the presence or absence of an abnormality. The screen 510 includes information based on the output result obtained in step S32.

Figure 22 is a diagram showing an example of a screen 510. Figure 22 shows a screen 510 (also referred to as 511) on which the input image 210 and a heat map of the second feature map 250 are arranged in the left-right direction. The screen 510 is also a screen that displays information visualized based on the second feature map 250.

The input image 210 itself is displayed on the left side of the screen 511. Also, in the screen 511, information regarding the presence or absence of an abnormality is displayed in a display area 520 above the input image 210. In FIG. 22, the character string "Abnormal (Scratch)" is displayed. This indicates that an abnormality ("scratch") exists in the object of the captured image. In detail, if the reliability regarding the presence or absence of an abnormality for the entire input image 210 is equal to or greater than a threshold value (e.g., 50%), it is determined that "an abnormality exists." Note that, although the numerical value of the reliability is not displayed here, this is not limiting, and the numerical value of the reliability may also be displayed.

A heat map 530 of the second feature map 250 (information visualizing the second feature map 250) is displayed on the right side of the screen 511. Each pixel in the second feature map 250 has a pixel value that indicates the possibility of an abnormality being present at the position of that pixel. It can also be said that each pixel value indicates the reliability of the pixel in the second feature map 250 (the reliability of the presence or absence of an abnormality on a pixel-by-pixel basis).

In the heat map 530, the pixel value of each pixel in the second feature map 250 is divided into multiple stages, and the rectangle of each pixel is displayed by filling it with a color assigned to each stage. For example, pixels with a pixel value of 0.8 or more and 1.0 or less are displayed in red (dot-hatched area in FIG. 22), pixels with a pixel value of 0.5 or more and less than 0.8 are displayed in orange (diagonal-hatched area in FIG. 22), and pixels with a pixel value less than 0.5 are displayed in white (colorless and transparent). The operator can identify the possibility of an abnormality (presence or absence of an abnormality, etc.) at each pixel position by the color of each pixel in the heat map 530.

The heat map 530 of the second feature map 250 is displayed (with its size adjusted) to have the same size as the input image 210.

The smallest rectangle of the heat map 530 corresponds to one pixel of the second feature map 250. Furthermore, the smallest rectangle has a size equivalent to (W0/W2) pixels of the input image 210 on each side. For example, when W0 = H0 = 512 and W2 = H2 = 16, one side of the smallest rectangle of the heat map 530 of the second feature map 250 has a size equivalent to 32 pixels (= 512/16) of the input image 210.

The display information on the screen 510 (information visualized based on the second feature map 250) allows the operator to easily ascertain the general location of the abnormality in the input image 210. In particular, by comparing the input image 210 with the heat map 530 (second feature map 250), the operator can easily grasp where the abnormality exists in the input image 210. More specifically, it can be seen that the abnormality in the input image 210 exists near a position corresponding to a red pixel position (or an orange pixel position) in the heat map 530.

Note that here, the input image 210 and the second feature map 250 (specifically, its heat map 530) are displayed in parallel arrangement, but this is not limited to this.

For example, information (such as a heat map 530) that visualizes the second feature map 250 may be displayed superimposed on the input image (captured image) 210. FIG. 23 is a diagram showing such a screen 510 (512) or the like. In the screen 512 on the right side of FIG. 23, a heat map 530 (see the lower left in FIG. 23) having the same size as the input image 210 (see the upper left in FIG. 23) is displayed superimposed on the input image 210. In such a superimposed display, it is preferable that the heat map 530 is displayed in a semi-transparent color.

Alternatively, the display state of the two images 210, 530 superimposed as shown in FIG. 23 (see screen 510) and the display state of the input image 210 alone may be switched alternately. In particular, the input image 210 may be constantly displayed, and the display of the heat map 530 may be switched on and off at a predetermined time interval (such as every second). That is, the display state of only the input image 210 and the display state of the heat map 530 superimposed on the input image 210 may be alternately repeated. Alternatively, the input image 210 and the heat map 530 having the same size may be alternately displayed in the same position.

In addition, here, pixels in the second feature map 250 that are equal to or greater than a predetermined value are filled with a predetermined color, but this is not limited to this. For example, the information contained in the second feature map 250 (such as the degree of abnormality at each pixel position) may be visualized and displayed by surrounding an area in the second feature map 250 that has pixels that are equal to or greater than a predetermined value with a line.

As described above, in the inference stage, an abnormality in an object is detected by inputting a captured image of the object (input image 210) into the learning model 410 (420).

<1-6. Comparative Example>
Next, a learning model 910 according to a comparative example will be described.

FIG. 25 is a diagram showing a learning model 910 according to a comparative example. This learning model 910 has a similar configuration to the learning model 410 according to the first embodiment up to the generation of the first feature map 230. The learning model 910 is a convolutional neural network (CNN) having a general configuration.

In learning model 910, after the generation of the first feature map 230, processing is performed differently from that in learning model 410.

Specifically, in the learning model 910, a global average pooling (GAP) process (hereinafter also referred to as GAP process) is performed on the first feature map 230. In other words, the feature extraction layer 320 is followed by a pooling layer 940 that performs GAP process.

GAP processing is an average pooling process that targets "all channels." GAP processing is a process that extracts (for each channel) the average pixel value for each channel of the target feature map (here, the first feature map 230). The average value of (W1 x H1) pixels (1024 pixels when W1 = H1 = 32) in each channel is extracted for each channel. If the first feature map 230 has C1 channels (for example, 512 channels), C1 average values (average values for each channel) are extracted.

The C1 average values are then arranged one-dimensionally (converted into one-dimensional vector format) (see one-dimensional array 950) and passed to the fully connected layer 960. The fully connected layer 960 has a multi-layer structure. In other words, multiple fully connected layers 960 are provided. Furthermore, an output layer 970 is provided next to the fully connected layer 960. The output layer 970 has nodes according to the number of classification destinations, and an output value is calculated for each node.

1-7. Effects of the embodiment
In the above comparative example, the GAP process is performed on the first feature map 230. GAP (global average pooling) process is useful for capturing the overall features within a channel.

However, if GAP processing is used to detect abnormalities (especially minute scratches, etc.) in a captured image, there is a high possibility that information about minute scratches, etc. in each channel of the first feature map 230 will be lost. This is because the pixel values of multiple pixels, including both abnormal and normal pixels, are averaged. In particular, if there are a small number of abnormal pixels among a large number of normal pixels, it is very likely that the information about the small number of abnormal pixels will be lost by the averaging process. Therefore, there is a possibility that minute abnormalities (such as scratches) in the image will be overlooked. For example, if GAP processing is performed on the second feature map 250 in FIG. 14, the output value will be a very small value (0.039 (= (1 + 0.4) / 36)). As a result, it will be determined that there is no abnormality.

On the other hand, in the first embodiment, as shown in FIG. 6 and FIG. 14, GMP (global maximum pooling) processing is performed on the second feature map 250. According to the GMP processing, if there is even one abnormal pixel among the multiple ((W2×H2) pixels) in one channel in the second feature map 250, the value of the abnormal pixel is output as the output value 270. For example, if there is even one pixel among all 256 pixels in one channel in the second feature map 250 that has a pixel value that indicates an abnormality (for example, a pixel value close to "1" in a normalized state), the pixel value close to "1" is output as the output value 270. In other words, the abnormal information remains as the output value 270 (the maximum value close to "1" is output). Therefore, it is possible to prevent abnormal pixels from being overlooked.

In this way, it is possible to detect even the smallest defects, such as scratches, without missing them.

Here, if GMP processing is used, there is a possibility of overreacting to minute scratches (picking up noise, etc.). However, when detecting abnormalities such as minute scratches, it is preferable to avoid missing an abnormality rather than overdetecting an abnormality (detecting something normal as abnormal). In other words, it is preferable to have an abnormality overdetected by GMP processing rather than missing an abnormality (due to the inclusion of GAP processing, etc.).

The above-mentioned learning model 410 is machine-trained using values obtained by performing global max pooling (GMP) processing on the second feature map 250, so that it is possible to prevent abnormality information in the second feature map 250 from being lost due to averaging processing. Therefore, compared to the case where averaging processing is performed on the second feature map 250, it is possible to suppress overlooking minute abnormalities in the image and detect abnormalities more accurately.

In the above embodiment, a fully connected layer is not provided after the pooling layer (GMP layer) 360 that performs GMP processing (on the output side or in the subsequent stage). In other words, the learning model 410 does not have a fully connected layer after the GMP layer 360 (in subsequent processing). Here, the fully connected layer is a layer in which a process is performed to output the inner product (or a value obtained by further adding a bias to the inner product) of an input value vector in which all input values are arranged one-dimensionally and a weighting vector in which weighting coefficients for each input value are arranged one-dimensionally as an output value to each output side node. The inner product is also expressed as a product-sum value obtained by adding (adding) values obtained by multiplying each of the input values by a weight (weighting coefficient). Note that the convolution layer differs from the fully connected layer in that the convolution layer is a layer in which a product-sum value (inner product) is obtained for some of the input values (only pixel values in the vicinity of a certain pixel position) among all the input values.

On the other hand, if a fully connected layer is placed after the GMP layer 360, optimization may be performed in the fully connected layer during machine learning. In that case, the immediately preceding position in the second feature map 250 and the abnormal position in the input image 210 may no longer be extracted in correspondence with each other. In other words, the pixel value distribution in the second feature map 250 may no longer have a direct correlation with the abnormal position in the input image 210.

In contrast, since no fully connected layer is provided after the pooling layer 360 that performs GMP processing, it is highly likely that the abnormality positions in the second feature map 250 maintain a correspondence with the abnormality positions in the input image 210. Therefore, it is possible to accurately extract rough information about the abnormality positions in the input image 210.

In other words, in the learning model 410 of the above embodiment, the value (maximum value in the channel of the second feature map 250) obtained by performing the GMP process on the second feature map 250 is compared with the correct answer data, and the machine learning of the learning model 410 is executed. Alternatively, the converted value (a value normalized by an activation function (e.g., a sigmoid function) or the like) obtained by converting the maximum value without multiplying the maximum value by the learning parameter is compared with the correct answer data, and the machine learning of the learning model 410 is executed. In other words, the output value output without multiplying (without intervening) the maximum value obtained by performing the GMP process on the second feature map 250 by the learning parameter is compared with the correct answer data, and the machine learning of the learning model 410 is executed. Therefore, it is highly likely that the abnormality position in the second feature map 250 maintains a correspondence relationship with the abnormality position in the input image 210. Therefore, it is possible to accurately extract rough abnormality position information in the input image 210.

In short, since the output layer 370 is provided immediately after the GMP layer 360, the output value from the GMP layer 360 is output almost as is (either as is or converted by an activation function or the like) as the output value of the learning model 410. This makes it possible to maintain a correspondence with the anomaly position in the input image 210 in the second feature map 250 immediately before the GMP layer 360.

As described above, the image processing device 30 displays the visualized information based on the second feature map 250 on the display unit 35b. Therefore, the user (operator) can easily grasp the rough location of the abnormality in the input image 210.

<2. Second embodiment>
In the above-described first embodiment, the second feature map 250 has a single channel, but is not limited thereto. For example, the second feature map 250 may have a plurality of channels. In the second embodiment, such an aspect will be described.

Figure 15 is a conceptual diagram showing a learning model 410 (also referred to as 410B) according to the second embodiment. As shown in Figure 15, the second feature map 250 of the learning model 410B has multiple (C2) channels.

In the first embodiment, as shown in FIG. 8 or FIG. 9, a second feature map 250 having only a single channel is generated using one convolution filter 341 of size 1×1×C1 or one convolution filter 342 of size N×N×C1.

On the other hand, in this second embodiment, as shown in FIG. 16 or FIG. 17, a second feature map 250 having C2 (multiple) channels is generated using C2 convolution filters 341 of size 1×1×C1 or C2 convolution filters 342 of size N×N×C1.

First, referring to FIG. 16, we will explain the process of generating a second feature map 250 having C2 channels using a convolution filter 341 of size 1×1×C1.

The kth convolution filter 341 (where k = 1, ..., C2) of the C2 1x1xC1 size convolution filters 341 is a filter for calculating pixel values at each pixel position of the kth channel of the second feature map 250. The kth convolution filter 341 is a filter that performs weighted addition of pixel values at the same position in multiple (C1) channels of the first feature map 230 (a filter that integrates pixel values at the same position). Note that the C2 convolution filters 341 are basically composed of mutually different values.

The kth channel of the second feature map 250 is generated by the kth convolution filter 341 in the same manner as in the first embodiment. In detail, a process is performed to calculate a pixel value of a corresponding pixel position (i, j) of the second feature map 250 (specifically, the kth channel) based on the kth convolution filter 341 and the pixel value of a certain pixel position (i, j) in the multiple channels of the first feature map 230. Then, such a process is similarly performed for each of the pixels at all other pixel positions of the first feature map 230. As a result, pixel values of all other pixel positions of the second feature map 250 (specifically, the kth channel) are calculated.

More specifically, first, the pixel values of all pixel positions in the first channel of the second feature map 250 are calculated using the first convolution filter 341 in the same manner as in the first embodiment.

Then, the pixel values of all pixel positions in the second channel of the second feature map 250 are calculated in a similar manner using the second convolution filter 341.

Then, the same operation (the process of generating the kth channel of the second feature map 250 using the remaining kth convolution filter 341) is performed while incrementing the value k until the value k reaches C2. This generates the second feature map 250 having the C2 channel.

In this manner, the process of calculating the pixel values of each pixel position (pixel values of all pixel positions) in the kth channel of the second feature map 250 using the kth convolution filter 341 is executed a predetermined number (C2 times) (k=1, ..., C2). This generates a predetermined number (C2) of channels in the second feature map 250.

Next, referring to FIG. 17, we will explain the process of generating a second feature map 250 having C2 channels using a convolution filter 342 of size N×N×C1.

The kth convolution filter 342 of the C2 convolution filters 342 of size NxNxC1 is a filter for calculating pixel values at each pixel position of the kth channel of the second feature map 250. The kth convolution filter 342 is a filter that performs weighted addition of pixel values at the same position and its neighboring positions in multiple (C1) channels of the first feature map 230 (a filter that integrates pixel values at the same position and its neighboring positions). Note that the C2 convolution filters 342 are basically composed of different values from each other.

In FIG. 17, the k-th channel of the second feature map 250 is generated by the k-th convolution filter 342 in the same manner as in the first embodiment (see FIG. 9). Also, compared to FIG. 16, the k-th channel of the second feature map 250 is generated by using the k-th convolution filter 342 instead of the k-th convolution filter 341.

In detail, a process is performed in which the kth convolution filter 342 calculates a pixel value at a corresponding pixel position (i, j) in the second feature map 250 (specifically, the kth channel) based on a pixel position (i, j) and the pixel values of its neighboring pixel positions in the multiple channels of the first feature map 230. Then, the same process is similarly performed for each of the pixels at all other pixel positions in the first feature map 230. In this way, pixel values at all other pixel positions in the second feature map 250 (specifically, the kth channel) are calculated.

By repeatedly performing this process using C2 convolution filters 342, pixel values are calculated for all pixel positions in the channels for all C2 channels of the second feature map 250. More specifically, the process of generating the kth channel of the second feature map 250 using the kth convolution filter 342 is performed C2 times while incrementing the value k until the value k reaches "C2" from "1". This generates the second feature map 250 having C2 channels.

In this manner, the second feature map 250 having C2 (multiple) channels is generated. Here, the second feature map 250 has two or more channels, which is a reduced number of channels compared to the number of channels C1 of the first feature map 230. That is, the value C2 is smaller than the value C1 (the number of channels of the first feature map 230). For example, C2=2. However, this is not limited to this, and the value C2 may be 3 or more. Furthermore, the value C2 may be the same as the value C1, or may be larger than the value C1.

Figure 18 shows how a second feature map 250 (C2=2) with two channels is output.

Here, GMP processing is performed on the second feature map 250 (second feature map 250 having two channels) with a size of W2×H2×2. The GMP processing extracts the maximum pixel value for each of the two channels 251, 252 in the second feature map 250. Specifically, the maximum value in the first channel 251 of the second feature map 250 is obtained as the output of the first channel 251, and the maximum value in the second channel 252 of the second feature map 250 is obtained as the output of the second channel 252. Furthermore, the output for each channel is normalized by an activation function (for example, a sigmoid function) or the like, converted to a value between "0" and "1", and output from the output layer 370 of the learning model 410.

In the learning stage, as shown in FIG. 18, a comparison with correct answer data is made. Here, the correct answer data of the first channel 251 and the correct answer data of the second channel 252 are different types of correct answer data. For example, the correct answer data of the first channel 251 is correct answer data regarding the presence or absence of "microscopic scratches," and the correct answer data of the second channel 252 is correct answer data regarding the presence or absence of "microscopic tears." However, this is not limited to this, and the correct answer data of each channel may be other types of correct answer data (for example, "microscopic stains," etc.).

By performing machine learning based on such correct answer data, the output value of the first channel 251 is trained to output a value indicating the presence or absence of "microscopic scratches," and the output value of the second channel 252 is trained to output a value indicating the presence or absence of "microscopic tears."

The second embodiment can achieve the same effect as the first embodiment. In the machine learning of the learning model 410 of the second embodiment, the multiple maximum values obtained for each channel by performing GMP processing on the second feature map 250 having two or more channels are not averaged. Specifically, each of the multiple maximum values is converted (by an activation function or the like) without averaging the multiple maximum values, and each converted value (or the maximum value itself) is compared with the correct answer data. In this way, multiple types of feature amounts can be extracted without averaging, so that multiple types of abnormalities (such as minute scratches, minute tears, and minute stains) can be detected individually and appropriately.

In addition, here, the output values (multiple output values) for each channel by the GMP processing are normalized, but this is not limited to this. For example, as shown in FIG. 19, the multiple output values for each channel by the GMP processing may be normalized together by a softmax function. In detail, the multiple output values for each channel by the GMP processing may be further processed by a softmax function, and the processing result may be compared with the correct answer data to perform machine learning. In this case, an output suitable for determining which of multiple types of abnormalities (for example, multiple types of abnormalities including "micro scratches", "micro tears", and "micro stains") has occurred is obtained. In other words, the possibility of occurrence of each of multiple types of abnormalities (for example, abnormalities including "micro scratches", "micro tears", and "micro stains") can be relatively determined.

<3. Third embodiment>
In each of the above embodiments, the GMP process is applied to all channels of the second feature map 250, but is not limited to this. For example, the GMP process may be applied to only some of all C2 channels of the second feature map 250. In the third embodiment, such an aspect will be described.

Figure 20 is a diagram showing a learning model 410 (also referred to as 410C) according to the third embodiment. In the learning model 410C, a second feature map 250 having a C1 channel is generated, similar to the learning model 410 (410B) according to the second embodiment.

However, thereafter, GMP processing is applied to only a portion of all C2 channels in the second feature map 250, and GAP processing is applied to the remaining C2 channels.

Specifically, as shown in FIG. 21, for example, when the second feature map 250 has a total of five channels (i.e., when C2=5), GMP processing is performed on the first to third channels, and GAP processing is performed on the fourth to fifth channels. In other words, a GMP layer 360 is provided for the first to third channels, and a GAP layer 369 is provided for the fourth to fifth channels.

In the learning stage, machine learning is performed using values obtained by performing GMP processing on multiple channels from the first channel to the third channel. Specifically, machine learning is performed to reduce the difference between each output after GMP processing on the first channel to the third channel and the correct answer data regarding the presence or absence of minute abnormalities (for example, minute scratches, minute tears, minute stains) in the input image 210. This generates a learning model 410 (trained model 420) that can accurately determine the presence or absence of each type of minute abnormality.

Machine learning is also performed using values obtained by performing GAP processing on multiple channels from the fourth channel to the fifth channel. Specifically, machine learning is performed to reduce the difference between each output after GAP processing on the fourth channel to the fifth channel and the correct answer data regarding the presence or absence of major abnormalities (e.g., large stains, overall distortion, etc.) in the input image 210. This generates a learning model 410 (trained model 420) that can accurately determine the presence or absence of each type of major abnormality (anomalies that span a relatively wide area in the image).

On the other hand, in the inference stage, the presence or absence of minute abnormalities (for example, minute scratches, minute tears, minute stains) in the input image 210 can be accurately estimated using the outputs after GMP processing for the first to third channels. Also, the presence or absence of major abnormalities (abnormalities covering a relatively wide area within the image) in the input image 210 can be accurately estimated using the outputs after GAP processing for the fourth to fifth channels.

4. Modifications, etc.
Although the embodiment of the present invention has been described above, the present invention is not limited to the above-described contents.

For example, in the above embodiment, the image processing device 30 executes both the learning process and the inference process. However, this is not limited to the above, and the learning process and the inference process may be realized by separate devices (30A, 30B). In this case, the device 30B that executes the inference process may obtain, for example, information (learned learning parameters, etc.) related to the trained model 420 generated by the device 30A that executed the learning process from the device 30A, and use the trained model 420.

1 Image processing system 20 Photographing device 30 Image processing device (learning model generating device)
35b Display unit 200 Convolutional neural network 210 Input image 230 First feature map 245 Temporary feature map 250 Second feature map 270 Score (output value)
310 Input layer 320 Feature extraction layer 340 Convolution layer 341, 342 Filter 360 Pooling layer (GMP layer)
370 Output layer 410, 420 Learning model 530 Heat map

Claims (17)

a control unit that detects an abnormality in a photographed image of an object based on a learning model that is machine-learned using teacher data in which each photographed image is input and information regarding an abnormality in each of the photographed images is output;
Equipped with
the learning model is a model that is machine-learned by generating a second feature map by integrating pixel values at the same position and/or adjacent positions of each channel in a first feature map , the second feature map being an intermediate feature map in a convolutional neural network and having a plurality of channels, and performing a global max pooling process on the second feature map;
The image processing device is characterized in that the control unit detects abnormalities in the object by inputting the captured image of the object to the learning model. The image processing device according to claim 1, characterized in that the first feature map is generated by an intermediate layer including multiple convolution layers and multiple pooling layers provided after an input layer in the convolutional neural network. The image processing device according to claim 1 or 2, characterized in that the convolutional neural network does not have a fully connected layer subsequent to a pooling layer that performs the global max pooling process on the second feature map. The image processing device according to claim 3, characterized in that the learning model is a machine-learned model that is machine-learned by comparing the maximum value obtained by performing the global max pooling process on the second feature map, or a transformed value obtained by transforming the maximum value without multiplying the maximum value by a learning parameter, with correct answer data. The image processing device according to claim 3 or 4, characterized in that the learning model is a machine-learned model obtained by performing the global max pooling process on the second feature map having two or more channels, converting each of the multiple maximum values obtained for each channel without averaging the multiple maximum values, and comparing the converted values with correct answer data. The image processing device according to any one of claims 1 to 5, characterized in that the process of generating the second feature map includes a process of integrating pixel values at the same pixel position in multiple channels of the first feature map using a 1x1 convolution filter that weights and adds them, and calculating a pixel value at each pixel position in the channel of the second feature map. The image processing device according to any one of claims 1 to 5, characterized in that the process of generating the second feature map includes a process of integrating pixel values at the same pixel position in multiple channels of a provisional feature map after convolution processing for each of the multiple channels of the first feature map by weighted addition, and calculating a pixel value at each pixel position in the channel of the second feature map. The image processing device according to any one of claims 1 to 5, characterized in that the process of generating the second feature map includes a process of calculating pixel values at each pixel position in one channel of the second feature map a predetermined number of times using a convolution filter that weights and adds pixel values obtained by integrating pixel values at the same position and/or adjacent positions in multiple channels of the first feature map, thereby generating a predetermined number of channels in the second feature map. the second feature map has two or more channels that are reduced in number from the number of channels of the first feature map;
9. The image processing device according to claim 1, wherein the learning model is a model that is machine-learned using values obtained by performing the global max pooling process on at least a portion of the two or more channels. the second feature map is a feature map having a single channel;
The image processing device according to claim 1 , wherein the learning model is a model machine-learned using a value obtained by performing the global max pooling process on the single channel. The image processing device according to any one of claims 1 to 10, characterized in that the control unit displays the information visualized based on the second feature map on a predetermined display unit. a) detecting anomalies in a photographed image of an object based on a learning model machine-learned using teacher data in which each photographed image is input and information regarding anomalies in each of the photographed images is output;
Equipped with
the learning model is a model that is machine-learned by generating a second feature map by integrating pixel values at the same position and/or adjacent positions of each channel in a first feature map , the second feature map being an intermediate feature map in a convolutional neural network and having a plurality of channels, and performing a global max pooling process on the second feature map;
An image processing method characterized in that in the step a), an abnormality in the object is detected by inputting the captured image of the object to the learning model. a) detecting anomalies in a photographed image of an object based on a learning model machine-learned using teacher data in which each photographed image is input and information regarding anomalies in each of the photographed images is output;
A program for causing a computer to execute the above,
the learning model is a model that is machine-learned by generating a second feature map by integrating pixel values at the same position and/or adjacent positions of each channel in a first feature map , the second feature map being an intermediate feature map in a convolutional neural network and having a plurality of channels, and performing a global max pooling process on the second feature map;
A program characterized in that in the step a), an abnormality in the object is detected by inputting the captured image of the object to the learning model. A learning model generation device that receives a captured image and generates a learning model that outputs information about an abnormality in the captured image,
a control unit that generates a second feature map by integrating pixel values at the same position and/or adjacent positions of each channel in a first feature map, the second feature map being an intermediate feature map in a convolutional neural network of the learning model and having a plurality of channels, based on the first feature map, and executes machine learning in the learning model using a value obtained by performing a global max pooling process on the second feature map;
A learning model generation device comprising: A learning model production method for producing a learning model that inputs a captured image and outputs information about an anomaly in the captured image, comprising:
a) generating a second feature map by integrating pixel values at the same position and/or adjacent positions of each channel in a first feature map, the second feature map being an intermediate feature map in a convolutional neural network of the learning model and having a plurality of channels, based on the first feature map, and performing machine learning in the learning model using values obtained by performing a global max pooling process on the second feature map;
A learning model production method comprising: A trained model that has been machine-learned to output information regarding an anomaly in an inference target image in response to an input of the inference target image,
A process of generating a first feature map for the inference target image, the first feature map being an intermediate feature map in a convolutional neural network and having a plurality of channels;
A process of generating a second feature map by integrating pixel values at the same position and/or adjacent positions of each channel in the first feature map based on the first feature map;
a process of outputting information related to the anomaly based on a value obtained by performing a global max pooling process on the second feature map;
A trained model for running the above on a computer. a) generating a second feature map by integrating pixel values at the same position and/or adjacent positions of each channel in a first feature map, the second feature map being an intermediate feature map having a plurality of channels in a convolutional neural network of a learning model that inputs a captured image and outputs information about anomalies in the captured image, and performing machine learning in the learning model using values obtained by performing a global max pooling process on the second feature map;
A program for causing a computer to execute the following.

Images