JP7291914B2 - Processing method and processing equipment using the same - Google Patents

Description

The present disclosure relates to processing technology, and more particularly to a processing method for executing processing on an image and a processing apparatus using the same.

In order to detect anomalies in products, signals from sensors attached to products are subjected to frequency analysis on the time axis by wavelet transform, and neural network processing is performed on correlation diagrams including frequency analysis results. (See Patent Document 1, for example).

Japanese Unexamined Patent Application Publication No. 2018-91640

If the sensor cannot be attached to the product, a microphone is used to detect the sound emitted from the product that has an abnormality such as a malfunction. When using a microphone, environmental sounds such as noise occurring around the product are also detected. The presence of environmental sounds degrades the detection accuracy of sounds emitted from products.

The present disclosure has been made in view of such circumstances, and an object of the present disclosure is to provide a technique for suppressing deterioration in accuracy of processing even when environmental sounds are present.

In order to solve the above problems, a processing device according to one aspect of the present disclosure includes a first input unit for inputting inspection information obtained from an object to be inspected and indicating a change in sound over time; A first conversion unit that converts inspection information input to the first input unit into an inspection image that indicates temporal changes in frequency components, and inputs environmental sound information that indicates temporal changes in environmental sounds when obtaining inspection information. a second input unit that converts the environmental sound information input to the second input unit into an environmental sound image that shows temporal changes in frequency components; and an environmental sound image that is converted by the second conversion unit. a processing unit for inputting the inspection image converted by the first conversion unit into a trained neural network, and then performing neural network processing on the result of merging the environmental sound image and the inspection image ; and an output unit that outputs an inspection result of the object to be inspected as a processing result of the processing unit.
A processing device according to another aspect of the present disclosure includes a first input unit for inputting inspection information obtained from an object to be inspected and indicating a change in sound over time; a first conversion unit that converts inspection information into an inspection image that indicates temporal changes in frequency components; a second input unit that inputs environmental sound information that indicates temporal changes in environmental sounds when obtaining the examination information; A second conversion unit that converts the environmental sound information input to the second input unit into an environmental sound image showing temporal changes in frequency components, and the environmental sound image converted by the second conversion unit and converted by the first conversion unit. After inputting the obtained inspection image into a trained neural network, processing is performed to combine the results of executing convolution layer processing on the environmental sound image and the inspection image, and the result of the combination output by the combining processing is A processing unit for executing neural network processing including convolutional layer processing, and an output unit for outputting an inspection result of an object to be inspected as a processing result of the processing unit.

Another aspect of the present disclosure is also a processing apparatus. This apparatus includes a first input unit for inputting inspection information obtained from an object to be inspected and indicating a change in sound over time; After inputting the second input unit for inputting the environmental sound information, the environmental sound information input to the second input unit, and the inspection information input to the first input unit to the trained neural network, the environmental sound information and an output unit for outputting an inspection result of an object to be inspected as a processing result of the processing unit.
A processing device according to another aspect of the present disclosure includes a first input unit for inputting inspection information obtained from an object to be inspected and indicating a change in sound over time; A second input unit for inputting environmental sound information indicating changes in the environmental sound over time, the environmental sound information input to the second input unit, and the inspection information input to the first input unit are input to a trained neural network. Then, a neural network that combines the results of executing convolutional layer processing on environmental sound information and inspection information, and includes convolutional layer processing on the combined results output by the combining processing. and an output unit for outputting an inspection result of an object to be inspected as a processing result of the processing unit.

Yet another aspect of the present disclosure is a processing method. This method comprises the steps of: inputting inspection information obtained from an object to be inspected and indicating temporal changes in sound; and converting the input inspection information into an inspection image indicating temporal changes in frequency components. a step of converting; a step of inputting environmental sound information indicating temporal changes in environmental sounds when acquiring inspection information; and converting the input environmental sound information into an environmental sound image indicating temporal changes in frequency components. a step of inputting the converted environmental sound image and the converted inspection image to a trained neural network, and performing neural network processing on the result of combining the environmental sound image and the inspection image. and outputting an inspection result of an object to be inspected as a processing result of the neural network.
A processing method according to another aspect of the present disclosure includes a step of inputting inspection information obtained from an object to be inspected and indicating a change in sound over time; a step of converting into an inspection image showing a change; a step of inputting environmental sound information showing a time change of the environmental sound when acquiring the inspection information; and inputting the converted environmental sound image and the converted inspection image to a trained neural network, and then performing convolution layer processing on the environmental sound image and the inspection image. a step of performing a process of combining the results obtained by the combining process, performing neural network processing including convolutional layer processing on the combined result output by the combining process; and outputting the inspection result.

Yet another aspect of the present disclosure is also a processing method. This method includes the steps of: inputting inspection information obtained from an object to be inspected and indicating a change in sound over time; a step of inputting information; inputting the input environmental sound information and the input inspection information to a trained neural network; and outputting an inspection result of an object to be inspected as a processing result of the neural network.
A processing method according to another aspect of the present disclosure includes a step of inputting inspection information obtained from an object to be inspected and indicating a change in sound over time; A step of inputting environmental sound information showing a change over time, and inputting the input environmental sound information and the input inspection information into a trained neural network, and then performing a convolution layer on the environmental sound information and the inspection information. a step of performing a process of combining the results of executing the processes of, performing a neural network process including a convolutional layer process on the result of the combination output by the combining process; and outputting inspection results of the object of interest.

It should be noted that any combination of the above-described components and expressions of the present disclosure converted between methods, devices, systems, computer programs, recording media recording computer programs, etc. are also effective as aspects of the present disclosure. be.

According to the present disclosure, it is possible to suppress deterioration in accuracy of processing even when environmental sounds are present.

1 is a diagram showing a configuration of a processing apparatus according to Example 1; FIG. FIGS. 2(a) and 2(b) are diagrams showing an overview of the processing of the conversion unit in FIG. 1. FIG. 3(a) to 3(c) are diagrams showing an overview of the processing of the conversion unit in FIG. 1. FIG. 4(a)-(c) are diagrams showing images used in the processing unit of FIG. 1. FIG. 5A to 5C are diagrams showing the configuration of the processing unit in FIG. 1. FIG. 2 is a diagram showing an overview of processing in a processing unit in FIG. 1; FIG. It is a figure which shows the structure of the process part which concerns on a 1st modification. FIG. 13 is a diagram showing the configuration of a processing unit according to a third modified example; FIGS. 9A to 9C are diagrams showing the configuration of a processing section according to the fourth modification. FIG. 10 is a diagram illustrating the configuration of a processing apparatus according to Example 2; FIG. 10 is a diagram illustrating the configuration of a processing apparatus according to Example 3;

(Example 1)
Before specifically describing the embodiments of the present disclosure, an outline of the present embodiments will be described. Embodiment 1 relates to a processing apparatus for determining whether or not a product such as a rotating device such as a motor has an abnormality. Hitherto, in order to determine whether or not a product contains abnormalities, all products have been manually inspected for abnormal motor noise. However, since it is a sensory test, it is difficult to reflect customer requests, such as variations in judgment criteria depending on the person, variations in judgment time depending on the person, and variations due to the condition of the person (physical condition, etc.). There were also difficulties with inspection records. In other words, there is no objective and measurable pass/fail criteria. Therefore, labor saving and rationalization and quality stabilization are required by automating abnormal noise identification.

Also, an accelerometer is directly connected to the product by a magnet or the like to detect the sound generated by the product. Under such circumstances, the influence of environmental sounds such as noise generated around the product is reduced. However, inspection becomes difficult for products that cannot be connected by magnets. In addition, it becomes difficult to inspect for sounds that do not vibrate the connection location of the acceleration pickup, such as wind noise. On the other hand, using a microphone to detect the sound generated by the product does not pose such a problem. However, it is affected by environmental sounds such as noise generated around the product. Therefore, it is required to suppress deterioration in accuracy of processing even when environmental sounds are present.

The processing device according to the present embodiment connects a microphone for recording sounds generated by an object to be inspected (hereinafter referred to as a "first microphone"), and a microphone for recording environmental sounds (hereinafter referred to as a "first microphone"). (referred to as the “second microphone”). Based on the information indicating the sound acquired by the first microphone (hereinafter referred to as "inspection information"), the processing device converts the waveform indicated in the inspection information into an image (hereinafter referred to as "inspection image ). In addition, based on the information indicating the sound acquired by the second microphone (hereinafter referred to as "environmental sound information"), the processing device converts the waveform indicated in the environmental sound information into an image (hereinafter referred to as "time change" of the frequency component) , called “environmental sound image”). The processing device performs neural network processing on the inspection image and the environmental sound image.

FIG. 1 shows the configuration of a processing device 100. As shown in FIG. The processing unit 100 includes a first microphone 10a, a second microphone 10b, collectively referred to as the microphone 10, a first amplifier 12a, a second amplifier 12b, collectively referred to as the amplifier 12, and a first A/D converter 14, collectively referred to as the A/D converter . A first input section 110a, a second input section 110b, and a first conversion section 112a and a second conversion section 112, which are connected to the converter 14a and the second A/D converter 14b and are collectively referred to as the input section 110. A unit 112b, a processing unit 114, and an output unit 118 are included. Here, the processing unit 114 is learned in the learning process, and the processing unit 114 is used in the determination process. The determination process is a process of determining whether the inspection image is normal or defective by using a neural network for the inspection image and the environmental sound image. The learning process may be performed by a device other than the processing device 100 and the learning result may be applied to the processing unit 114 .

A first microphone 10a is placed near an object to be inspected to acquire sound. For example, when an object is making an abnormal sound due to a failure or the like, the first microphone 10a acquires the abnormal sound. On the other hand, when the object is normal, the first microphone 10a does not pick up the abnormal sound. The first microphone 10a converts the acquired sound into an electronic signal. Sound converted into an electrical signal (hereinafter also referred to as "sound") is represented by a voltage waveform over time. The occurrence of vibration is indicated when the absolute value of the voltage is greater than the threshold, and the absence of vibration is indicated when the absolute value of the voltage is equal to or less than the threshold. The first microphone 10a outputs sound to the first amplifier 12a. The first amplifier 12a amplifies the sound from the first microphone 10a and outputs the amplified sound to the first A/D converter 14a. The first A/D converter 14a converts analog signal sound into digital signal sound by performing A/D conversion on the sound from the first amplifier 12a. The first A/D converter 14 a outputs the sound of the digital signal to the processing device 100 .

The first input section 110a of the processing device 100 receives the sound of the digital signal from the first A/D converter 14a. Here, the sound used in the learning process is called "learning information", and the sound used in the determination process is called "test information". It can be said that both the learning information and the examination information are information indicating temporal changes in sound. The first input unit 110a outputs learning information or examination information to the first conversion unit 112a.

In the learning process, the first conversion unit 112a converts the learning information received from the first input unit 110a into a learning image showing the time change of the frequency component. The inspection information received from the unit 110a is converted into an inspection image showing temporal changes in frequency components. Here, the processing in the first conversion unit 112a will be described with reference to FIGS. 2(a)-(b) and FIGS. 3(a)-(c). 2(a) and 2(b) show an overview of the processing of the first conversion unit 112a. FIG. 2(a) shows learning information or inspection information. The horizontal axis indicates time, and the vertical axis indicates intensity, eg, voltage. As described above, the learning information or inspection information indicates waveforms. Certain short periods are defined, such as a first period of time 300, a second period of time 302, and a third period of time 304. FIG. These are, for example, 6 milliseconds. Adjacent periods such as the first period 300 and the second period 302 and the second period 302 and the third period 304 are arranged so as to partially overlap.

The first transforming unit 112a Fourier transforms the waveform in the first period 300 . As shown in FIG. 2(b), the result of the Fourier transform is shown as first frequency data 400. FIG. Also, the first transforming unit 112a Fourier transforms the waveforms in the second period 302 and the third period 304 . The results of the Fourier transform are shown as second frequency data 402 and third frequency data 404, respectively, as shown in FIG. 2(b). The first frequency data 400, the second frequency data 402, and the third frequency data 404 are collectively referred to as frequency data, the horizontal axis of frequency data indicates time, and the vertical axis of frequency data indicates intensity.

FIGS. 3A to 3C show an overview of the processing of the first conversion unit 112a. 3(a) shows the first frequency data 400 of FIG. 2(b), and FIG. 3(b) shows the second frequency data 402 of FIG. 2(b). FIG. 3(c) shows a spectrogram generated based on frequency data such as the first frequency data 400. FIG. In the spectrogram, the horizontal axis indicates time and the vertical axis indicates frequency. This is generated by arranging each frequency data in the vertical direction while shifting them in the horizontal direction. In the spectrogram, the intensity of frequency data is indicated by different colors, different densities, and the like. Here, the spectrogram in the learning process corresponds to the learning image, and the spectrogram in the determination process corresponds to the inspection image. Return to FIG. The first conversion unit 112 a outputs the learning image or inspection image to the processing unit 114 .

The second microphone 10b is installed in a space where an object is placed, and acquires environmental sounds such as noise when the first microphone 10a acquires sound from the object. The second microphone 10b converts the acquired environmental sound into an electronic signal. The environmental sound converted into an electric signal (hereinafter also referred to as "environmental sound") is represented by a voltage waveform over time. The second microphone 10b outputs environmental sounds to the second amplifier 12b. The second amplifier 12b amplifies the environmental sound from the second microphone 10b and outputs the amplified environmental sound to the second A/D converter 14b. The second A/D converter 14b performs A/D conversion on the environmental sound from the second amplifier 12b, thereby converting analog signal sound into digital signal sound. The second A/D converter 14 b outputs the digital signal of the environmental sound to the processing device 100 .

The second input unit 110b of the processing device 100 receives the digital signal of the environmental sound from the second A/D converter 14b. Here, the received environmental sound is called "environmental sound information". The environmental sound information can be said to be information indicating temporal changes in the environmental sound when learning information or examination information is acquired. The second input unit 110b outputs the environmental sound information to the second conversion unit 112b.

The second conversion unit 112b converts the environmental sound information received from the second input unit 110b into an environmental sound image showing temporal changes in frequency components. The conversion processing in the second conversion unit 112b is the same as the conversion processing in the first conversion unit 112a, so the explanation is omitted here. The second conversion unit 112 b outputs the environmental sound image to the processing unit 114 .

The processing unit 114 receives the learning image or the inspection image from the first conversion unit 112a. 4(a)-(c) show images used in the processing unit 114. FIG. FIGS. 4(a) and 4(b) show training images or inspection images in a different situation from FIG. 3(c). FIG. 4(a) shows a learning image or inspection image when the object is normal, and FIG. 4(b) shows a learning image or inspection image when the object is making an abnormal noise due to a failure or the like. show. FIG. 4(c) will be described later, returning to FIG. The processing unit 114 also receives the environmental sound image from the second conversion unit 112b.

The processing unit 114 performs neural network processing on the environmental sound image and the learning image in the learning process, and performs neural network processing on the environmental sound image and the inspection image in the determination process. to run. 5A to 5C show the configuration of the processing unit 114. FIG. FIG. 5A shows an example of the configuration of the processing unit 114. As shown in FIG. The processing unit 114 includes a first convolution layer 142a, a second convolution layer 142b, a third convolution layer 142c, a fourth convolution layer 142d, a first pooling layer 144a, a pooling layer 144, a second convolution layer 142b, and a pooling layer 144. It includes two pooling layers 144 b , a third pooling layer 144 c , a fourth pooling layer 144 d , a combination layer 146 and a fully connected layer 148 .

The combination layer 146 is connected to the first conversion unit 112a and the second conversion unit 112b in FIG. 1, and receives the inspection image from the first conversion unit 112a and the environmental sound image from the second conversion unit 112b. Here, for clarity of explanation, an inspection image will be described, but the same applies to a learning image. A combination layer 146 combines the inspection image and the environmental sound image. In a first combination example, two inputs, an inspection image and an ambient sound image, are merged into one as separate channels. In this case, a combination of the inspection image and the environmental sound image is generated. In the second example of combination, the difference between corresponding pixels of the inspection image and the environmental sound image, which are two inputs, is calculated, and an image (hereinafter referred to as a "difference image") is generated by arranging the differences for each pixel. be. In a third combination example, the inspection image, ambient sound image, and difference image are merged into one as separate channels. In this case, a combination of an inspection image, an environmental sound image, and a difference image is generated. In a fourth example of combination, the ambient sound image and the difference image are merged into one as separate channels. In this case, a combination of the environmental sound image and the difference image is generated. In a fifth example of combination, the inspection image and the difference image are merged into one as separate channels. In this case, a combination of the inspection image and the difference image is generated. The combination layer 146 outputs the result of combination (hereinafter referred to as "combination image"). The inspection image, ambient sound image, and differential image are collectively referred to as "images."

Convolutional layer 142 performs spatial filtering on each channel of the combined image, shifting the spatial filter with a size smaller than the size of the image. Spatial filtering is a well-known technique, so its explanation is omitted here, but this spatial filtering corresponds to convolution processing, and the feature amount of the image is extracted by convolution processing. Padding and the like may be performed in the convolutional layer 142 . In addition, convolutional layer 142 may perform multiple spatial filtering in parallel using multiple spatial filters in parallel on the image for each channel. This parallel use of multiple spatial filters increases the image. This corresponds to an increase in the number of channels in the combined image.

The pooling layer 144 reduces the size of the image by combining a plurality of pixels contained in an arbitrary region within the image of each channel in the combined image into one pixel. Here, average pooling or max pooling is performed to combine multiple pixels into one pixel. In average pooling, the average value of multiple pixel values in a region is used for one pixel, and in max pooling, the maximum of multiple pixel values in a region is used for one pixel. . The pooling process is performed to enhance robustness against translational movement of the representative value or average value in the region of interest.

Here, the first convolutional layer 142a, the first pooling layer 144a, the second convolutional layer 142b, the second pooling layer 144b, the third convolutional layer 142c, the third pooling layer 144c, the fourth convolutional layer 142d, the fourth pooling layer 144d are executed in order. That is, convolution processing and pooling processing are repeatedly performed on the combined image. In addition, the image size of each channel is reduced in order by repeating the convolution process and the pooling process. As a result, a combined image having a spatial dimension of 1×1 and having a number of channels greater than or equal to 1 is output to the fully connected layer 148 .

A fully connected layer 148 receives an image from which features have been extracted. Fully connected layer 148 identifies images by performing classification into multiple classes based on features. A well-known technique may be used for processing in the fully connected layer 148, so a description thereof is omitted here. Probabilities are indicated. Here, "OK" corresponds to a case where the combination of the environmental sound image and the inspection image does not contain any abnormal sound component, and "NG" corresponds to the case where the combination of the environmental sound image and the inspection image contains an abnormal sound component. is included. Return to FIG.

In the learning process, the processing unit 114 uses a learning image. It is known that training images contain abnormal noise components or do not contain abnormal noise components, unlike inspection images, in which it is unknown whether or not abnormal noise components are included. is. The processing unit 114 also receives teacher data indicating whether or not the learning image includes an abnormal sound component. The training data indicates "OK" or "NG". The processing unit 114 learns the spatial filter coefficients of each convolution layer 142 so that the combination of the environmental sound image and the learning image serves as teacher data. A well-known technique may be used for the learning of the coefficients of the spatial filter, so the explanation is omitted here. In this way, the neural network in the processing section 114 is trained so that the inspection results output from the output section 118 indicate non-defective products or defects.

In the determination process after the learning process, the inspection image is used in the processing unit 114 . Here, when the learning process is performed by a device other than the processing device 100 , the coefficients of the spatial filter derived by the learning processing by another device are set in the processing unit 114 . The processing unit 114 executes neural network processing on the environmental sound image and the inspection image. The output unit 118 outputs, as the processing result of the processing unit 114, information indicating whether or not the combination of the environmental sound image and the inspection image includes an abnormal sound component, that is, the inspection result of the object to be inspected.

Another example of the configuration of the processing unit 114 is shown below. FIG. 5B shows another example of the configuration of the processing unit 114. As shown in FIG. The processing unit 114 includes a 1-1 convolution layer 142aa, a 1-2 convolution layer 142ab, a 1-3 convolution layer 142ac, a 2-1 convolution layer 142ba, and a 2-2 convolution layer, which are collectively referred to as the convolution layers 142. 142bb, 2nd-3rd convolutional layer 142bc, fourth convolutional layer 142d, 1-1st pooling layer 144aa, 1-2nd pooling layer 144ab, 1-3rd pooling layer 144ac, 2nd- 1 pooling layer 144ba, 2nd-2nd pooling layer 144bb, 2nd-3rd pooling layer 144bc, fourth pooling layer 144d, combination layer 146, and all-connected layer 148 are included.

The 1-1 convolution layer 142aa, the 1-1 pooling layer 144aa, the 1-2 convolution layer 142ab, the 1-2 pooling layer 144ab, the 1-3 convolution layer 142ac, and the 1-3 pooling layer 144ac are arranged in order. be done. These perform the aforementioned processing on inspection images. The 2-1 convolution layer 142ba, the 2-1 pooling layer 144ba, the 2-2 convolution layer 142bb, the 2-2 pooling layer 144bb, the 2-3 convolution layer 142bc, and the 2-3 pooling layer 144bc are arranged in order. be done. These perform the above-described processing on the ambient sound image.

The combination layer 146 inputs the processing result from the 1st-3 pooling layer 144ac and the processing result from the 2nd-3rd pooling layer 144bc. The processing result from the 1-3 pooling layer 144ac is the processing result for the inspection image (hereinafter referred to as "first processing result"), and the processing result from the 2-3 pooling layer 144bc is the processing result for the environmental sound image ( hereinafter referred to as "second processing result"). Here, the first processing result and the second processing result may be composed of a plurality of channels. A combination layer 146 combines the first and second processing results. Combinations may be made as in any one of the first to fifth examples described above. A difference image is generated between corresponding channels of the first processing result and the second processing result. The combination layer 146 outputs the result of combination (hereinafter also referred to as "combination image").

The fourth convolution layer 142d and the fourth pooling layer 144d sequentially perform convolution processing and pooling processing on the combined image. The all-bonded layer 148 is constructed in the same manner as in FIG. 5(a). The learning process and determination process of the processing device 100 for the configuration of such a processing unit 114 are the same as before, and therefore descriptions thereof are omitted here. As a result of learning, the weighting factor when performing the processing of the convolution layer 142 on the inspection image and the weighting factor when performing the processing of the convolution layer 142 on the environmental sound image may be made common. Specifically, the weight coefficient is shared between the 1-1 convolution layer 142aa and the 2-1 convolution layer 142ba. In addition, the weight coefficients are shared between the 1-2 convolution layer 142ab and the 2-2 convolution layer 142bb, and the weight coefficients are shared between the 1-3 convolution layer 142ac and the 2-3 convolution layer 142bc. made common.

FIG. 5C shows still another example of the configuration of the processing unit 114. As shown in FIG. The processing unit 114 includes a 1-1 convolution layer 142aa, a 1-2 convolution layer 142ab, a 1-3 convolution layer 142ac, a 1-4 convolution layer 142ad, and a 2-1 convolution layer, which are collectively referred to as the convolution layers 142. 142ba, the 2nd-2nd convolutional layer 142bb, the 2nd-3rd convolutional layer 142bc, the 2nd-4th convolutional layer 142bd, the 1st-1st pooling layer 144aa, the 1st-2nd pooling layer 144ab, and the 1-3 pooling layer 144ac, 1-4 pooling layer 144ad, 2-1 pooling layer 144ba, 2-2 pooling layer 144bb, 2-3 pooling layer 144bc, 2-4 pooling layer 144bd, combination layer 146 , including all-connected layers 148 .

1-1 convolution layer 142aa, 1-1 pooling layer 144aa, 1-2 convolution layer 142ab, 1-2 pooling layer 144ab, 1-3 convolution layer 142ac, 1-3 pooling layer 144ac, 1st The -4 convolutional layer 142ad and the 1-4th pooling layer 144ad are arranged in order. These perform the aforementioned processing on inspection images. 2-1 convolution layer 142ba, 2-1 pooling layer 144ba, 2-2 convolution layer 142bb, 2-2 pooling layer 144bb, 2-3 convolution layer 142bc, 2-3 pooling layer 144bc, 2nd The -4 convolutional layer 142bd and the 2-4th pooling layer 144bd are arranged in order. These perform the above-described processing on the ambient sound image.

The combination layer 146 inputs the processing result from the 1st-4th pooling layer 144ad and the processing result from the 2nd-4th pooling layer 144bd. The processing result from the 1-4th pooling layer 144ad is the processing result for the inspection image (hereinafter also referred to as "first processing result"), and the processing result from the 2-4th pooling layer 144bd is for the environmental sound image. A processing result (hereinafter also referred to as a “second processing result”). A combination layer 146 combines the first and second processing results. Combinations may be made as in any one of the first to fifth examples described above. The combination layer 146 outputs the result of combination (hereinafter also referred to as "combination image").

The all-bonded layer 148 is constructed in the same manner as in FIG. 5(a). The learning process and determination process of the processing device 100 for the configuration of such a processing unit 114 are the same as before, and therefore descriptions thereof are omitted here. As a result of learning, the weighting factor when performing the processing of the convolution layer 142 on the inspection image and the weighting factor when performing the processing of the convolution layer 142 on the environmental sound image may be made common. Specifically, weight coefficients are shared between the 1-1 convolution layer 142aa and the 2-1 convolution layer 142ba, and between the 1-2 convolution layer 142ab and the 2-2 convolution layer 142bb Weighting factors are shared. In addition, the weight coefficients are shared between the 1-3 convolution layer 142ac and the 2-3 convolution layer 142bc, and the weight coefficients are shared between the 1-4 convolution layer 142ad and the 2-4 convolution layer 142bd. made common.

The fully connected layer 148 may be excluded from the configuration of the processing section 114 . Such a configuration will be described with reference to FIG. FIG. 6 shows an outline of processing in the processing unit 114 . Here, it is shown as one input and the combinational layer 146 is omitted for clarity of explanation. However, as in FIGS. 5(a)-(c), it only needs to have two inputs and a combinational layer 146 included. The processing unit 114 includes a first convolution layer 142a, a second convolution layer 142b, a third convolution layer 142c, a fourth convolution layer 142d, a fifth convolution layer 142e, a sixth convolution layer 142f, and a pooling layer 144, which are collectively referred to as the convolution layers 142. A first pooling layer 144a, a second pooling layer 144b, a third pooling layer 144c, a fourth pooling layer 144d, and a fifth pooling layer 144e are collectively referred to as pooling layers. The convolution layer 142 and the pooling layer 144 are images of blocks that execute each process.

The input image 140 is the above-described inspection image, training image, environmental sound image, combined image, and other images. Therefore, no limit is placed on the size of the input image 140 . An input image 140 is input to a first convolutional layer 142a. In the processing unit 114, a first convolutional layer 142a, a first pooling layer 144a, a second convolutional layer 142b, a second pooling layer 144b, a third convolutional layer 142c, a third pooling layer 144c, a fourth convolutional layer 142d, and a fourth pooling Layer 144d, fifth convolutional layer 142e, fifth pooling layer 144e, and sixth convolutional layer 142f are arranged in order. That is, the convolution process and the pooling process are repeatedly executed as before.

The processing unit 114 having such a configuration also learns the coefficient of the spatial filter of each convolution layer 142 so that the combination of the learning image and the environmental sound image becomes the teacher data. In this embodiment, the size of the training data has a spatial dimension of 1×1. Therefore, the training data only indicates one of a limited number of classes for the combination of the learning image and the environmental sound image. In other words, the teacher data for one channel only needs to indicate whether or not it corresponds to one class. Therefore, compared to the case of generating an image in which the object is segmented by coloring, the amount of work for generating one piece of teacher data is reduced. As a result, it is possible to increase the number of training data while suppressing an increase in the amount of work.

On the other hand, the combination of the learning image and the environmental sound image is the original image such that the teacher data is output when the judgment process is executed correctly, and its size is 1×1 spatial dimension of the teacher data. is determined to be Here, since there is no restriction on the size of the input image 140, the image used for the learning process and the image used for the determination process may have different sizes. A well-known technique may be used for the learning of the coefficients of the spatial filter, so the explanation is omitted here.

Below, the modification of the processing apparatus 100 is demonstrated.
(First modification)
In FIG. 1, one or more microphones 10 may be provided in addition to the first microphone 10a to be arranged in the vicinity of the object to be inspected and to acquire sound. An amplifier 12 and an A/D converter 14 are also provided to correspond to one or more such microphones 10 . The additional one or more microphones 10 are mounted at different locations on the object under test than the first microphone 10a to pick up sounds at different locations. The added amplifier 12 and the added A/D converter 14 are the same as before, and the first input section 110a of the processing device 100 inputs a plurality of examination information. A plurality of inspection information is information detected by a plurality of microphones 10 . The first converter 112a converts a plurality of pieces of inspection information into a plurality of inspection images. The processing in the first conversion unit 112a is as described above. The first conversion unit 112 a outputs a plurality of inspection images to the processing unit 114 .

The processing unit 114 executes neural network processing on the environmental sound image and the plurality of inspection images. FIG. 7 shows the configuration of the processing unit 114. As shown in FIG. The processing unit 114 includes a 1-1 convolution layer 142aa, a 1-2 convolution layer 142ab, a 1-3 convolution layer 142ac, a 2-1 convolution layer 142ba, and a 2-2 convolution layer, which are collectively referred to as the convolution layers 142. 142bb, 2-3 convolution layer 142bc, 3-1 convolution layer 142ca, 3-2 convolution layer 142cb, 3-3 convolution layer 142cc, N-1 convolution layer 142na, N-2 convolution layer 142nb, 1-1 pooling layer 144aa, 1-2 pooling layer 144ab, 2-1 pooling layer 144ba, 2-2 pooling layer 144bb, 3- 1 pooling layer 144ca, 3-2nd pooling layer 144cb, (N-1)th pooling layer 144na, (N-2)th pooling layer 144nb, 3rd pooling layer 144c, 4th pooling layer 144d, combination layer 146, dense block 150 It includes a first dense block 150a and a second dense block 150b.

The processing unit 114 receives from the conversion unit 112 the first inspection image to the (N−1)th inspection image, that is, the N−1 kinds of inspection images. The 1-1 convolution layer 142aa receives the first inspection image, the 2-1 convolution layer 142ba receives the environmental sound image, the 3-1 convolution layer 142ca receives the second inspection image, and the N-th convolution layer 142ca receives the second inspection image. 1 convolutional layer 142na receives the N-1th inspection image. The 1-1 convolution layer 142aa, the 1-1 pooling layer 144aa, the 1-2 convolution layer 142ab, the 1-3 convolution layer 142ac, and the 1-2 pooling layer 144ab perform neural network processing on the first inspection image. to run. The same applies to other inspection images. The 2-1 convolution layer 142ba, the 2-1 pooling layer 144ba, the 2-2 convolution layer 142bb, the 2-3 convolution layer 142bc, and the 2-2 pooling layer 144bb perform neural network processing on the environmental sound image. Execute.

The combination layer 146 receives the processing results from each of the 1-2nd pooling layer 144ab to the N-2th pooling layer 144nb. A combination layer 146 combines these processing results as described above. At the time of combination, different inspection images may be combined with different colors, thereby enabling processing based on color determination. The combination layer 146 outputs the result of combination (hereinafter also referred to as "combination image"). The first dense block 150a is composed of a combination of convolution layers 142, combination layers 146, and the like. A well-known technique may be used for the dense block 150, so the explanation is omitted here. The third pooling layer 144c, the second dense block 150b, and the fourth pooling layer 144d, which follow the first dense block 150a, perform the same processing as before, so description thereof is omitted here. Convolutional layer 142 and dense block 150 in FIG. 7 are trained according to FIG.

The configuration of the processing unit 114 shown in FIG. 7 is similar to that of FIG. 5(b). However, the configuration of the processing unit 114 may be similar to that shown in FIG. 5(a) or similar to that shown in FIG. 5(c).

(Second modification)
The inspection image and the ambient sound image may be images with N-dimensional M channels (N, M integers). Here, the N dimensions include one dimension, two dimensions, three dimensions, four dimensions, and so on. The processing unit 114 may perform the same processing as before on such inspection images and environmental sound images.

(Third modification)
The processing unit 114 so far includes the convolution layer 142 and the pooling layer 144 , but the convolution layer 142 and the pooling layer 144 may not be included. FIG. 8 shows the configuration of the processing unit 114. As shown in FIG. The processing unit 114 includes a combination layer 146, a 1-1 fully-bonded layer 148aa, a 1-2 fully-bonded layer 148ab, a 1-3 fully-bonded layer 148ac, and a 2-1 fully-bonded layer 148aa, which are collectively referred to as a fully-bonded layer 148. It includes layer 148ba, second-second all-bonded layer 148bb, second-third all-bonded layer 148bc, and fourth fully-bonded layer 148d. That is, with the exception of combination layer 146, only fully bonded layer 148 is included. The 1-1 fully connected layer 148aa receives an inspection image, and the 2-1 fully connected layer 148ba receives an environmental sound image. The 1-1 fully connected layer 148aa to the 1-3 fully connected layer 148ac perform fully connected layer processing on the inspection image, and the 2-1 fully connected layer 148ba to the 2-3 fully connected layer 148bc performs fully connected layer processing on the ambient sound image. The combination layer 146 receives processing results from the 1st-3rd all-bonded layers 148ac and the 2nd-3rd all-bonded layers 148bc. A combination layer 146 combines these processing results as described above. The combination layer 146 outputs the result of combination (hereinafter also referred to as "combination image"). A fourth fully connected layer 148d performs fully connected layer processing on the combined image.

The configuration of the processing unit 114 shown in FIG. 8 is similar to that of FIG. 5(b). However, the configuration of the processing unit 114 may be similar to that shown in FIG. 5(a) or similar to that shown in FIG. 5(c).

(Fourth modification)
In the previous combination layer 146, multiple channels are merged into one, as in combination examples 1, 3-5. Mergers are not limited to this. 9A to 9C show the configuration of the processing section 114. FIG. A first input image 200a and a second input image 200b are two images input to the combination layer 146, and include a learning image, an inspection image, an environmental sound image, and a processing result. FIG. 9(a) shows the merging so far in the combination layer 146, where the two channel first input image 200a, second input image 200b are merged into one channel. The number of input images 200 collectively including the first input image 200a and the second input image 200b is not limited to "2". Also, an image made into one channel corresponds to the aforementioned combined image. In FIG. 9B, a first input image 200a and a second input image 200b of two channels are arranged in the x-axis direction to form one channel. In FIG. 9C, the first input image 200a and the second input image 200b of two channels are arranged in the y-axis direction to form one channel.

The subject of an apparatus, system, or method in this disclosure comprises a computer. The main functions of the device, system, or method of the present disclosure are realized by the computer executing the program. A computer has a processor that operates according to a program as its main hardware configuration. Any type of processor can be used as long as it can implement functions by executing a program. The processor is composed of one or more electronic circuits including a semiconductor integrated circuit (IC) or LSI (Large Scale Integration). A plurality of electronic circuits may be integrated on one chip or may be provided on a plurality of chips. A plurality of chips may be integrated into one device, or may be provided in a plurality of devices. The program is recorded in a non-temporary recording medium such as a computer-readable ROM, optical disk, hard disk drive, or the like. The program may be pre-stored in a recording medium, or may be supplied to the recording medium via a wide area network including the Internet.

According to this embodiment, neural network processing including convolutional layer processing is performed on the environmental sound image and the inspection image, so even when the environmental sound is present, the accuracy of the processing is suppressed. can. In addition, since the microphone 10 is used, it is possible to collect wind noise and the like, so that an inspection that is closer to human sensory inspection can be performed. In addition, the use of the microphone 10 eliminates the need for a soundproof room for inspection, so that in-line operation can be easily realized.

In addition, since neural network processing is performed on the environmental sound image and a plurality of inspection images, it is determined whether or not the inspection image contains an abnormal sound component based on the sounds from the plurality of microphones 10. Therefore, the judgment accuracy can be improved. Also, since the neural network is trained so that the inspection result output from the output unit 118 indicates a non-defective item or a defect, it can inspect the object.

Moreover, since the processing of the convolution layer 142 and the processing of the pooling layer 144 are performed on the combination after combining the inspection image and the environmental sound image, an increase in the amount of processing can be suppressed. Also, after performing the processing of the convolution layer 142 and the processing of the pooling layer 144 on each of the inspection image and the environmental sound image, the processing of the convolution layer 142 and the processing of the pooling layer 144 are performed on the combination thereof. Therefore, the accuracy of processing can be improved. Further, since the processing of the convolution layer 142 and the processing of the pooling layer 144 are performed on each of the inspection image and the environmental sound image, and then these are combined, the accuracy of the processing can be improved.

In addition, since the weighting factor for performing the processing of the convolution layer 142 on the inspection image and the weighting factor when performing the processing of the convolution layer 142 on the environmental sound image are common, the accuracy of the processing can be improved. can improve. Also, as a combination, two inputs are combined into one, so processing can be simplified. Moreover, since the difference between the two inputs is derived as a combination, the accuracy of processing can be improved. In addition, as a combination, at least one of the two inputs is combined with the difference, so the accuracy of processing can be improved. Since a convolutional neural network in which the spatial filter of the convolution layer 142 has been trained is used for teacher data having a spatial dimension of 1×1, the workload for creating teacher data can be reduced. Moreover, since the amount of work for creating teacher data is reduced, an increase in the amount of work for learning can be suppressed. Also, since the amount of work involved in creating teaching data is reduced, the number of teaching data can be increased. Also, since the number of training data is increased, the accuracy of learning can be improved.

Moreover, since the inspection image and the environmental sound image are images having N-dimensional M channels (N and M are integers), the scope of application of this embodiment can be expanded. In addition, since the inspection image and the environmental sound image are combined and then the processing of the fully connected layer 148 is executed for the combination, the degree of freedom in configuration can be improved. In addition, the processing of the fully connected layer 148 is performed on the inspection image, and the processing of the fully connected layer 148 is performed on the environmental sound image. On the other hand, since the processing of the fully connected layer 148 is executed, the degree of freedom of configuration can be improved. In addition, the processing of the fully connected layer 148 is performed on the inspection image, the processing of the fully connected layer 148 is performed on the environmental sound image, and the processing result of the inspection image and the processing result of the environmental sound image are combined. , the degree of freedom of configuration can be improved.

A summary of one aspect of the present disclosure is as follows. A processing device (100) according to an aspect of the present disclosure includes a first input unit (110a) for inputting inspection information obtained from an object to be inspected and indicating a change in sound over time; A first conversion unit (112a) that converts examination information input to an input unit (110a) into an examination image that indicates temporal changes in frequency components, and an environment that indicates temporal changes in environmental sounds when obtaining examination information. A second input unit (110b) for inputting sound information, and a second conversion unit (112b) for converting environmental sound information input to the second input unit (110b) into an environmental sound image showing temporal changes in frequency components. and a process of executing neural network processing including convolution layer processing on the environmental sound image converted by the second conversion unit (112b) and the inspection image converted by the first conversion unit (112a). A unit (114) and an output unit (118) for outputting an inspection result of an object to be inspected as a processing result of the processing unit (114).

A first input unit (110a) inputs a plurality of inspection information, and a first conversion unit (112a) converts a plurality of inspection information input to the first input unit (110a) into a plurality of inspection images for processing. The unit (114) may perform neural network processing on the environmental sound image and the plurality of inspection images converted by the first conversion unit (112a).

The neural network in the processing section (114) may be trained such that the inspection results output from the output section (118) indicate good or defective products.

The output unit (118) may also output the feature amount during processing of the neural network in the processing unit (114).

Yet another aspect of the present disclosure is a processing method. This method comprises the steps of: inputting inspection information obtained from an object to be inspected and indicating temporal changes in sound; and converting the input inspection information into an inspection image indicating temporal changes in frequency components. a step of converting; a step of inputting environmental sound information indicating temporal changes in environmental sounds when acquiring inspection information; and converting the input environmental sound information into an environmental sound image indicating temporal changes in frequency components. performing neural network processing including convolutional layer processing on the transformed environmental sound image and the transformed inspection image; inspecting an object to be inspected as a processing result of the neural network; and outputting the result.

(Example 2)
Next, Example 2 will be described. The second embodiment, like the first embodiment, relates to a processing apparatus for determining whether or not a product, such as a rotating device such as a motor, contains an abnormality. When using a neural network, re-learning is required for each motor product number. Such re-learning increases the amount of learning work. Insufficient learning, however, reduces processing accuracy. The processing apparatus according to the second embodiment prepares an image (hereinafter referred to as a “reference image”) that shows temporal changes in frequency components of a normal object and that can be compared with an inspection image. The processing device determines whether the inspection image is normal or defective by executing neural network processing on the combination of the inspection image, the environmental sound image, and the reference image. At this time, in the processing device, learning is performed in advance using information as to whether the inspection image is normal or defective with respect to the combination of the inspection image, the environmental sound image, and the reference image, as teacher data. In other words, it is learned whether the result of comparison between the combination of the inspection image and the environmental sound image and the reference image is normal or defective. As a result, even if the reference image changes due to product renewal, it is sufficient to input new reference images, inspection images, and environmental sound images to the processing device, and even unlearned sound patterns can be determined. Here we will focus on the differences.

FIG. 10 shows the configuration of the processing device 100. As shown in FIG. In the processing device 100, a storage unit 116 is added to the processing device 100 of FIG. Storage unit 116 stores the reference image. Since the reference image is a spectrogram obtained in advance for a normal object, it is an image showing temporal changes in frequency components. FIG. 4(c) shows a reference spectrogram for the object under inspection. The processing unit 114 executes neural network processing including a reference image showing temporal changes in frequency components of a normal object. The processing unit 114 performs neural network processing on the reference image, the learning image, and the environmental sound image in the case of the learning process, and performs neural network processing on the reference image, the inspection image, and the environmental sound image in the case of the determination process. Neural network processing is performed on the Processing unit 114 is shown similar to FIG.

A processing unit 114 receives an inspection image, an environmental sound image, and a reference image. The 1-1 convolution layer 142aa receives the inspection image, the 2-1 convolution layer 142ba receives the ambient sound image, and the 3-1 convolution layer 142ca receives the reference image. The 1-1 convolution layer 142aa, the 1-1 pooling layer 144aa, the 1-2 convolution layer 142ab, the 1-3 convolution layer 142ac, and the 1-2 pooling layer 144ab perform neural network processing on the inspection image. do. The 2-1 convolution layer 142ba, the 2-1 pooling layer 144ba, the 2-2 convolution layer 142bb, the 2-3 convolution layer 142bc, and the 2-2 pooling layer 144bb perform neural network processing on the environmental sound image. Execute. The 3-1 convolution layer 142ca, the 3-1 pooling layer 144ca, the 3-2 convolution layer 142cb, the 3-3 convolution layer 142cc, and the 3-2 pooling layer 144cb perform neural network processing on the reference image. do. The combination layer 146 receives the processing results from each of the 1-2nd pooling layer 144ab to the 3-2nd pooling layer 144cb. A combination layer 146 combines these processing results as described above.

In the learning process, the processing unit 114 uses a learning image. It is known that training images contain abnormal noise components or do not contain abnormal noise components, unlike inspection images, in which it is unknown whether or not abnormal noise components are included. is. In addition, the processing unit 114 receives teacher data indicating whether or not the learning image includes an abnormal sound component with respect to the reference image. The training data indicates "OK" or "NG". The processing unit 114 learns the coefficient of the spatial filter of each convolution layer 142 so that the relationship between the combination of the learning image and the environmental sound image and the reference image becomes teacher data. A well-known technique may be used for the learning of the coefficients of the spatial filter, so the explanation is omitted here. In this way, the neural network in the processing section 114 is trained so that the inspection results output from the output section 118 indicate non-defective products or defects.

In the determination process after the learning process, the inspection image is used in the processing unit 114 . Here, when the learning process is performed by a device other than the processing device 100 , the coefficients of the spatial filter derived by the learning processing by another device are set in the processing unit 114 . The processing unit 114 performs neural network processing on the environmental sound image, the inspection image, and the reference image. The output unit 118 outputs, as the processing result of the processing unit 114, information indicating whether or not the combination of the inspection image and the environmental sound image includes an abnormal sound component with respect to the reference image, that is, the inspection result of the object to be inspected. Output.

According to the present embodiment, by executing neural network processing on the inspection image, the environmental sound image, and the reference image, a combination of the inspection image and the environmental sound image and the reference image are compared to obtain a difference in the inspection image. Information indicating whether or not a sound component is included can be output. Moreover, since the result of comparison between the combination of the inspection image and the environmental sound image and the reference image is output, even if the reference image is changed, the amount of learning using a new reference image can be reduced. Also, even if the reference image is changed, the amount of learning based on the new reference image is reduced, so an increase in the amount of learning work can be suppressed.

A summary of one aspect of the present disclosure is as follows. The processing unit (114) may also perform neural network processing including a reference image showing temporal changes in frequency components of a normal object.

(Example 3)
Next, Example 3 will be described. The third embodiment, like the first embodiment, relates to a processing apparatus for determining whether or not a product, such as a rotating device such as a motor, contains an abnormality. Until now, neural network processing has been performed on images such as learning images, inspection images, and environmental sound images. The processing device according to the second embodiment executes a neural network on information such as learning information, examination information, and environmental sound information. Here we will focus on the differences.

FIG. 11 shows the configuration of the processing device 100. As shown in FIG. The processing unit 100 includes a first microphone 10a, a second microphone 10b, collectively referred to as the microphone 10, a first amplifier 12a, a second amplifier 12b, collectively referred to as the amplifier 12, and a first A/D converter 14, collectively referred to as the A/D converter . It includes a first input section 110 a , a second input section 110 b , a processing section 114 and an output section 118 , which are connected to the converter 14 a and the second A/D converter 14 b and collectively referred to as the input section 110 . The processing unit 114 receives learning information or examination information from the first input unit 110a, and receives environmental sound information from the second input unit 110b. The processing unit 114 performs neural network processing on the learning information and the environmental sound information in the case of the learning process, and performs neural network processing on the inspection information and the environmental sound information in the case of the determination process. to run. Since the processing in the processing unit 114 may be the same as before, the description is omitted here.

According to this embodiment, neural network processing is performed on inspection information indicating waveforms and environmental sound information indicating waveforms. It is possible to eliminate the need for conversion to a sound image. In addition, conversion from inspection information to an inspection image and conversion from environmental sound information to an environmental sound image become unnecessary, so the amount of processing can be reduced.

A summary of one aspect of the present disclosure is as follows. Another aspect of the present disclosure is also a processing device (100). This apparatus includes a first input unit (110a) for inputting inspection information obtained from an object to be inspected and indicating a change in sound over time; A second input section (110b) for inputting environmental sound information indicating changes, environmental sound information input to the second input section (110b), and examination information input to the first input section (110a) , a processing unit (114) for executing neural network processing including convolutional layer processing, an output unit (118) for outputting inspection results of an object to be inspected as a processing result of the processing unit (114), Prepare.

A first input unit (110a) inputs a plurality of pieces of examination information, and a processing unit (114) performs neural network processing on the environmental sound information and the plurality of pieces of examination information inputted to the first input unit (110a). processing may be performed.

The processing unit (114) may also perform neural network processing including reference information indicating temporal changes in sound in a normal object.

Yet another aspect of the present disclosure is also a processing method. This method comprises the steps of: inputting inspection information obtained from an object to be inspected and indicating a change in sound over time; a step of inputting information; a step of performing neural network processing including convolutional layer processing on the input environmental sound information and the input inspection information; and outputting inspection results of the object.

The present disclosure has been described above based on the embodiments. It should be understood by those skilled in the art that this embodiment is an example, and that various modifications are possible in the combination of each component or each treatment process, and such modifications are within the scope of the present disclosure. .

In the configuration of the processing unit 114 in Example 1, Modified Examples 1 to 4, Example 2, and Example 3, a plurality of convolution layers 142 and a plurality of pooling layers 144 are alternately arranged. However, the processing unit 114 is not limited to this, and may have a configuration such as a GoogLeNet system, a DenseNet system, or the like. According to this modified example, the degree of freedom in configuration can be improved.

In the configuration of the processing unit 114 in Example 1, Modified Examples 1 to 4, Example 2, and Example 3, a plurality of convolution layers 142 and a plurality of pooling layers 144 are alternately arranged. However, the present invention is not limited to this. For example, the processing unit 114 may include only the convolution layer 142 and perform only convolution processing, or may include only the pooling layer 144 and perform only pooling processing. According to this modified example, the degree of freedom in configuration can be improved.

The output unit 118 in the first embodiment, the first to fourth modifications, the second embodiment, and the third embodiment outputs the processing result of the neural network in the processing unit 114 . However, not limited to this, for example, the output unit 118 may output the feature amount during processing of the neural network in the processing unit 114 . According to this modified example, the degree of freedom in configuration can be improved.

Any combination of Embodiment 1, Modifications 1 to 4, Embodiment 2, and Embodiment 3 is also effective. According to this modified example, it is possible to obtain the effect of any combination of the first modified example, the first modified example to the fourth modified example, the second modified example, and the third modified example. For example, as a combination of the second embodiment and the first modified example, the first input unit 110a inputs a plurality of inspection images. The processing unit 114 performs neural network processing on the environmental sound image, the plurality of inspection images, and the reference image. According to this modified example, the processing accuracy can be improved. For example, as a combination of the third embodiment and the first modified example, the input unit 110 inputs a plurality of examination information. The processing unit 114 performs neural network processing on the environmental sound information and a plurality of pieces of examination information input to the input unit 110 . In that case, the conversion unit 112 is omitted. According to this modification, it is possible to improve the processing accuracy while suppressing an increase in the processing amount.

According to the present disclosure, it is possible to suppress deterioration in accuracy of processing even when environmental sounds are present.

10 microphone, 12 amplifier, 14 A/D converter, 100 processing device, 110 input unit, 112 conversion unit, 114 processing unit, 116 storage unit, 118 output unit, 140 input image, 142 convolution layer, 144 pooling layer, 146 combination Layers, 148 fully connected layers, 150 dense blocks.

Claims (14)

a first input unit for inputting inspection information obtained from an object to be inspected and indicating a change in sound over time;
a first conversion unit that converts the inspection information input to the first input unit into an inspection image showing temporal changes in frequency components;
a second input unit for inputting environmental sound information indicating temporal changes in the environmental sound when acquiring the examination information;
a second conversion unit that converts the environmental sound information input to the second input unit into an environmental sound image showing temporal changes in frequency components;
After inputting the environmental sound image converted by the second conversion unit and the inspection image converted by the first conversion unit into a trained neural network, the environmental sound image and the inspection image are merged. a processing unit that executes processing of the neural network on the results obtained ;
an output unit that outputs an inspection result of the object to be inspected as a processing result of the processing unit;
A processing device comprising: a first input unit for inputting inspection information obtained from an object to be inspected and indicating a change in sound over time;
a first conversion unit that converts the inspection information input to the first input unit into an inspection image showing temporal changes in frequency components;
a second input unit for inputting environmental sound information indicating temporal changes in the environmental sound when acquiring the examination information;
a second conversion unit that converts the environmental sound information input to the second input unit into an environmental sound image showing temporal changes in frequency components;
After inputting the environmental sound image converted by the second conversion unit and the inspection image converted by the first conversion unit into a trained neural network, a processing unit that performs processing for combining results of executing convolutional layer processing, and performs processing for the neural network that includes convolutional layer processing for the combination result output by the combining processing;
an output unit that outputs an inspection result of the object to be inspected as a processing result of the processing unit;
A processing device comprising: The first input unit inputs a plurality of pieces of examination information,
The first conversion unit converts the plurality of inspection information input to the first input unit into a plurality of inspection images,
The processing unit inputs the environmental sound image and the plurality of inspection images converted by the first conversion unit to a trained neural network, and then executes processing of the neural network. 3. The processing apparatus according to claim 1 or 2 . 4. The processing apparatus according to any one of claims 1 to 3, wherein the processing unit executes neural network processing including a reference image showing temporal changes in frequency components of a normal object. a first input unit for inputting inspection information obtained from an object to be inspected and indicating a change in sound over time;
a second input unit for inputting environmental sound information indicating temporal changes in the environmental sound when acquiring the examination information;
After inputting the environmental sound information input to the second input unit and the inspection information input to the first input unit into a trained neural network, the environmental sound information and the inspection information are merged. a processing unit that executes processing of the neural network on the results obtained ;
an output unit that outputs an inspection result of the object to be inspected as a processing result of the processing unit;
A processing device comprising: a first input unit for inputting inspection information obtained from an object to be inspected and indicating a change in sound over time;
a second input unit for inputting environmental sound information indicating temporal changes in the environmental sound when acquiring the examination information;
After inputting the environmental sound information input to the second input unit and the inspection information input to the first input unit into a trained neural network, a processing unit that performs processing for combining results of executing convolutional layer processing, and performs processing for the neural network that includes convolutional layer processing for the combination result output by the combining processing;
an output unit that outputs an inspection result of the object to be inspected as a processing result of the processing unit;
A processing device comprising: The first input unit inputs a plurality of pieces of examination information,
The processing unit inputs the environmental sound information and the plurality of inspection information input to the first input unit to a trained neural network, and then processes the neural network including processing of a convolution layer. 7. The processing apparatus according to claim 5 or 6 , characterized by performing 8. The processing apparatus according to any one of claims 5 to 7, wherein the processing unit performs neural network processing including reference information indicating temporal changes in sound of a normal object. 9. The processing apparatus according to any one of claims 1 to 8 , wherein the neural network in the processing unit is trained so that the inspection results output from the output unit indicate non-defective products or defects. . 10. The processing apparatus according to any one of claims 1 to 9 , wherein the output unit also outputs feature values in the course of neural network processing in the processing unit. a step of inputting inspection information obtained from an object to be inspected and indicating a change in sound over time;
a step of converting the input inspection information into an inspection image showing temporal changes in frequency components;
a step of inputting environmental sound information indicating a temporal change of the environmental sound when acquiring the examination information;
a step of converting the input environmental sound information into an environmental sound image showing temporal changes in frequency components;
After inputting the converted environmental sound image and the converted inspection image to a trained neural network, the neural network processes a result of merging the environmental sound image and the inspection image. and
a step of outputting an inspection result of the object to be inspected as a processing result of the neural network;
A processing method comprising: a step of inputting inspection information obtained from an object to be inspected and indicating a change in sound over time;
a step of converting the input inspection information into an inspection image showing temporal changes in frequency components;
a step of inputting environmental sound information indicating a temporal change of the environmental sound when acquiring the examination information;
a step of converting the input environmental sound information into an environmental sound image showing temporal changes in frequency components;
Inputting the converted environmental sound image and the converted inspection image to a trained neural network, and then combining the results of performing convolution layer processing on the environmental sound image and the inspection image. and executing the processing of the neural network including the processing of the convolutional layer on the result of the combination output by the combining processing;
a step of outputting an inspection result of the object to be inspected as a processing result of the neural network;
A processing method comprising: a step of inputting inspection information obtained from an object to be inspected and indicating a change in sound over time;
a step of inputting environmental sound information indicating a temporal change of the environmental sound when acquiring the inspection information;
After inputting the input environmental sound information and the input inspection information to a trained neural network, the neural network processes the result of combining the environmental sound information and the inspection information. and
a step of outputting an inspection result of the object to be inspected as a processing result of the neural network;
A processing method comprising: a step of inputting inspection information obtained from an object to be inspected and indicating a change in sound over time;
a step of inputting environmental sound information indicating a temporal change of the environmental sound when acquiring the examination information;
A process of inputting the input environmental sound information and the input inspection information into a trained neural network, and then combining the results of performing convolution layer processing on the environmental sound information and the inspection information. and executing the processing of the neural network including the processing of the convolutional layer on the result of the combination output by the combining processing;
a step of outputting an inspection result of the object to be inspected as a processing result of the neural network;
A processing method comprising:

Images