JP7786950B2 - Abnormal sound detection system and method - Google Patents

Description

The present invention relates to an abnormal sound detection system and method.

Abnormal sound detection technology is a technology that determines whether a sound is normal or abnormal. An abnormal sound is a sound that differs from ordinary (normal) sounds. Abnormal sound detection technology can detect abnormal sounds caused by malfunctions in factory machinery, etc., and is useful for automating machine maintenance and upkeep. Some machines operate continuously, while others alternate between operating and not operating.

In Non-Patent Document 1, an autoencoder is trained on the features of normal sounds. Next, Non-Patent Document 1 detects abnormal sounds based on the reconstruction error of the autoencoder, and detects sounds that differ from the trained sound features as abnormal sounds.

E. Marchi, “A novel approach for automatic acoustic novelty detection using a denoising autoencoder with Bidirectional LSTM neural networks,” in Proc of ICASSP, 2015

In cases where a variety of machines are operating simultaneously in a factory, the sounds input to an abnormal sound detection system may include background noise other than the sound from the machine being detected for abnormal sound (hereinafter referred to as the target machine). As a result, for example, if the operating sound of the target machine is quieter than the background noise, the difference between the mixed sound of the target machine's normal operating sound (hereinafter referred to as normal operating sound) plus background noise and the mixed sound of the target machine's abnormal operating sound (hereinafter referred to as abnormal operating sound) plus background noise will be small. Therefore, with an abnormal sound detection system, the difference between the abnormality level obtained by inputting a mixed sound of normal operating sound and background noise and the abnormality level obtained by inputting a mixed sound of abnormal operating sound and background noise will be small, resulting in a first problem: the system cannot accurately determine whether a sound is normal or abnormal. Furthermore, there is a second problem: if background noise is mixed in a frequency band different from the main frequency band of normal operating sound, the abnormal sound detection system may erroneously determine the input sound as abnormal.

An abnormal sound detection system (hereinafter referred to as an example of an abnormal sound detection system) that serves as a comparative example for comparison with the abnormal sound detection system of the present disclosure, rather than as prior art, may include, for example, a feature extraction unit, a likelihood calculation unit, and an abnormality determination unit.

The feature extraction unit extracts time-frequency domain features from the time waveform of the input sound. The likelihood calculation unit estimates the negative log-likelihood of the time-frequency domain features. The likelihood calculation unit is pre-trained to output a low negative log-likelihood for the time-frequency domain features of normal sound samples. The likelihood calculation unit then outputs the estimated negative log-likelihood as the degree of abnormality of the sound. Finally, if the degree of abnormality is equal to or greater than a predetermined threshold, the abnormality determination unit determines the input sound to be an abnormal sound and outputs the determination result. On the other hand, if the degree of abnormality is less than the threshold, the abnormality determination unit determines the input sound to be a normal sound and outputs the determination result.

An example of the second problem is when, while the target machine is operating normally, repair work such as hammering metal is occasionally carried out near the target machine. In this example, the input sound to the abnormal sound detection system is a mixture of not only the operating sound of the target machine, but also the high-frequency sound of metal being hammered. If sounds caused by repair work occur only occasionally, the likelihood calculation unit will estimate the negative log likelihood of the input sound to be high. Therefore, even if the target machine is operating normally, the example of the abnormal sound detection system mentioned above has the problem of incorrectly determining that there is an abnormality.

The present invention therefore provides an abnormal sound detection system and method that outputs a low level of abnormality for normal sounds and a high level of abnormality for abnormal sounds, even when the operating sound of the target machine is quieter than background noise. The present invention also provides an abnormal sound detection system and method that outputs a low level of abnormality for normal sounds and a high level of abnormality for abnormal sounds, even when background noise occurs at a frequency unrelated to normal operating sounds.

In order to solve at least one of the above problems, a representative example of the invention disclosed in this application is an abnormal sound detection system that includes an operating section detection subsystem that uses learning results from a set of pairs of normal sound signals of a target machine and operating section information indicating the actual operating and non-operating sections of the target machine to estimate the operating and non-operating sections of the target machine in response to input of a normal sound signal of the target machine, and an abnormality judgment unit that judges whether the input signal of an abnormal sound detection target of the target machine is normal or abnormal. When receiving the input signal of the abnormal sound detection target of the target machine and the operating section information indicating the actual operating and non-operating sections of the target machine corresponding to the input signal, the operating section detection subsystem estimates the operating and non-operating sections of the target machine from the input signal of the abnormal sound detection target using the learning results, and the abnormality judgment unit judges whether the input signal of the abnormal sound detection target of the target machine is normal or abnormal based on the degree of deviation between the estimated operating and non-operating sections and the actual operating and non-operating sections indicated in the operating section information.

According to the present invention, abnormal sounds can be detected with high accuracy even when background noise is mixed with the input sound.

1 is a block diagram showing the overall configuration of an abnormal sound detection system according to a first embodiment. FIG. FIG. 2 is a block diagram showing a learning subsystem in the first embodiment. FIG. 10 is a diagram showing a processing flow of a learning subsystem in the first embodiment. FIG. 2 is a block diagram illustrating an abnormal sound detection subsystem according to the first embodiment. FIG. 4 is a diagram showing a processing flow of an abnormal sound detection subsystem in the first embodiment. FIG. 1 is a diagram illustrating an example of a hardware configuration of an abnormal sound detection system according to a first embodiment. FIG. 10 is a block diagram showing the overall configuration of an abnormal sound detection system according to a second embodiment. FIG. 10 is a block diagram showing a learning subsystem in the second embodiment. FIG. 10 is a diagram showing a processing flow of a learning subsystem in the second embodiment. FIG. 10 is a block diagram illustrating an abnormal sound detection subsystem according to a second embodiment. FIG. 11 is a diagram showing a processing flow of an abnormal sound detection subsystem in the second embodiment. FIG. 10 is a diagram illustrating an example of a hardware configuration of an abnormal sound detection system according to a second embodiment. FIG. 11 is a block diagram showing a learning subsystem in a third embodiment. FIG. 11 is a diagram showing a processing flow of a learning subsystem in the third embodiment. FIG. 10 is a diagram showing a block configuration of a learning subsystem in a fourth embodiment. FIG. 10 is a diagram illustrating an example of operation section information. FIG. 10 is a diagram showing an example of an action label.

Embodiments of the present invention are described in detail below. The abnormal sound detection system of this embodiment determines an abnormality in a target machine based on the operating state of the target machine and an input signal (input sound) obtained for the target machine. The abnormal sound detection system of this embodiment detects abnormal sounds even when background noise is mixed with the input sound. In this embodiment, the system uses the learning results of a set of pairs of normal sound signals of the target machine and operating section information indicating the actual operating section of the target machine to determine whether an abnormality exists in the input signal that is the target for abnormal sound detection of the target machine.

In this embodiment, when an input signal for abnormal sound detection of a target machine is received, the learning results are used to estimate the operating section of the target machine from the input signal for abnormal sound detection, and an abnormality is determined in the input signal for abnormal sound detection of the target machine based on the degree of deviation between the estimated operating section and the actual operating section indicated in the operating section information.

The abnormal sound detection system of this embodiment can be used to detect abnormalities in target machines M used in various locations, such as power plants, steel plants, chemical plants, food plants, and machine assembly plants. Target machines M include various types of machines that generate noise during operation, such as motors, valves, pumps, solenoids, compressors, boilers, conveyors, agitators, and heating furnaces.

Example 1 will be described using Figures 1 to 6, 16, and 17. Figure 1 is a block diagram of the abnormal sound detection system in Example 1. The abnormal sound detection system reads a time waveform 100 of a normal sound signal and operation section information 200 from a training dataset database (DB) 1, and inputs them to a learning subsystem 1000.

The normal sound signal includes normal operating sounds of the target machine M and background noise. The operating interval information 200 is information that indicates at what time points in the time waveform 100 of the corresponding normal sound signal the target machine M was operating. As shown in FIG. 16, for example, the operating interval information 200 is a collection of pairs of operation start times and operation end times. The time when the target machine M is not operating is the non-operating interval of the target machine M. In the following explanation, the symbol M for the target machine will be omitted as appropriate.

Returning to Figure 1, the learning subsystem 1000 performs a learning process based on the normal sound time waveform 100 and the movement section information 200, and outputs parameters for the feature embedding unit 10 and the movement/non-movement estimation unit 11. The movement/non-movement estimation unit 11 has the function of estimating whether the target machine is moving or not. The parameters for the feature embedding unit 10 are stored in the feature embedding parameter DB2. The parameters for the movement/non-movement estimation unit 11 are stored in the movement/non-movement estimation parameter DB3.

The feature embedding unit 10 and the movement/non-movement estimation unit 11 are cascaded to form the movement segment detection subsystem 1100, which estimates movement segment information from the input signal. Therefore, by using the parameters of the feature embedding unit 10 stored in DB2 and the parameters of the movement/non-movement estimation unit 11 stored in DB3, the movement segment detection subsystem 1100 can be obtained, which estimates the correct movement segment for a normal sound signal.

The abnormal sound detection subsystem 2000 reads the parameters of the feature embedding unit 20 and the parameters of the movement/non-movement estimation unit 21 from DBs 2 and 3. The abnormal sound detection subsystem 2000 receives as input movement interval information 400 and the signal time waveform 300. The abnormal sound detection subsystem 2000 determines whether the input signal is normal or abnormal.

Details will be explained in Figure 4, but the determination of whether the input waveform signal 300 is normal or abnormal is performed as follows. First, the operating section of the target machine is estimated from the signal time waveform 300 using the operating section detection subsystem 2100, which is composed of the feature embedding unit 20 and the operating/non-operating estimation unit 21. Next, the degree of abnormality of the waveform signal 300 is calculated according to the degree of deviation between the estimated operating section and the input operating section information 400. Finally, a determination signal 500 is output according to the magnitude of the abnormality. The determination signal 500 is sent to another system (not shown). In Example 1, a signal for which the operating section of the target machine has not been estimated is determined to be abnormal.

The following describes how Example 1 solves the above problem. First, estimating the operating interval of the target machine is equivalent to distinguishing between sounds in non-operating intervals and sounds in operating intervals. Sound in non-operating intervals is background noise only. On the other hand, sounds in operating intervals are a mixture of the operating sounds of the target machine and background noise. Therefore, the operating interval detection subsystem 2100 is a model that distinguishes between background noise and mixed sounds.

When the background noise is louder than the normal operation sound, the only difference between the background noise and the mixed sound is the presence or absence of a slight normal operation sound. Therefore, the operation section detection subsystem 2100 becomes a model that captures the slight characteristics of normal operation sound. Abnormal operation sounds that deviate from these slight characteristics cannot be detected as operation sections, making it possible to detect abnormal sounds. On the other hand, the example of an abnormal sound detection system described above as a comparative example learns only the characteristics of mixed sounds and detects abnormal sounds based on this. Example 1 detects abnormal sounds using a model that directly captures the characteristics of normal operation sounds, and therefore can more accurately detect sounds that deviate from the characteristics of normal operation sounds than the abnormal sound detection system of the comparative example.

As shown in FIG. 2, the learning subsystem 1000 in Example 1 includes a time-frequency feature generation unit 12, a feature embedding unit 10, an action/non-action estimation unit 11, an action label generation unit 13, and a learning unit 14.

Figure 3 shows the processing flow of the learning subsystem 1000 in Example 1. The learning subsystem 1000 reads a set of normal sound signal time waveforms 100 and movement interval information 200 from the training dataset DB1. In the subsequent processing, in step S301, steps S302 to S306 (described below) are repeated until the number of iterations C1 becomes equal to or greater than the threshold value ThC, which is the learning termination condition.

In step S302, the time-frequency feature generator 12 converts the time waveform 100 of the normal sound signal into time-frequency domain features 110. For example, the time-frequency domain features 110 may be a power spectrogram or a logarithmic Mel power spectrogram obtained by performing a short-time Fourier transform on the time waveform.

In step S303, the action label generation unit 13 generates an action label 210 from the action interval information 200. The action label 210 is, for example, a binary vector with the same length as the number of time frames in the time-frequency domain features 110. The value of the binary vector is 1 if the target machine is operating in each time frame, and 0 if it is not operating. An example of this is shown in FIG. 17. Alternatively, a scalar value may be used instead of a vector, and 1 may be set in the central time frame of the time-frequency domain features 110 if the target machine is operating, and 0 if it is not operating. For example, if the number of time frames in the time-frequency domain features 110 is five, the action label for the third time frame is determined depending on whether the target machine is operating or not operating. Furthermore, for both of the above two action labels, if the target machine switches between operating and not operating during a single time frame, the value of the corresponding action label may be the proportion of the time frame length during which it was operating. In other words, if the target machine was operating for half the time frame length, the value of the corresponding action label is set to 0.5.

In step S304, the feature embedding unit 10 converts the time-frequency domain features 110 into intermediate features 120 for estimating action labels. The feature embedding unit 10 may be a stack of fully connected layers and activation functions, a stack of multiple convolutional layers and activation functions, or a recurrent neural network.

In step S305, the motion/non-motion estimation unit 11 uses the intermediate features 120 to output motion label estimates 130. The motion/non-motion estimation unit 11 may be, for example, a stack of fully connected layers and softmax functions.

In step S306, the learning unit 14 updates the parameters of the feature embedding unit 10 and the parameters of the action/non-action estimation unit 11 by minimizing the value of a loss function defined by the error between the action label estimate 130 and the action label 210. The error between the action label estimate 130 and the action label 210 may be, for example, the cross entropy between them. Optimization algorithms such as SGD, Momentum SGD, RMSProp, AdaGrad, AdaDelta, and Adam can be used to minimize the value of the loss function.

If the number of iterations C1 is equal to or greater than ThC in step S301 (S301: YES), in step S307, the parameters of the feature embedding unit 10 are stored in DB2 and also in the parameter DB3 of the movement/non-movement estimation unit 11, and the processing flow of the learning subsystem 1000 is terminated.

As shown in FIG. 4, the abnormal sound detection subsystem 2000 in Example 1 includes a time-frequency feature generation unit 22, a feature embedding unit 20, a movement/non-movement estimation unit 21, an movement label generation unit 23, an abnormality degree calculation unit 24, and an abnormality determination unit 25.

Figure 5 shows the processing flow of the abnormal sound detection subsystem 2000 in Example 1. In step S501, the time-frequency feature generator 22 generates time-frequency domain features 310 from the signal time waveform 300. The time-frequency feature generator 22 is the same as the time-frequency feature generator 12 of the learning subsystem 1000 shown in Figure 2.

In step S502, the action label generation unit 23 generates action labels 410 from the action section information 400. The action label generation unit 23 is the same as the action label generation unit 13 of the learning subsystem 1000 shown in Figure 2.

In step S503, the feature embedding unit 20 converts the time-frequency domain features 310 into intermediate features 320 for estimating action labels by using parameters read from the feature embedding parameter DB2.

In step S504, the movement/non-movement estimation unit 21 outputs the movement label estimation value 330 from the intermediate feature 320 by using the parameters read from the movement/non-movement estimation parameter DB 3.

In step S505, the anomaly degree calculation unit 24 calculates a predefined degree of deviation between the action label 410 and the action label estimate 330, and outputs the calculated degree of deviation as the anomaly degree 340.

The discrepancy may be, for example, the cross-entropy error between the action label and the action label estimate. Alternatively, only the labels for the interval in which the target machine is operating may be extracted from the action labels and action label estimates, and the cross-entropy between them may be used as the discrepancy. In other words, the discrepancy between the action label and the action label estimate for the interval in which the target machine is not operating may be ignored. Furthermore, the discrepancy between the action label and the action label estimate for several frames before and after the target machine switches between operating and not operating may also be ignored, and the cross-entropy error between the action label and the non-operation label for the remaining interval may be used.

In step S506, the abnormality determination unit 25 determines whether the signal time waveform 300 is normal or abnormal based on the abnormality degree 340, and outputs the determination result 500. If the abnormality degree exceeds a predetermined threshold, it is determined to be abnormal; if it does not exceed the threshold, it is determined to be normal.

Figure 6 is a diagram showing an example of the configuration of an abnormal sound detection system H1 according to Example 1. The abnormal sound detection system H1 includes a computer H2 and a microphone (hereinafter referred to as "mic") H7. There may be two or more computers H2 and microphones H7.

Microphone H7 is a device that collects operating sounds generated by the target machine for anomaly detection. Microphone H7 transmits audio data from the collected sounds to a computer. Microphone H7 is placed near the target machine. Microphone H7 may be attached to the target machine or may be attached in a location away from the target machine.

Computer H2 determines whether the sounds collected from the target machine are abnormal or normal. Computer H2 includes, for example, processor H5, main memory H3, secondary memory H4, and connection interface H6. Each piece of hardware is connected to each other via an internal path.

The connection interface H6 is an interface for connecting to an external device. The connection interface H6 is, for example, a network interface and an I/O interface. Sound captured by the microphone H7 is input to the computer H2 via the connection interface H6.

Processor H5 executes computer programs stored in main memory device H3. By executing processing in accordance with the computer programs, processor H5 operates as a functional unit (module) that realizes a specific function. In the following explanation, when processing is described using a functional unit as the subject, this indicates that processor H5 is executing a program that realizes that functional unit.

Main memory device H3 stores computer programs executed by processor H5 and data used by the computer programs. Main memory device H3 also includes a work area temporarily used by the computer programs. Main memory device H3 is, for example, DRAM (Dynamic Random Access Memory). The computer programs stored in main memory device H3 will be described later.

The computer programs and data stored in the main memory device H3 may be stored in the secondary memory device H4. In this case, the processor H5 reads the computer programs and data from the secondary memory device H4 and stores them in the main memory device H3.

The secondary storage device H4 permanently stores data. The secondary storage device H4 is, for example, an HDD (Hard Disk Drive) or an SSD (Solid State Drive). The data stored in the secondary storage device H4 will be described later.

The computer programs and data held by computer H2 will now be described. The components of the learning subsystem 1000, the block diagram of which is shown in Figure 2, and the components of the abnormal sound detection subsystem 2000, the block diagram of which is shown in Figure 4, are all computer programs stored in main memory device H3.

The training dataset DB1, which stores the data read by the learning subsystem 1000, is stored in the secondary storage device H4. The feature embedding parameter DB2, which holds the parameters of the feature embedding section and is stored by the learning subsystem 1000 and read by the abnormal sound detection subsystem 2000, is also stored in the secondary storage device H4. Similarly, the action/non-action estimation parameter DB3, which holds the parameters of the action/non-action estimation section and is stored by the learning subsystem 1000 and read by the abnormal sound detection subsystem 2000, is also stored in the secondary storage device H4.

According to Example 1, even when background noise is input overlapping with the operating sound of the target machine, it is possible to accurately determine whether the target machine is normal or abnormal.

Example 2 will be described using Figures 7 to 12. The following description, including this example, will focus on the differences from Example 1. In Example 1, the simultaneous operation information 200 input to the learning subsystem 1000 and the operation interval information 400 input to the abnormal sound detection subsystem 2000 can be automatically generated from, for example, the control signal of the target machine. Therefore, if the control signal of the target machine can be obtained, the operation interval information 200, 400 can also be obtained, and the method described in Example 1 can be implemented. On the other hand, if the control signal of the target machine cannot be obtained, the operation interval information 200, 400 must be generated manually. However, generating the operation interval information 200, 400 takes time and effort. Furthermore, generating the operation interval information 400 each time the abnormality level is calculated using the abnormal sound detection subsystem 2000 is a heavy workload.

Figure 7 shows a second embodiment that addresses this issue. As shown in Figure 7, the abnormal sound detection subsystem 2000A determines whether the time waveform 300 of the abnormal sound detection target signal is normal or abnormal, based on parameters from the outlier detection parameter DB 4 and parameters from the feature embedding parameter DB 2, without using operation section information.

As shown in FIG. 8, the learning subsystem 1000A in Example 2 includes a time-frequency feature generation unit 12, a feature embedding unit 10, a motion/non-motion estimation unit 11, an action label generation unit 13, a motion interval learning unit 15, an outlier detection unit 16, and an outlier learning unit 17.

Figure 9 shows the processing flow of the learning subsystem 1000A in Example 2. The learning subsystem 1000A reads out a set of time waveforms 100 of normal sound signals and operation interval information 200 from the training dataset DB1. The operation interval information 200 is information that indicates at what times in the time waveform 100 of the normal sound signal the target machine was operating. For example, the operation interval information 200 is a set of pairs of operation start times and operation end times.

In step S901, the following processes are sequentially repeated until the number of iterations C1 becomes greater than the threshold value ThC1: time-frequency feature generation (S902), action label generation (S903), feature embedding (S904), action/non-action estimation (S905), and parameter update of the feature embedding unit and action/non-action parameter update (S906). These steps are performed in the same manner as in Example 1.

If the learning termination condition is met in step S901 (S901: YES), proceed to step S907. The following steps S908 to S910 are repeated until it is determined in step S907 that the number of iterations C2 is greater than the threshold value ThC2.

In step S908, the time-frequency feature generation unit 12 generates time-frequency domain features 110 of a normal sound signal from the time waveform 100 of the normal sound signal. In step S909, the feature embedding unit 10 generates intermediate features 120 from the time-frequency domain features 110. In step S910, the outlier learning unit 17 updates the parameters of the outlier detection unit 16 so that outlier detection is performed using the intermediate features 120 as normal data.

The outlier detection unit 16 can use well-known models used for outlier detection, such as a Gaussian mixture distribution, an autoencoder, or a variational autoencoder. The parameter update algorithm can be any well-known algorithm used to update the parameters of each model. As will be described later, by using the outlier detection unit 16 in the abnormal sound detection subsystem 2000A, it is possible to determine whether a signal subject to abnormal sound detection is normal or abnormal without using an action label.

If it is determined in step S907 that the learning termination condition is met (S907: YES), in step S911, the parameters of the feature embedding unit 10 are saved in DB2, and the parameters of the outlier detection unit 16 are saved in DB4, and the processing flow of the learning subsystem 1000A is terminated.

As shown in FIG. 10, the abnormal sound detection subsystem 2000A in Example 2 includes a time-frequency feature generation unit 22, a feature embedding unit 20, an outlier detection unit 26, and an abnormality determination unit 25.

Figure 11 shows the processing flow of the abnormal sound detection subsystem 2000A in Example 2. In step S1101, the time-frequency feature generator 22 generates time-frequency domain features 310 of the signal from the signal's time waveform 300. This time-frequency feature generator 22 is the same as the time-frequency feature generator 22 of the learning subsystem 1000 shown in Figure 2.

In step S1102, the feature embedding unit 20 reads the parameters of the feature embedding unit from the feature embedding parameter DB2 and converts the time-frequency domain features 310 into intermediate features 320.

In step S1103, the outlier detection unit 26 reads parameters from the outlier detection parameter DB4 and outputs the anomaly degree 340 using the intermediate feature 320.

In step S1104, the abnormality determination unit 25 determines whether the signal time waveform 300 is normal or abnormal based on the abnormality degree 340, and outputs the determination result 500. If the abnormality degree exceeds a predetermined threshold, it is determined to be abnormal, and if it does not exceed the threshold, it is determined to be normal.

As described above, in Example 2, by performing outlier detection on the intermediate features 320 output by the feature embedding unit 20, it is possible to determine whether a signal subject to abnormal sound detection is normal or abnormal without using action labels.

The parameters of the feature embedding unit 20 are parameters that have been trained in the learning subsystem 1000A so that the action label 210 can be correctly estimated. Therefore, in order to correctly distinguish between action and non-action intervals, the intermediate features obtained by inputting the time-frequency features in the action interval into the feature embedding unit 20 differ significantly from the intermediate features obtained by inputting the time-frequency features in the non-action interval into the feature embedding unit 20. This means that the feature embedding unit 20 changes its output in response to subtle features of normal operation sounds. Therefore, like the first example, the second example detects abnormal sounds using a model that directly captures the features of normal operation sounds, and therefore can correctly detect sounds that deviate from the features of normal operation sounds compared to the example of an abnormal sound detection system described as a comparative example.

Figure 12 is a diagram showing an example of the configuration of an abnormal sound detection system H1A of Example 2. Similar to the configuration of the abnormal sound detection system H1 of Example 1 shown in Figure 6, the abnormal sound detection system H1A includes a computer H2 and a microphone H7. The computer H2 includes a connection interface H6, a processor H5, a main memory device H3, and a secondary memory device H4. The roles of the microphone H7, connection interface H6, processor H5, main memory device H3, and secondary memory device H4 are as described in Figure 6.

Example 3 will be described using Figures 13 and 14. Figure 13 shows the block configuration of learning subsystem 1000B in Example 3, which correctly detects abnormal sounds even when rare background noise appears in a frequency band where normal operation sounds are not present. Learning subsystem 1000B can be used as learning subsystem 1000 in Example 1 shown in Figure 1. Alternatively, by adding outlier detection unit 16 and outlier learning unit 17 included in learning subsystem 1000A in Example 2 shown in Figure 8 to Figure 13, it may be used as learning subsystem 1000A in Example 2 shown in Figure 7.

The learning subsystem 1000B of Example 3 shown in Figure 13 includes a time-frequency feature generation unit 12, a feature embedding unit 10, a movement/non-movement estimation unit 11, a frequency band utilization calculation unit 18, a movement label generation unit 13, and a learning unit 14.

Figure 14 shows the processing flow of the learning subsystem 1000B in Example 3. In step S1401, the following steps S1402 to S1407 are executed until the learning termination condition, that is, the number of iterations C3 becomes greater than the threshold value ThC3, is met.

In step S1402, the time-frequency feature generator 12 converts the time waveform 100 of the normal sound signal into time-frequency domain features 110 of the signal.

In step S1403, the action label generation unit 13 generates an action label 210 from the action section information 200.

In step S1404, the feature embedding unit 10 converts the time-frequency domain features 110 into intermediate features 120 for estimating action labels. The feature embedding unit 10 also outputs model parameters 140 for the feature embedding unit 10. The feature embedding unit 10 is a neural network that includes a fully connected layer. For example, it may be a layered structure of a fully connected layer and an activation function, or a layered structure of multiple convolutional layers and activation functions, with a fully connected layer added at the end.

In step S1405, the action/non-action estimation unit 11 uses the intermediate feature 120 to output the action label estimate 130.

In step S1406, the utilization frequency band calculation unit 18 extracts the weight matrix of the fully connected layer included in the feature embedding unit 10 from the model parameters 140, and outputs the predefined norm of the weight matrix as the learning regularization term 150. The norm is used to induce the weight matrix of the fully connected layer so that the feature embedding unit 10 has frequency band selectivity during learning.

The frequency band selectivity of the feature embedding unit 10 refers to the property of the feature embedding unit 10 to calculate the intermediate features 120 only from information on a limited frequency band of the time-frequency domain features 110. The regularization term 150 may be, for example, a mixed norm of the weight matrix of the fully connected layer that first appears in the feature embedding unit 10, defined as follows: First, each element of the weight matrix is grouped according to the frequency of the element of the time-frequency domain features 110 to be multiplied by that element. Next, the root mean square of the set of elements included in each group is calculated. Finally, the calculation results for all groups are summed to obtain the mixed norm value. Alternatively, other methods may be used to group the elements of the weight matrix. For example, frequency band groups may be defined in advance, and each element of the weight matrix may be assigned to a frequency band group that includes the frequency of the time-frequency domain features 110 to be multiplied, and then a similar norm calculation may be performed. In any case, the calculated regularization term 150 becomes smaller as the feature embedding unit 10 embeds the intermediate feature 120 only from feature values in a narrow frequency band of the time-frequency domain feature 110.

In step S1407, the learning unit 14 learns the parameters of the feature embedding unit 10 and the parameters of the action/non-action estimation unit 11 by minimizing a loss function defined by the sum of the error between the action label estimate 130 and the action label 210 and the regularization term 150.

The error between the motion label estimate 130 and the motion label 210 may be, for example, the cross entropy between them. To minimize the value of the loss function, an optimization algorithm such as SGD, Momentum SGD, RMSProp, AdaGrad, AdaDelta, or Adam can be used.

From the definition of the loss function, the learning unit 14 learns the parameters of the feature embedding unit 10 and the parameters of the movement/non-movement estimation unit 11 so that the feature embedding unit 10 has frequency band selectivity while also correctly estimating the movement label 210. Therefore, as learning progresses, the feature embedding unit 10 and the movement/non-movement estimation unit 11 will correctly estimate the movement label 210 from only a limited frequency band of the time-frequency domain features 110.

At this time, to correctly estimate the operation label 210, it is necessary to correctly detect normal operation sounds. Therefore, the frequency band that can most accurately estimate the operation label 210 is the main frequency band of normal operation sounds. Therefore, through this learning, the feature embedding unit 10 uses the main frequency band of normal operation sounds as the frequency band to use for calculating the intermediate feature 120. Therefore, the intermediate feature 120 is not affected by rare background noise that appears at frequencies different from the main frequency band of normal operation sounds. Similarly, the operation label estimate 130 is also not affected by this rare background noise. Therefore, erroneous determination of abnormal sound detection due to background noise does not occur.

In step S1401, if the number of iterations C3 becomes greater than the threshold value ThC3 and the learning termination condition is met (S1401: YES), in step S1408, the parameters of the feature embedding unit 10 are saved in DB2, and the parameters of the movement/non-movement estimation unit 11 are saved in DB3, and the processing flow of the learning subsystem 1000B is terminated.

Note that the utilization frequency band calculation unit 18 of Example 3 is not limited to application to the parameters of the model output by the feature embedding unit 10, but may also be applied to the parameters of models of other abnormal sound detection systems. For example, it may be applied to the parameters of the model used in the abnormal sound detection system of "R. Giri, "Self-supervised classification for detecting anomalous sounds," in Proc of DCASE Workshop, 2020."

Example 4 will be explained using Figure 15. In Example 3, in order to deal with background noise that rarely appears at frequencies different from the main frequency band of normal operation sounds, the learning of the feature embedding unit 10 is guided, and intermediate features 120 are created from only the features in the main frequency band of normal operation sounds among the time-frequency domain features 110. On the other hand, if the main frequency band of normal operation sounds is known in advance, the same problem can also be addressed by creating the time-frequency domain features 110 themselves from only that frequency band. Example 4 shows this method.

Figure 15 shows the block configuration of learning subsystem 1000C in Example 4. This subsystem 1000C can be used as learning subsystem 1000 in Example 1 shown in Figure 1. Alternatively, by adding the outlier detection unit 16 and outlier learning unit 17 included in learning subsystem 1000A in Example 2 shown in Figure 8 to Figure 15, it can also be used as learning subsystem 1000A in Example 2 shown in Figure 7.

As shown in FIG. 15, the learning subsystem 1000C in Example 4 includes a time-frequency feature generation unit 12, a feature embedding unit 10, an action/non-action estimation unit 11, an action label generation unit 13, and a learning unit 14.

The processing flow of the learning subsystem 1000C in Example 4 is the same as the processing flow of the learning subsystem 1000 in Example 1 shown in Figure 3.

Of the specific processes performed in the processing flow of the learning subsystem 1000C in Example 4, the only difference from the processes performed in the processing flow of the learning subsystem 1000 in Example 1 is the process performed in the "time-frequency feature generation" step in S302 of Figure 3. In Example 4, in S302, the time-frequency feature generation unit 12 outputs time-frequency domain features 110 only within the frequency band 160 of use from the time waveform 100 of the normal sound signal and the frequency band 160 of use. For example, the time-frequency domain features 110 may be a power spectrogram obtained by performing a short-time Fourier transform on the time waveform, or a logarithmic Mel power spectrogram from which only the elements included in the frequency band of use are extracted.

The abnormal sound detection system described above may be configured as follows:

(1) An abnormal sound detection system comprising an operating section detection subsystem that estimates the operating/non-operating sections of a target machine in response to input of a normal sound signal from the target machine using the learning results of a set of pairs of the target machine's normal sound signal and operating section information indicating the target machine's actual operating/non-operating sections, and an abnormality judgment unit that judges whether the input signal of the target machine's abnormal sound detection target is normal or abnormal. When receiving the input signal of the target machine's abnormal sound detection target and operating section information indicating the target machine's actual operating/non-operating sections corresponding to the input signal, the operating section detection subsystem estimates the operating/non-operating sections of the target machine from the input signal of the abnormal sound detection target using the learning results, and the abnormality judgment unit judges whether the input signal of the target machine's abnormal sound detection target is normal or abnormal based on the degree of deviation between the estimated operating/non-operating sections and the actual operating/non-operating sections indicated in the operating section information.

This makes it possible to suppress the influence of background noise and detect abnormal sounds with high accuracy, even when background noise is mixed with the input sound.

(2) In the abnormal sound detection system described in (1) above, the learning results include feature embedding parameters used to generate features from the normal sound signal of the target machine and operation/non-operation estimation parameters used to estimate the operation or non-operation of the target machine from the extracted features, and the operation section detection subsystem uses the feature embedding parameters and the operation/non-operation estimation parameters to estimate the operation or non-operation of the target machine from the normal sound signal of the target machine or the input signal of the abnormal sound detection target.

(3) The abnormal sound detection system of (2) above further comprises an outlier detection unit that calculates the degree of abnormality of the intermediate features of the input signal of the abnormal sound detection target of the target machine, generated using the feature embedding parameters, relative to the distribution of intermediate features of the normal sound signal, based on the learning results obtained by learning the intermediate features of the normal sound signal of the target machine, generated using the feature embedding parameters included in the learning results, and the abnormality determination unit determines whether the input signal of the abnormal sound detection target of the target machine is normal or abnormal based on the calculated outlier.

This makes it possible to suppress the influence of background noise and detect abnormal sounds with high accuracy, even when it is not possible to obtain the input signal for which abnormal sounds are to be detected and the corresponding operating section information.

(4) In the abnormal sound detection system described in (1) above, the normal sound signal of the target machine includes the normal operating sound of the target machine and background noise, and the learning result guides the system to estimate the operating/non-operating section from the frequency band corresponding to the normal operating sound of the target machine among the normal sound signal or the input signal of the abnormal sound detection target.

This provides an abnormal sound detection system and method that outputs a low level of abnormality for normal sounds and a high level of abnormality for abnormal sounds, even when background noise occurs at a frequency unrelated to normal operation sounds.

The present invention is not limited to the above-described examples, and includes various modifications. The above examples have been described in detail to clearly explain the present invention, and are not necessarily limited to those that include all of the described configurations. The configurations and processing methods of the embodiments can be modified as appropriate within the scope of achieving the objectives of the present invention.

Furthermore, each component of the present invention can be selected arbitrarily, and inventions incorporating selected configurations are also included in the present invention. Furthermore, the configurations described in the claims can be combined in combinations other than those explicitly stated in the claims.

1...Training dataset DB
2...Feature embedding parameter DB
3...Movement/non-movement estimation parameter DB
4...Outlier detection parameter DB
10... feature embedding unit 11... action/non-action estimation unit 12... time frequency feature generation unit 13... action label generation unit 14... learning unit 15... action interval learning unit 16... outlier detection unit 17... outlier learning unit 18... usage frequency standby calculation unit 20... feature embedding unit 21... action/non-action estimation unit 22... time frequency feature generation unit 23... action label generation unit 24... abnormality calculation unit 25... abnormality determination unit 26... outlier detection unit 100... time waveform of normal sound signal 110... time frequency feature of normal sound signal 120... intermediate feature 130... action/non-action Estimation unit 140...model parameters 150...regularization term 160...used frequency band 200...action interval information 210...action label 300...time waveform 310 of abnormal sound detection target signal...signal time-frequency domain feature 320...intermediate feature 330...action label estimate 340...abnormality degree 400...action interval information 410...action label 500...normal/abnormal 1000, 1000A, 1000B, 1000C...learning subsystem 1100...action interval detection subsystem 2000...abnormal sound detection subsystem 2100...action interval detection subsystem

H1: Anomaly detection system H2: Computer H3: Main memory device H4: Sub-memory device H5: Processor H6: Connection interface H7: Microphone

S301: Step of confirming the condition for terminating learning S302: Step of generating time-frequency features S303: Step of generating action labels S304: Step of embedding features S305: Step of estimating action/non-action S306: Step of updating parameters of the feature embedding unit and the action/non-action estimating unit S307: Step of saving parameters of the feature embedding unit and the action/non-action estimating unit in DB S501: Step of generating time-frequency features S502: Step of generating action labels S503: Step of embedding features S504: Step of estimating action/non-action S505: Step of calculating the degree of anomaly S506: Step of determining anomaly S901: Step of confirming the condition for terminating learning S902: Time frequency feature generation step S903...action label generation step S904...feature embedding step S905...action/non-action estimation step S906...parameter update step of feature embedding unit and action/non-action estimation unit S907...learning end condition confirmation step S908...time frequency feature generation step S909...feature embedding step S910...parameter update step of outlier detection unit S911...step of saving parameters of feature embedding unit and action/non-action estimation unit in DB S1101...time frequency feature generation step S1102...feature embedding step S1103...anomaly degree calculation step S1104...anomaly determination step

S1401: Step of checking learning termination conditions S1402: Step of generating time-frequency features S1403: Step of generating action labels S1404: Step of embedding features S1405: Step of estimating action/non-action S1406: Step of calculating frequency band to be used S1407: Step of updating parameters of the feature embedding unit and the action/non-action estimating unit S1408: Step of saving parameters of the feature embedding unit and the action/non-action estimating unit in DB

Claims (4)

An abnormal sound detection system comprising: an operating section detection subsystem that estimates an operating section and a non-operating section of a target machine in response to an input of a normal sound signal of the target machine, using a learning result obtained by learning a set of pairs of normal sound signals of the target machine and operating section information indicating the actual operating and non-operating sections of the target machine; and an abnormality determination unit that determines whether an input signal of an abnormal sound detection target of the target machine is normal or abnormal,
When receiving an input signal of the target machine that is the target of abnormal sound detection and the operating section information that indicates the actual operating and non-operating sections of the target machine that correspond to the input signal,
the operating section detection subsystem estimates operating and non-operating sections of the target machine from the input signal of the abnormal sound detection target using the learning result,
the abnormality determination unit determines whether an input signal of an abnormal sound detection target of the target machine is normal or abnormal based on a degree of deviation between the estimated operating and non-operating sections and the actual operating and non-operating sections indicated in the operating section information.
Abnormal sound detection system. The abnormal sound detection system according to claim 1,
the learning result includes a feature embedding parameter used to generate a feature from a normal sound signal of the target machine, and an action/non-action estimation parameter used to estimate an action/non-action of the target machine from the feature,
The operation section detection subsystem is an abnormal sound detection system that uses the feature embedding parameters and the operation/non-operation estimation parameters to estimate whether the target machine is in operation or non-operation from a normal sound signal of the target machine or an input signal of an abnormal sound detection target. An abnormal sound detection method using an abnormal sound detection system that uses a learning result obtained by learning a set of pairs of normal sound signals of a target machine and operation section information indicating the actual operation and non-operation sections of the target machine to estimate operation and non-operation sections of the target machine in response to an input of a normal sound signal of the target machine, and determines whether an input signal of an abnormal sound detection target of the target machine is normal or abnormal,
When receiving an input signal of the target machine that is the target of abnormal sound detection and the operating section information that indicates the actual operating and non-operating sections of the target machine that correspond to the input signal,
estimating an operating section and an inoperating section of the target machine from the input signal of the abnormal sound detection target using the learning result;
determining whether an input signal of an abnormal sound detection target of the target machine is normal or abnormal based on the degree of deviation between the estimated operating and non-operating sections and the actual operating and non-operating sections indicated in the operating section information;
Abnormal sound detection method. The abnormal sound detection method according to claim 3 ,
the learning result includes a feature embedding parameter used to generate a feature from a normal sound signal of the target machine, and an action/non-action estimation parameter used to estimate an action/non-action of the target machine from the feature,
An abnormal sound detection method that uses the feature embedding parameters and the operation/non-operation estimation parameters to estimate whether the target machine is in operation or not from a normal sound signal of the target machine or an input signal of an abnormal sound detection target.