CN116798448A - A sound event detection method based on lightweight CRNN model - Google Patents

Abstract

The application discloses a sound event detection method based on a lightweight CRNN model, which comprises the steps of preprocessing a sound signal and performing fast Fourier transform to enable a time domain signal of the sound signal to be transformed into a frequency domain signal; energy is obtained for each frequency point of each frame of signal in the frequency domain signal to obtain an energy spectrum; carrying out Mel filtering treatment on the energy spectrum and obtaining logarithm to obtain a logarithmic Mel diagram; inputting the logarithmic mel diagram into a lightweight CRNN model to obtain a classification result of the sound event. According to the lightweight CRNN model-based sound event detection method, the GhostNet feature extraction network in the lightweight CRNN model uses the super-parametric complex convolution layer to replace the common convolution layer in the original GhostNet feature extraction network, so that the effects of reducing parameters and reducing model complexity are achieved, and the attention mechanism is added into the GRU recurrent neural network, so that the prediction effect can be effectively improved.

Description

Sound event detection method based on lightweight CRNN model

Technical Field

The application belongs to the field of sound event detection, and particularly relates to a sound event detection method based on a lightweight CRNN model.

Background

Sound signals contain very rich and meaningful information and have been regarded as an important way for human beings to perceive information about the external environment, so researches on sound signals have been attracting attention. Sound event detection (SoundEventDetection, SED) refers to the fact that the system can detect the type of sound event present at each moment in time, based on the audio file given. The sounds referred to herein are not limited to human speech only, but include a wide variety of other sound events that exist in nature and that exhibit considerable variability in many respects.

With the rapid development of the internet of things technology, anomaly detection has become one of the hot topics in the application field of machine learning. Although most of the current methods are based on anomaly detection in the visual field, there is a great limitation in such applications that problems occur when the detected object cannot be completely covered by the camera, resulting in a blind spot where prediction cannot be performed. The problem can be effectively solved by detecting the abnormality through sound, and the voiceprint is gradually applied to the field of abnormality detection due to the advantages of stability and obvious characteristics. However, the conventional acoustic event detection model, namely the convolutional neural network (CRNN), has huge parameters and complex calculation, and is difficult to detect in real time in the embedded device, so that a network with small parameter amount and low calculation complexity is needed to be designed for fault detection.

Disclosure of Invention

The application aims to solve the problems in the background art and provides a sound event detection method based on a lightweight CRNN model.

In order to achieve the above purpose, the technical scheme adopted by the application is as follows:

the application provides a sound event detection method based on a lightweight CRNN model, which comprises the steps of preprocessing a sound signal and performing fast Fourier transform to enable a time domain signal of the sound signal to be transformed into a frequency domain signal.

And obtaining energy spectrum by obtaining energy for each frequency point of each frame of signal in the frequency domain signal.

And carrying out Mel filtering treatment on the energy spectrum and obtaining logarithm to obtain a logarithmic Mel diagram.

Inputting the logarithmic mel diagram into a lightweight CRNN model to obtain a classification result of the sound event, wherein the lightweight CRNN model comprises a GhostNet feature extraction network, a GRU recurrent neural network and an attention mechanism, and the GhostNet feature extraction network comprises a hyper-parametric complex convolution layer and a plurality of GhostBottleneck structures.

Preferably, the preprocessing operation is performed on the sound signal, including:

noise reduction, framing and windowing, endpoint detection and data enhancement processing are sequentially carried out on the sound.

Preferably, the fast fourier transform formula is as follows:

wherein ,X_i (k ') represents a frequency domain signal of a kth' sampling point of an ith frame signal, x _i (N') represents a time domain signal of an nth sampling point of the i-th frame signal, N represents a total sampling point of fourier transform, and j represents complex number.

Preferably, the formula for obtaining the energy spectrum by calculating the energy for each frequency point of each frame of signal in the frequency domain signal is as follows:

S _i (k’)＝|X _i (k’)| ² ,k’＝0,1,···,N-1

wherein ,S_i (k ') represents the energy spectrum of the kth' sampling point of the ith frame signal.

Preferably, the energy spectrum is subjected to mel filtering and logarithmically to obtain a logarithmic mel plot, including:

when the energy spectrum is subjected to the mel filtering treatment, the mel filtering treatment is performed by a mel filter bank formed by a group of filters, and the energy formula after the mel filtering is obtained is as follows:

wherein ,

wherein ,E_i,q Represents the energy after Mel filtration, H _q (K ') represents the frequency response, K, of the kth' sample point of the qth filter in the Mel filter bank _k’ The frequency of the kth sampling point is represented, Q represents the number of filters, k _q Represents the center frequency, k, of the q-th filter _q-1 Represents the center frequency, k, of the q-1 th filter _q+1 Represents the center frequency of the (q+1) th filter, m represents the Mel frequency, f represents the actual frequency, f _s Represents the sampling frequency, f _l and f_h Represents the lowest frequency and the highest frequency of the actual frequency, M (f _l) and M(f_h ) Representing the lowest and highest frequencies, respectively, of the mel frequency;

the equation for logarithmically obtaining logarithmic mel energy from the mel-filtered energy is as follows:

mbe _i,q ＝ln(E _i,q )

wherein ,mbe_i,q Represents logarithmic mel energy;

the logarithmic mel plot is obtained by plotting points according to the value of logarithmic mel energy.

Preferably, the hyper-parametric complex convolution layer comprises:

the definition of the super-parametric complex convolution layer PHC (x) is as follows:

PHC(x)＝H*x+b

wherein ,

A _i ∈R ^n×n

then the first time period of the first time period,

H∈R ^s×d×k×k

wherein ,R^s×d×k×k Representing vectors in the s×d×k×k dimensions, R ^n×n Representing a vector of dimension n x n,representation->Vector of dimensions, s represents the input dimension of the superparameter complex convolutional layer, d represents the output dimension of the superparameter complex convolutional layer, k represents the filter size, x represents the parameters input to the superparameter complex convolutional layer, b represents the bias term, A _i Expressed as algebraic rules, F _i Representing the ith filter bank, H representing the final weight matrix arranged according to algebraic rules;

when n=1, i.e. a ₁ Is a one-dimensional vector which is a vector,

when n=2, i.e. a ₂ Is a vector in a two-dimensional form,

when A is _n In the case of an n-dimensional vector,

A _i from one dimension to n dimensions, the dimension of the final weight matrix H remains unchanged, the ith filter bank F _i Becomes 1/n so that the parameters of the filter are reduced.

Preferably, the lightweight CRNN model includes a Ghostnet feature extraction network, a GRU recurrent neural network, and an attention mechanism, the Ghostnet feature extraction network including a hyper-parametric complex convolution layer and several GhostBottleneck structures, including:

the GhostNet feature extraction network comprises a network structure formed by connecting a super-parametric complex convolution layer and a plurality of Ghost Bottleneck in series, wherein the Ghost Bottleneck structure is a Bottleneck structure formed by a Ghost Module structure, the Ghost Module structure comprises a first feature map obtained by compressing the number of channels of an input picture by the super-parametric complex convolution layer, then the first feature map is subjected to depth separable convolution to obtain a plurality of second feature maps, and then the first feature map and the plurality of second feature maps are spliced to generate a complete feature map;

and (3) obtaining a plurality of third feature graphs from the logarithmic mel graph through a GhostNet feature extraction network, inputting each third feature graph into a GRU recurrent neural network to obtain feature vectors of a time sequence, inputting each feature vector into an attention mechanism to calculate weight probability, and finally obtaining a classification result through a full connection layer.

Compared with the prior art, the application has the beneficial effects that:

according to the lightweight CRNN model-based sound event detection method, the GhostNet feature extraction network in the lightweight CRNN model uses the super-parametric complex convolution layer to replace the common convolution layer in the original GhostNet feature extraction network, so that the effects of reducing parameters and reducing model complexity are achieved, and the attention mechanism is added into the GRU recurrent neural network, so that the prediction effect can be effectively improved.

Drawings

FIG. 1 is a flow chart of a sound event detection method based on a lightweight CRNN model of the present application;

FIG. 2 is a schematic diagram of the GhostModule structure of the present application;

fig. 3 is a schematic diagram of the structure of the mel filter bank of the present application.

Detailed Description

The following description of the embodiments of the present application will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.

It will be understood that when an element is referred to as being "connected" to another element, it can be directly connected to the other element or intervening elements may also be present. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the description of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

As shown in fig. 1-3, a sound event detection method based on a lightweight CRNN model includes:

s1, performing a preprocessing operation on the sound signal, and performing fast Fourier transform to enable a time domain signal of the sound signal to be transformed into a frequency domain signal.

Performing a preprocessing operation on the sound signal, comprising:

noise reduction, framing and windowing, endpoint detection and data enhancement processing are sequentially carried out on the sound.

Specifically, during noise reduction processing, a noise-carrying sound signal is decomposed into a high-frequency part and a low-frequency part, and a low-pass filter is used for filtering to separate a low-frequency effective signal; when the framing and windowing processing is carried out, the separated low-frequency effective signals are truncated into a plurality of small fragments (framing), and then the signals at the truncated positions can be smoothly transited through the windowing operation; during the endpoint detection processing, the endpoint detection is carried out on the signal after framing and windowing processing so as to remove the mute segment in the signal and keep the effective sound segment; during data enhancement processing, the data enhancement is performed by changing the sampling rate of the signal after the end point detection processing, adding white noise, reverberation, channel distortion and other modes to generate more training data, so that the generalization capability of the model is improved.

The fast fourier transform formula is as follows:

wherein ,X_i (k ') represents a frequency domain signal of a kth' sampling point of an ith frame signal, x _i (N') represents a time domain signal of an nth sampling point of the i-th frame signal, N represents a total sampling point of fourier transform, and j represents complex number.

S2, obtaining energy from each frequency point of each frame of signal in the frequency domain signal.

Specifically, the correlation formula is:

S _i (k’)＝|X _i (k’)| ² ,k’＝0,1,···,N-1

wherein ,S_i (k ') represents the energy spectrum of the kth' sampling point of the ith frame signal.

S3, carrying out Mel filtering processing on the energy spectrum and obtaining a logarithmic Mel diagram by logarithm.

Specifically, when the mel filtering process is performed on the energy spectrum, the mel filtering process is performed through a mel filter bank formed by a set of filters, and the energy formula after the mel filtering is obtained is as follows:

wherein ,

wherein ,E_i,q Represents the energy after Mel filtration, H _q (K ') represents the frequency response, K, of the kth' sample point of the qth filter in the Mel filter bank _k’ The frequency of the kth sampling point is represented, Q represents the number of filters, k _q Represents the center frequency, k, of the q-th filter _q-1 Represents the center frequency, k, of the q-1 th filter _q+1 Represents the center frequency of the (q+1) th filter, m represents the Mel frequency, f represents the actual frequency, f _s Represents the sampling frequency, f _l and f_h Represents the lowest frequency and the highest frequency of the actual frequency, M (f _l) and M(f_h ) Representing the lowest and highest frequencies, respectively, of the mel frequency;

the equation for logarithmically obtaining logarithmic mel energy from the mel-filtered energy is as follows:

mbe _i,q ＝ln(E _i,q )

wherein ,mbe_i,q Represents logarithmic mel energy;

the logarithmic mel plot is obtained by plotting points according to the value of logarithmic mel energy.

It should be noted that, as shown in fig. 3, a schematic diagram of a mel filter bank is shown, and a mel is shownThe filter bank consists of a set of filters with triangular filter characteristics, each filter being triangular, k in the figure ₁ To k ₆ To represent the center frequencies of the corresponding filters, respectively, the frequency response amplitude at the center frequency is 1 and linearly decreases toward 0 until the frequency response amplitude is 0 when the center frequencies of the two adjacent filters are reached.

S4, inputting the logarithmic Mel diagram into a lightweight CRNN model to obtain a classification result of the sound event, wherein the lightweight CRNN model comprises a GhostNet feature extraction network, a GRU recurrent neural network and an attention mechanism, and the GhostNet feature extraction network comprises a hyper-parametric complex convolution layer and a plurality of GhostBottleneck structures.

Specifically, the super-parametric complex convolution layer includes:

the definition of the super-parametric complex convolution layer PHC (x) is as follows:

PHC(x)＝H*x+b

wherein ,

A _i ∈R ^n×n

then the first time period of the first time period,

H∈R ^s×d×k×k

wherein ,R^s×d×k×k Representing vectors in the s×d×k×k dimensions, R ^n×n Representing a vector of dimension n x n,representation->Vector of dimensions, s represents the input dimension of the hyper-parametric complex convolution layer, d represents the output dimension of the hyper-parametric complex convolution layer, k represents the filter size, x representsParameters input to the hyper-parametric complex convolution layer are shown, b represents the bias term, A _i Expressed as algebraic rules, F _i Representing the ith filter bank, H representing the final weight matrix arranged according to algebraic rules;

when n=1, i.e. a ₁ Is a one-dimensional vector which is a vector,

when n=2, i.e. a ₂ Is a vector in a two-dimensional form,

when A is _n In the case of an n-dimensional vector,

A _i from one dimension to n dimensions, the dimension of the final weight matrix H remains unchanged, the ith filter bank F _i The dimension of (2) becomes 1/n, so that the parameters of the filter are reduced, thereby reducing the parameters and reducing the complexity of the model.

The lightweight CRNN model comprises a GhostNet feature extraction network, a GRU recurrent neural network and an attention mechanism, wherein the GhostNet feature extraction network comprises a hyper-parametric complex convolution layer and a plurality of Ghost Bottleneck structures, and the lightweight CRNN model comprises:

the GhostNet feature extraction network comprises a network structure formed by connecting a super-parametric complex convolution layer and a plurality of Ghost Bottleneck in series, wherein the Ghost Bottleneck structure is a Bottleneck structure formed by a Ghost Module structure, the Ghost Module structure comprises a first feature map obtained by compressing the number of channels of an input picture by the super-parametric complex convolution layer, then the first feature map is subjected to depth separable convolution to obtain a plurality of second feature maps, and then the first feature map and the plurality of second feature maps are spliced to generate a complete feature map (shown in figure 2);

and (3) obtaining a plurality of third feature graphs from the logarithmic mel graph through a GhostNet feature extraction network, inputting each third feature graph into a GRU recurrent neural network to obtain feature vectors of a time sequence, inputting each feature vector into an attention mechanism to calculate weight probability, and finally obtaining a classification result through a full connection layer.

It should be noted that, the GhostNet feature extraction network in the method uses a superparameter complex convolution layer to replace a common convolution layer in the original GhostNet feature extraction network, so as to achieve the purposes of reducing parameters and reducing complexity of a model, and the GRU (Gated Recurrent Unit, gate-controlled loop unit) is a recurrent neural network similar to an LSTM (Long Short-Term Memory network), which is generally easier to train and has fewer parameters, and the GRU can avoid the problem of gradient disappearance. The attention mechanism is added into the GRU recurrent neural network, so that the prediction effect can be effectively improved.

According to the lightweight CRNN model-based sound event detection method, the GhostNet feature extraction network in the lightweight CRNN model uses the super-parametric complex convolution layer to replace the common convolution layer in the original GhostNet feature extraction network, so that the effects of reducing parameters and reducing model complexity are achieved, and the attention mechanism is added into the GRU recurrent neural network, so that the prediction effect can be effectively improved.

The technical features of the above-described embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above-described embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.

The above-described embodiments represent only the more specific and detailed embodiments of the present application, but are not to be construed as limiting the claims. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of protection of the present application is to be determined by the appended claims.

Claims (7)

1. A sound event detection method based on a lightweight CRNN model is characterized in that: the sound event detection method based on the lightweight CRNN model comprises the following steps:

performing a preprocessing operation on the sound signal and then performing fast Fourier transform to enable a time domain signal of the sound signal to be transformed into a frequency domain signal;

energy is obtained for each frequency point of each frame of signal in the frequency domain signal to obtain an energy spectrum;

carrying out Mel filtering treatment on the energy spectrum and obtaining logarithm to obtain a logarithmic Mel diagram;

inputting the logarithmic mel diagram into a lightweight CRNN model to obtain a classification result of the sound event, wherein the lightweight CRNN model comprises a GhostNet feature extraction network, a GRU recurrent neural network and an attention mechanism, and the GhostNet feature extraction network comprises a hyper-parametric complex convolution layer and a plurality of GhostBottleneck structures.

2. The sound event detection method based on the lightweight CRNN model as set forth in claim 1, wherein: the preprocessing operation for the sound signal comprises the following steps:

noise reduction, framing and windowing, endpoint detection and data enhancement processing are sequentially carried out on the sound.

3. The sound event detection method based on the lightweight CRNN model as set forth in claim 1, wherein: the fast fourier transform formula is as follows:

wherein ,X_i (k ') represents a frequency domain signal of a kth' sampling point of an ith frame signal, x _i (N') represents a time domain signal of an nth sampling point of the i-th frame signal, N represents a total sampling point of fourier transform, and j represents complex number.

4. The sound event detection method based on the lightweight CRNN model as set forth in claim 3, wherein: the formula for obtaining the energy spectrum by solving the energy of each frequency point of each frame of signal in the frequency domain signal is as follows:

S _i (k’)＝|X _i (k’)| ² ,k’＝0,1,···,N-1

wherein ,S_i (k ') represents the energy spectrum of the kth' sampling point of the ith frame signal.

5. The sound event detection method based on the lightweight CRNN model as set forth in claim 1, wherein: the method for carrying out Mel filtering processing on the energy spectrum and obtaining the logarithm to obtain a logarithmic Mel diagram comprises the following steps:

when the energy spectrum is subjected to the mel filtering treatment, the mel filtering treatment is performed by a mel filter bank formed by a group of filters, and the energy formula after the mel filtering is obtained is as follows:

wherein ,

wherein ,E_i,q Represents the energy after Mel filtration, H _q (K ') represents the frequency response, K, of the kth' sample point of the qth filter in the Mel filter bank _k’ The frequency of the kth sampling point is represented, Q represents the number of filters, k _q Represents the center frequency, k, of the q-th filter _q-1 Represents the center frequency, k, of the q-1 th filter _q+1 Represents the center frequency of the (q+1) th filter, m represents the Mel frequency, f represents the actual frequency, f _s Represents the sampling frequency, f _l and f_h Represents the lowest frequency and the highest frequency of the actual frequency, M (f _l) and M(f_h ) Representing the lowest and highest frequencies, respectively, of the mel frequency;

the equation for logarithmically obtaining logarithmic mel energy from the mel-filtered energy is as follows:

mbe _i,q ＝ln(E _i,q )

wherein ,mbe_i,q Represents logarithmic mel energy;

the logarithmic mel plot is obtained by plotting points according to the value of logarithmic mel energy.

6. The sound event detection method based on the lightweight CRNN model as set forth in claim 1, wherein: the super-parametric complex convolution layer comprises:

the definition of the super-parametric complex convolution layer PHC (x) is as follows:

PHC(x)＝H*x+b

wherein ,

A _i ∈R ^n×n

then the first time period of the first time period,

H∈R ^s×d×k×k

wherein ,R^s×d×k×k Representing vectors in the s×d×k×k dimensions, R ^n×n Representing a vector of dimension n x n,representation ofVector of dimensions, s represents the input dimension of the superparameter complex convolutional layer, d represents the output dimension of the superparameter complex convolutional layer, k represents the filter size, x represents the parameters input to the superparameter complex convolutional layer, b represents the bias term, A _i Expressed as algebraic rules, F _i Representing the ith filter bank, H representing the final weight matrix arranged according to algebraic rules;

when n=1, i.e. a ₁ Is a one-dimensional vector which is a vector,

when n=2, i.e. a ₂ Is a vector in a two-dimensional form,

when A is _n In the case of an n-dimensional vector,

A _i from one dimension to n dimensions, the dimension of the final weight matrix H remains unchanged, the ith filter bank F _i Becomes 1/n so that the parameters of the filter are reduced.

7. The method for detecting acoustic events based on lightweight CRNN model as set forth in claim 6, wherein: the lightweight CRNN model comprises a GhostNet feature extraction network, a GRU recurrent neural network and an attention mechanism, wherein the GhostNet feature extraction network comprises a hyper-parametric complex convolution layer and a plurality of GhostBottleneck structures, and the lightweight CRNN model comprises:

the GhostNet feature extraction network comprises a network structure formed by connecting a super-parametric complex convolution layer and a plurality of GhostBottleneck in series, wherein the GhostBottleneck structure is a Bottleneck structure formed by a GhostModule structure, the GhostModule structure comprises a first feature map obtained by compressing the number of channels of an input picture by the super-parametric complex convolution layer, then the first feature map is subjected to depth separable convolution to obtain a plurality of second feature maps, and then the first feature map and the plurality of second feature maps are spliced to generate a complete feature map;

and (3) obtaining a plurality of third feature graphs from the logarithmic mel graph through a GhostNet feature extraction network, inputting each third feature graph into a GRU recurrent neural network to obtain feature vectors of a time sequence, inputting each feature vector into an attention mechanism to calculate weight probability, and finally obtaining a classification result through a full connection layer.