JP7658212B2 - Signal analysis device, signal analysis method, and signal analysis program - Google Patents

Description

The disclosed technology relates to a signal analysis device, a signal analysis method, and a signal analysis program.

Blind Source Separation (BSS) is a technology that separates and extracts individual source signals from only the observed mixed signals without using prior information such as information about the sound sources or transfer functions between the sound sources and microphones. Under overdetermined conditions where the number of microphones is greater than or equal to the number of sound sources, Independent Component Analysis (ICA), which aims to estimate a separation filter to maximize the independence between source signals, is known to be effective, and many methods that extend the principles of ICA have been proposed. Among these, methods formulated in the time-frequency domain have the advantage of being able to effectively utilize various assumptions that hold in the time-frequency domain regarding sound sources and assumptions regarding the frequency response of microphone arrays. For example, the Independent Low-Rank Matrix Analysis (ILRMA) described in Non-Patent Document 1 is based on the assumption that the power spectrogram of each sound source signal can be modeled as the product of two non-negative matrices (low-rank non-negative matrix). However, the separation performance of this method is inevitably limited for sound sources that do not follow this assumption.

In recent years, attempts have been made to improve separation accuracy by introducing deep neural network (DNN) into signal processing-based methods such as ICA. The multichannel variational autoencoder (MVAE) method described in Non-Patent Document 2 is a method in which a generation model of a sound source spectrogram represented by a conditional VAE (CVAE) is pre-trained, and the decoder input of the CVAE is estimated together with the separation matrix during separation. This method has achieved particularly high separation accuracy among methods using DNN. In this method, parameters are updated so that the likelihood function increases in each iteration, so that the convergence of the likelihood function to a stationary point is guaranteed, but there is a problem in that the error backpropagation method is used to update the decoder input value, which requires high calculation costs.

The FastMVAE method described in Non-Patent Document 3 is a method proposed for the purpose of reducing the calculation cost of the MVAE method. It uses a VAE with a classifier (Auxiliary Classifier VAE: ACVAE) to learn a classifier distribution and an encoder distribution that approximate the distribution of source classes and the posterior distribution of latent variables together with a decoder, which is a generation model of the sound source spectrogram, and predicts the decoder input value that maximizes the posterior distribution using the classifier and encoder obtained by learning. While this method can speed up the sound source separation algorithm compared to the MVAE method, separation performance tends to decrease when the conditions during testing do not match those during learning, such as in the case of unknown speakers or long reverberation.

Daichi Kitamura, Nobutaka Ono, Hiroshi Sawada, Hirokazu Kameoka, and Hiroshi Saruwatari, “Determined blind source separation unifying independent vector analysis and nonnegative matrix factorization,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 24, no.9, pp. 1626-1641, Sep. 2016. Hirokazu Kameoka, Li Li, Shota Inoue, and Shoji Makino, “Supervised Determined Source Separation with Multichannel Variational Autoencoder,” Neural Computation, vol. 31, no. 9, pp.1891-1914, Sep. 2019. Li Li, Hirokazu Kameoka, Shota Inoue, and Shoji Makino, “FastMVAE: A Fast Optimization Algorithm for the Multichannel Variational Autoencoder Method,” IEEE Access, vol. 8, pp. 228740-228753, Dec. 2020.

In the MVAE method, parameters are updated so that the log-likelihood increases with each iteration, which has the advantage of guaranteeing that the log-likelihood converges to a stationary point. However, there is an issue in that updating the parameters of the speech generation model using the backpropagation method requires a huge computational cost.

In contrast, the FastMVAE method in Non-Patent Document 3 achieves a significant increase in the speed of the sound source separation algorithm by predicting the update value of the parameter using an encoder and a classifier that have been trained in advance together with the decoder. However, since the output values of the encoder and classifier in the FastMVAE method are merely approximations of the update value in the direction of the steepest increase in the log-likelihood for the parameter, it has been experimentally confirmed that the FastMVAE method is inferior to the MVAE method in terms of sound source separation accuracy.

The disclosed technology has been made in consideration of the above points, and aims to provide a signal analysis device, method, and program that can accurately separate each component sound from a mixed signal in which the component sounds are mixed, while reducing calculation costs.

A first aspect of the present disclosure is a signal analysis device, comprising: an encoder that estimates a latent vector sequence by taking a spectrogram of a sound as input based on a spectrogram for each constituent sound and an attribute class indicating an attribute of the constituent sound; a classifier that identifies an attribute class indicating an attribute of the sound by taking the spectrogram of the sound as input; and a decoder that generates a variance of the spectrogram of the sound by taking the latent vector sequence and the attribute class as input; and a learning unit that learns the following: an encoder that estimates a latent vector sequence for each constituent sound separated by the separation matrix by the trained encoder by taking an observation signal in which the constituent sounds are mixed as input; The device includes: the attribute class identified by the trained classifier for each of the constituent sounds separated by the separation matrix; a spectrogram of the constituent sounds calculated from the variance of the spectrogram of the constituent sounds and a scale parameter generated by the trained decoder for each of the constituent sounds; a scale parameter of the spectrogram of each of the constituent sounds; a separation matrix for separating a mixed sound in which the constituent sounds are mixed in the time-frequency domain into each of the constituent sounds; and a parameter estimation unit that estimates the separation matrix and the scale parameter so as to optimize an objective function represented by using a signal obtained by separating the observed signal into each of the constituent sounds.

A second aspect of the present disclosure is a signal analysis method, in which a learning unit learns an encoder that estimates a latent vector sequence by taking a spectrogram of a sound as an input based on a spectrogram for each constituent sound and an attribute class indicating an attribute of the constituent sound, a classifier that receives the spectrogram of the sound as an input and identifies an attribute class indicating an attribute of the sound, and a decoder that receives the latent vector sequence and the attribute class as input and generates a variance of the spectrogram of the sound, and a parameter estimation unit that receives an observation signal in which the constituent sounds are mixed and estimates a variance for each constituent sound separated by the separation matrix by the trained encoder. The separation matrix and the scale parameter are estimated so as to optimize an objective function represented by the latent vector sequence, the attribute class identified by the trained classifier for each of the constituent sounds separated by the separation matrix, the spectrogram of the constituent sounds calculated from the variance of the spectrogram of the constituent sounds and the scale parameter generated by the trained decoder for each of the constituent sounds, the scale parameter of the spectrogram of each of the constituent sounds, the separation matrix for separating a mixed sound in which the constituent sounds are mixed in the time-frequency domain into each of the constituent sounds, and the signal obtained by separating the observed signal into each of the constituent sounds.

A third aspect of the present disclosure is a program for causing a computer to function as the signal analysis device of the first aspect.

The disclosed technology has the effect of reducing calculation costs and accurately separating each component sound from a mixed signal in which each component sound is mixed.

FIG. 2 is a conceptual diagram for explaining the configurations of a teacher encoder and a teacher decoder according to the present embodiment. FIG. 2 is a conceptual diagram for explaining the configurations of an encoder, a discriminator, and a decoder according to the present embodiment. 3A to 3C are diagrams illustrating examples of the configuration of an encoder and a discriminator, and an example of the configuration of a decoder according to the present embodiment. FIG. 2 is a schematic block diagram of an example of a computer that functions as a signal analysis device according to the present embodiment. 1 is a block diagram showing a configuration of a signal analysis device according to an embodiment of the present invention; 4 is a flowchart showing a learning processing routine in the signal analyzing device of the present embodiment. 4 is a flowchart showing a signal analysis processing routine in the signal analyzing device of the present embodiment. FIG. 1 is a diagram showing the arrangement of microphones and sound sources in an experimental example. FIG. 11 is a diagram showing the calculation time in each iteration according to the method of this embodiment and the conventional method.

Below, an example of an embodiment of the disclosed technology will be described with reference to the drawings. Note that the same reference symbols are used for identical or equivalent components and parts in each drawing. Also, the dimensional ratios in the drawings have been exaggerated for the convenience of explanation and may differ from the actual ratios.

<Outline of this embodiment>
First, an overview of this embodiment will be described.

In this embodiment, the encoder and classifier in the FastMVAE method are integrated into a single multitask neural network (NN), thereby achieving even greater speed. In addition, knowledge distillation (KD) is performed on the multitask NN and decoder so that their respective output distributions are as close as possible to the respective output distributions of the encoder and decoder acquired by pre-learning in the MVAE method, thereby making each NN behave in a manner similar to the parameter update of the speech generation model in the MVAE method, thereby approaching the high separation accuracy of the MVAE method. These ideas enable faster and more accurate sound source separation to be achieved even for unknown speakers compared to the conventional FastMVAE method.

<Principle of this embodiment>
<Formulation of multi-channel sound source separation problem under overdetermined conditions>
Consider the case where signals coming from J sound sources are observed by I microphones. The complex spectrograms of the observed signal of microphone i and the signal of sound source j are x _i (f, n) and s _j (f, n), respectively. Also, let us define a vector with these as elements as

(1)

(2)
However, here we consider the overdetermination condition of I=J, where ( ) ^T represents transposition, and f and n are frequency and time indices, respectively.

Under the condition of I = J, the instantaneous separation system is expressed as the relation between the vector s(f,n) of the complex spectrogram of the sound source signal and the vector x(f,n) of the observed signal.

(3)

(4)
It can be assumed that W ^H (f) is the separation matrix and ( ) ^H is the Hermitian transpose.

Under the above assumption of an instantaneous mixture system, the complex spectrogram s _j (f,n) of the sound source j has a mean of 0 and a variance of

Complex normal distribution of

(5)
If each source signal s _j (f,n) and s _{j '} (f,n), j ≠ j ' are statistically independent, then the source signal s(f,n) is

(6)
Follow.

Here, V(f,n) is a diagonal matrix having elements v ₁ (f,n), ..., v _I (f,n). From equations (3) and (6), the observed signal x is

(7)
Follow.

Therefore, the log-likelihood function of the separation matrix W = {W(f)} _f and the power spectrogram V = {v j (f,n)} j, f, _n for the observed signal X = { _x (f,n)} _{f, n} is

(8)
Here, = ^c represents an equality sign related only to the terms that depend on the parameters. When there is no constraint on the source power spectrogram v _j (f, n), Equation (8) is decomposed into terms for each frequency f, so that the index of the separated signal obtained by W calculated based on Equation (8) has arbitrariness of permutation. When v _j (f, n) has a structural constraint in the frequency direction, it is possible to derive an approach that simultaneously solves permutation matching and source separation by utilizing the constraint. Examples include the ILRMA and MVAE methods.

<Prior art 1: MVAE method>
In the MVAE method, the decoder distribution of the CVAE with the source class label as an auxiliary input is used as a generation model of the complex spectrogram of each source. The complex spectrogram of a certain source signal is S = {s(f,n)} _f,n , and the corresponding source class label is a one-hot vector c. Figure 1 shows a conceptual diagram of the CVAE. ^The CVAE learns the NN parameters φ and _θ of the encoder and decoder so that the encoder distribution q ^* _φ (z | S, c) and the decoder distribution p ^* _θ (S | z, c) are consistent, that is, so that the posterior distribution p ^* _θ (z | S, c) ∝ p ^* _θ (S | z, c) p (z) derived from q ^* φ (z | S, c) and p * _θ (S | z, c) matches as much as possible. Here, the decoder distribution of the CVAE is a probability model with the same shape as the local Gaussian source model of Equation (5).

(9)

(10)
where the variance σ ^* _θ ² (f,n;z,c) is the output of the decoder network, and g is a variable representing the scale of the power spectrogram.

On the other hand, the encoder distribution q ^* _φ (z|S, c) follows the standard normal distribution, as in the case of ordinary CVAE.

(11)
Assume that.

Here, μ ^* _φ (S,c) and σ ^* _φ2 (S,c) are the encoder outputs. The CVAE parameters θ and φ are calculated using the training samples {S _m ^, c _m } ^M _m=1 of the complex spectrograms of the sound source signals of various classes.

(12)
is trained to maximize

represents the sample average of the learning sample, and KL[·||·] is the Kullback-Leibler (KL) divergence. The decoder distribution p ^* _θ (S|z,c,g) learned as described above is called the CVAE sound source model. The CVAE sound source model is a generative model capable of expressing complex spectrograms of various classes of sound sources contained in the learning sample, and c can be regarded as a variable that adjusts the categorical features of the sound source class, and z can be regarded as a variable that adjusts the variation within the class.

By expressing the generation model of the complex spectrogram _Sj ={ _sj (f,n)} _f,n of the sound source j by a decoder distribution with _zj , _cj , and _gj as inputs, the likelihood function of the parameters of the sound source model can be reduced to a likelihood function of the same form as that of formula (8). Therefore, by iteratively updating the separation matrix W, the CVAE sound source model parameters Ψ={zj, _cj } _j , and the scale parameter G={ _{gj }j so} that the likelihood function of formula (8) becomes large, it is possible to search for a stationary point of formula (8). To update W to raise formula (8), iterative projection (IP) is used as in ILRMA.

(13)

(14)
can be used.

however,

where e _j is the j-th column vector of the I × I unit matrix. The update of Ψ, which raises equation (8), is performed using the backpropagation method, and the update of G is performed using

(15)
Here, equation (15) is an update equation that maximizes equation (8) when W and Ψ are fixed. From the above, the inference process of MVAE can be summarized as follows.

1. Learn θ and φ using equation (12) as a criterion.
2. Initialize W to be the identity matrix and initialize Ψ.
3. Repeat steps a through c below for each j.
(Step a) Update {w _j (f)} _j,f according to equations (13) and (14).
(Step b) Update Ψ _j ={z _j , c _j } by the backpropagation method.
(Step c) Update _gj using equation (15).

<Conventional technique 2: Fast MVAE method>
In the MVAE method, since the parameters are updated so that the log likelihood increases in each iteration, the convergence of the log likelihood to a stationary point is guaranteed. However, there is a problem in that a large calculation cost is required to update the parameters z _j and c _j that maximize p _θ (z _j , c _j |S _j ) by the backpropagation method. In the FastMVAE method of Non-Patent Document 3, the posterior distribution p _θ (z, c |S) is decomposed into a product of two conditional distributions such as p _θ (z |S, c) p _θ (c |S), and the distributions q ^* _φ (z |S, c) and r ^* _ψ (c |S) are expressed by NNs to approximate each distribution, and pre-trained. As a result, the parameter search by the backpropagation method in the MVAE method can be replaced by the forward calculation of each NN, enabling high-speed inference. However, since the output values of the encoder q ^* _φ (z|S, c) and the classifier r ^* _ψ (c|S) in the FastMVAE method are merely approximations of the updated value in the direction of the steepest increase of the log-likelihood for the parameters, it has been experimentally confirmed that the FastMVAE method is inferior to the MVAE method in terms of sound source separation accuracy.

<Method of the Present Embodiment>
In the FastMVAE2 method used in this embodiment, it is first assumed that the latent variable z and the attribute class c of the sound source are conditionally independent. This is equivalent to assuming that the speaker information c and the information z on the speech content are independent under a given spectrogram S. In other words, it differs from the conventional method in that it is assumed that the posterior probability p _θ (z, c|S) can be expressed as p _θ (z|S) p _θ (c|S). If an approximation distribution of these two conditional distributions is obtained, it is possible to perform parameter search at high speed by forward calculation of NN, as in the FastMVAE method.

<Chimera ACVAE sound source model>
ACVAE is an extension of CVAE, which was originally proposed for the purpose of applying it to speech conversion. It is a method of training the encoder and decoder using the mutual information I(c, S|z) between the decoder output and the class label c as a regularization term in order to emphasize the influence of the input class label c on the decoder output. It is not easy to directly optimize a criterion including I(c, S|z), but the original criterion can be indirectly increased by introducing a variational lower bound as in the case of CVAE training and raising the criterion that combines the variational lower bound and J(φ, θ). I(c, S|z) is given as the sum of the expected value of log p(c|S) and a constant, but replacing p(c|S) with an appropriate auxiliary distribution r(c|S) becomes the lower bound of I(c, S|z). By modeling this auxiliary distribution r(c|S) with a NN with parameter _ψ , ψ can be trained together with φ and θ using the above lower bound as a criterion. The auxiliary distribution represented by a NN of parameters ψ is represented as r _ψ (c|S) and is called a classifier.

In contrast, the "ChimeraACVAE" of this embodiment is a model in which the ACVAE encoder and discriminator are expressed as an integrated multitask NN. In other words, it is a model in which the distributions q ⁺ _φ (z|S) and r ⁺ _ψ (c|S) of z and c are simultaneously inferred from the spectrogram S. A conceptual diagram of ChimeraACVAE is shown in FIG. 2.

ChimeraACVAE is structured to extract the latent variable z only from the input spectrogram, making it possible to avoid inference errors in z caused by estimation errors in the class label c. In addition, since it can be described with a more compact network structure than conventional ACVAE models, it is expected to enable faster inference.

The criterion for training the ChimeraACVAE, i.e., the objective function to be maximized with respect to the NN parameters θ, φ, and ψ, is the CVAE training criterion:

(16)
and mutual information

(17)
In addition, since the labeled training sample {S _m , c _m } ^M _m can also be used for training, _the negative _cross entropy

(18)
can also be included in the criteria for learning. Up to this point, the method is the same as the conventional ACVAE except for the model structure.

However, while the ACVAE trained according to the above criteria was capable of highly accurate inference when the test conditions and training conditions matched, when they did not match, the estimated latent variables tended to deviate from the assumed distribution, and the model's generalization ability was insufficient.

Therefore, in order to improve the generalization ability of the model, the following criteria and knowledge distillation are used in addition to the above criteria in the learning of ChimeraACVAE. In ChimeraACVAE, the spectrogram S can be reconstructed using the estimated class information. Since this process is also used during inference, it is believed that training a model using a criterion for evaluating the accuracy of the spectrogram S reconstructed by the same process will lead to improved accuracy during inference. Therefore, the reconstruction criterion of equation (19) to be maximized and the class identification criterion of equation (20) are also included in the criteria for learning.

(19)

(20)

Alternatively, instead of these criteria, approximations may be used to simplify implementation.

(21)

(22)
However,

It is.

Knowledge Distillation (KD) is a methodology for using a large NN trained with a large amount of data in advance as a teacher model and inheriting the knowledge to a student model having a lightweight or different NN structure, and it is known that a student model with high generalization ability can be obtained. Here, we consider using a CVAE model that can achieve high separation accuracy even for unknown speakers as a teacher model, and inheriting the knowledge of the distribution of latent variables q ^* _φ (z|S,c) and the spectrogram generation model p ^* _θ (S|z,c) learned by the CVAE to the student model, ChimeraACVAE. Specifically, the distribution of latent variables inferred by CVAE q ^* _φ (z|S,c) and the normal distribution N(0, diag(σ ^{* θ2} (z,c))) using the variance σ ^* _φ2 output by the decoder are _set as the output distribution q ⁺ _φ (z|S) of the ^student model and the prior distribution of the normal distribution using the decoder output σ ⁺ _φ2 ^, respectively, and the student model is trained so that its output approaches the prior distribution. However, the degree of divergence between the distributions of the teacher model and the student model is measured using the KL divergence and used as the knowledge distillation criterion as shown in equations (23) to (25).

(23)

(24)

(25)

From the above, the criterion to be maximized when learning ChimeraACVAE is

(26)
Here, λ is a non-negative value and is a weighting coefficient for each criterion. Figure 2 shows a conceptual diagram of learning of ChimeraACVAE using knowledge distillation.

Figure 3 shows an example of the network structure of ChimeraACVAE. Each layer of the encoder and classifier is composed of a convolutional layer, Layer Normalization (LN), and Sigmoid Linear Unit (SiLU), while each layer of the decoder is composed of a deconvolutional layer, LN, and SiLU. By using LN, it is possible to avoid inconsistencies in the normalization calculation method during learning and inference. SiLU is a data-driven activation function that uses gates to control the information passed between layers, similar to the Gated Linear Unit (GLU) used in the CVA sound source model, and it is possible to halve the number of parameters of GLU.

<FastMVAE2 method: high-speed inference algorithm>
By using the encoder and classifier trained by ChimeraACVAE, the maximization step of _pθ ( _zj , _cj | _Sj ) in the conventional MVAE method can be replaced with the forward calculation of q ⁺ _φ ( _zj | _Sj ) and r ⁺ _ψ ( _cj | _Sj ). Therefore, the following algorithm is obtained. This is called the FastMVAE2 method.

1. Learn θ, φ, and ψ using equation (26) as the learning criterion.
2. Initialize W to be the identity matrix.
3. Repeat steps a through c below for each j.
(Step a) Update {w _j (f)} _j,f according to equations (13) and (14).
(Step b) The spectrogram separated using W is input, and z _j and c _j are updated to the mean of the Gaussian distribution output from the encoder and the output value (continuous value vector) of the discriminator, respectively.
(Step c) Update _gj using equation (15).

<Configuration of the signal analysis device according to this embodiment>
FIG. 4 is a block diagram showing a hardware configuration of the signal analyzing device 100 of the present embodiment.

As shown in FIG. 4, the signal analysis device 100 has a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a display unit 16, and a communication interface (I/F) 17. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

The CPU 11 is a central processing unit that executes various programs and controls each part. That is, the CPU 11 reads a program from the ROM 12 or storage 14, and executes the program using the RAM 13 as a working area. The CPU 11 controls each of the above components and performs various calculation processes according to the program stored in the ROM 12 or storage 14. In this embodiment, the ROM 12 or storage 14 stores a learning program for executing the learning process, and a signal analysis program for executing the signal analysis process. The learning program and the signal analysis program may be a single program, or may be a group of programs consisting of multiple programs or modules.

The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is composed of a HDD (Hard Disk Drive) or SSD (Solid State Drive) and stores various programs including the operating system and various data.

The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used to perform various input operations.

The input unit 15 receives, as learning data, time series data of the signal of each of the multiple constituent sounds and an attribute class indicating the attribute of the signal of the multiple constituent sounds. The input unit 15 also receives, as data to be analyzed, time series data of a mixed signal (hereinafter, observed signal) in which multiple constituent sounds are mixed. Note that the attribute class indicating the attribute of the signal of the constituent sounds may be manually provided. Furthermore, examples of the attributes of the signal of the constituent sounds include gender, adult/child, speaker ID, etc.

The display unit 16 is, for example, a liquid crystal display, and displays various information. The display unit 16 may also function as the input unit 15 by adopting a touch panel system.

The communication interface 17 is an interface for communicating with other devices, and uses standards such as Ethernet (registered trademark), FDDI, and Wi-Fi (registered trademark).

Next, the functional configuration of the signal analysis device 100 will be described. FIG. 5 is a block diagram showing an example of the functional configuration of the signal analysis device 100.

As shown in FIG. 5, the signal analysis device 100 is functionally configured to include a time-frequency expansion unit 24, a teacher learning unit 30, a learning unit 32, a sound source signal model storage unit 34, a parameter estimation unit 36, and an output unit 38.

For each constituent sound, the time-frequency expansion unit 24 calculates a power spectrogram representing the spectrum at each time based on the time series data of the signal of the constituent sound. The time-frequency expansion unit 24 also calculates a power spectrogram representing the spectrum at each time based on the time series data of the observed signal. In this embodiment, time-frequency expansion such as short-time Fourier transform and wavelet transform is performed.

The teacher learning unit 30 learns a teacher encoder that estimates a latent vector sequence using the spectrogram and attribute class of the sound as input, based on the spectrogram and attribute class of each constituent sound input as learning data, and a teacher decoder that generates the variance of the sound spectrogram using the latent vector sequence and attribute class as input, and stores them in the sound source signal model storage unit 34.

Specifically, the teacher learning unit 30 learns the teacher encoder and teacher decoder so as to maximize the value of the objective function of the above formula (12), which is expressed for each constituent sound using the error between the power spectrogram generated by the teacher decoder and the power spectrogram in the signal of the original constituent sound, and the distance between the latent vector sequence estimated by the teacher encoder and the latent vector sequence in the signal of the original constituent sound, and stores the trained results in the sound source signal model storage unit 34. Here, each of the teacher encoder and teacher decoder is configured using a convolutional network or a recurrent network.

The learning unit 32 learns an encoder that estimates a latent vector sequence using the spectrogram of the sound as input based on the spectrogram and attribute class for each constituent sound input as learning data, a classifier that identifies the attribute class using the spectrogram of the sound as input, and a decoder that generates the variance of the sound spectrogram using the latent vector sequence and attribute class as input.

Specifically, the learning unit 32 learns the encoder, the discriminator, and the decoder so as to maximize the criteria of the above formula (26), which includes a learning criterion for evaluating the output of the encoder and the output of the decoder, the mutual information of the decoder output and the attribute class, a reconstruction criterion for evaluating a spectrogram generated using the output of the decoder to which the output of the encoder and the output of the discriminator are input, a class discrimination criterion for evaluating the output of the discriminator to which the spectrogram generated using the output of the decoder to which the output of the encoder and the output of the discriminator are input, and a knowledge distillation criterion for matching the output of the encoder and the output of the trained teacher encoder, and for matching the output of the decoder and the output of the trained teacher decoder, and stores the encoder, the discriminator, and the decoder in the sound source signal model storage unit 34. Here, the encoder and the discriminator are an integrated neural network, and the encoder and the discriminator share some layers. In addition, each of the encoder, the discriminator, and the decoder is configured using a convolutional network or a recurrent network.

The parameter estimation unit 36 estimates the separation matrix and the scale parameter based on the power spectrogram of the observed signal, using the observed signal in which each component sound is mixed as input, so as to optimize the objective function of the above formula (8) expressed using the latent vector sequence estimated for each component sound separated by the separation matrix by the trained encoder, the attribute class identified for each component sound separated by the separation matrix by the trained classifier, the spectrogram of each component sound calculated from the variance of the spectrogram of the component sound and the scale parameter generated by the trained decoder for each component sound, the scale parameter of the spectrogram of each component sound, the separation matrix for separating the mixed sound in which each component sound is mixed in the time-frequency domain into each component sound, and the signal in which the observed signal is separated into each component sound.

Specifically, the parameter estimation unit 36 includes an initial value setting unit 40, a separation matrix update unit 42, a latent variable class update unit 44, a scale parameter update unit 46, and a convergence determination unit 48.

The initial value setting unit 40 sets initial values for the separation matrix, the latent vector sequence of each constituent tone, the attribute class of each constituent tone, and the scale parameter of each constituent tone.

The separation matrix update unit 42 updates the separation matrix according to the above formulas (13) and (14) based on the power spectrogram of the observed signal, and the previously updated or initialized latent vector series of each constituent sound, the attribute class of each constituent sound, the scale parameter of each constituent sound, and the separation matrix, so as to increase the objective function shown in the above formula (8).

The latent variable class update unit 44 updates the latent vector series of each constituent sound obtained by using the average of the Gaussian distribution of the output of the encoder to which the power spectrogram of each constituent sound obtained by using the observation signal and the separation matrix is input, and updates the attribute class of each constituent sound by using the output of the classifier to which the power spectrogram of each constituent sound obtained by using the observation signal and the separation matrix is input.

The scale parameter update unit 46 updates the scale parameters according to the above formula (15) based on the power spectrogram of the observed signal, the updated latent vector series of each constituent sound, the attribute class of each constituent sound, the scale parameters of each constituent sound, and the separation matrix, so as to increase the objective function shown in the above formula (8).

The convergence determination unit 48 determines whether the convergence condition is satisfied, and repeats the update process in the separation matrix update unit 42, the update process in the latent variable class update unit 44, and the update process in the scale parameter update unit 46 until the convergence condition is satisfied.

The convergence condition can be, for example, that the number of iterations has reached an upper limit. Alternatively, the convergence condition can be that the difference between the value of the objective function in formula (8) above and the value of the previous objective function is equal to or less than a predetermined threshold.

The output unit 38 obtains a power spectrogram of each constituent sound generated using a decoder based on the latent vector series of each constituent sound, the attribute class of each constituent sound, and the scale parameter of each constituent sound acquired by the parameter estimation unit 36, and generates and outputs a signal of each constituent sound from the power spectrogram of each constituent sound.

<Function of the signal analysis device according to the present embodiment>
Next, the operation of the signal analyzing device 100 according to this embodiment will be described.

Figure 6 is a flowchart showing the flow of the learning process by the signal analysis device 100. The learning process is performed by the CPU 11 reading out a learning program from the ROM 12 or storage 14, expanding it into the RAM 13, and executing it. In addition, for each of a plurality of constituent tones, time series data of the signal of that constituent tone and an attribute class indicating the attribute of the signal of that constituent tone are input to the signal analysis device 100 as learning data.

First, in step S100, the CPU 11, functioning as the time-frequency expansion unit 24, calculates a power spectrogram representing the spectrum at each time for each constituent sound based on the time series data of the signal of that constituent sound.

In the next step S102, the CPU 11, as the teacher learning unit 30, learns a teacher encoder that estimates a latent vector sequence using the spectrogram and attribute class of the sound as input, based on the spectrogram and attribute class for each constituent sound input as learning data, and a teacher decoder that generates the variance of the sound spectrogram using the latent vector sequence and attribute class as input.

In step S104, the CPU 11, as the learning unit 32, learns an encoder that estimates a latent vector sequence using the spectrogram of the sound as input based on the spectrogram and attribute class for each constituent sound input as learning data, a discriminator that identifies the attribute class using the spectrogram of the sound as input, and a decoder that generates the variance of the sound spectrogram using the latent vector sequence and attribute class as input, and stores the trained parameters of the encoder, discriminator, and decoder in the sound source signal model storage unit 34.

Figure 7 is a flowchart showing the flow of signal analysis processing by the signal analysis device 100. The CPU 11 reads out a signal analysis program from the ROM 12 or storage 14, deploys it in the RAM 13, and executes it to perform signal analysis processing. In addition, time-series data of an observed signal in which each component sound is mixed is input to the signal analysis device 100.

First, in step S120, the CPU 11, functioning as the time-frequency expansion unit 24, calculates a power spectrogram representing the spectrum at each time based on the time series data of the observed signal.

In step S122, the CPU 11, as the initial value setting unit 40, sets initial values for the separation matrix, the latent vector series of each constituent tone, the attribute class of each constituent tone, and the scale parameter of each constituent tone.

In step S124, the CPU 11, as the separation matrix update unit 42, updates the separation matrix in accordance with the above formulas (13) and (14) based on the power spectrogram of the observed signal calculated in the above step S120, and the previously updated or initialized latent vector series of each constituent sound, the attribute class of each constituent sound, the scale parameter of each constituent sound, and the separation matrix, so as to increase the objective function shown in the above formula (8).

In step S126, the CPU 11, as the latent variable class update unit 44, updates the latent vector series of each constituent sound to a latent vector series of each constituent sound obtained by using the average of the Gaussian distribution of the output of the encoder to which the power spectrogram of each constituent sound obtained by using the observation signal and the separation matrix is input, and updates the attribute class of each constituent sound by using the output of the classifier to which the power spectrogram of each constituent sound obtained by using the observation signal and the separation matrix is input.

In step S128, the CPU 11, as the scale parameter update unit 46, updates the scale parameters according to the above formula (15) based on the power spectrogram of the observed signal calculated in the above step S120, the updated latent vector series of each constituent sound, the attribute class of each constituent sound, the scale parameters of each constituent sound, and the separation matrix, so as to increase the objective function shown in the above formula (8).

Next, in step S130, it is determined whether the convergence condition is satisfied. If the convergence condition is satisfied, the process proceeds to step S132. If the convergence condition is not satisfied, the process proceeds to step S124, and the processes of steps S124 to S128 are repeated.

In step S132, a power spectrogram of each constituent tone is generated using a decoder based on the latent vector series of each constituent tone, the attribute class of each constituent tone, and the scale parameter of each constituent tone, which were finally updated in steps S124 to S128 above. A signal of each constituent tone is generated from the power spectrogram of each constituent tone and output from the output unit 38, and the signal analysis process is terminated.

<Experimental Results>
In order to verify the sound source separation performance by the method of this embodiment, a speaker-dependent separation experiment using the Voice Conversion Challenge (VCC) 2018 speech database and an arbitrary speaker separation experiment using the WSJ0 speech database were performed. The comparison targets were the ILRMA described in Non-Patent Document 1, the MVAE method described in Non-Patent Document 2, and the FastMVAE method described in Non-Patent Document 3, and the source-to-distortions ratio (SDR), source-to-interferences ratio (SIR), and sources-to-artifacts ratio (SAR) were used as evaluation criteria. In all methods, the separation matrix W(f) was initialized to a unit matrix and updated 60 times.

The basis number for ILRMA was set to 2. Table 1 shows the number of parameters for each model.

ChimeraACVAE was able to reduce the number of parameters by up to 40% compared to ACVAE. The experimental results are shown in Table 2.

Under all conditions, the method of this embodiment (FastMVAE2 method) showed higher separation performance than the ILRMA and FastMVAE methods, significantly narrowing the gap with the MVAE method.

To evaluate the separation performance and calculation time of each method when the number of sound sources is more than two, a mixed signal with the number of sound sources {2, 3, 6, 9} was created using the speech of 18 speakers from the WSJ0 speech database. The impulse response was created using the mirror method, and the wall reflection coefficient was set to 0.2. Figure 8 shows the arrangement of the microphones and sound sources. Ten mixed signals were created for each condition. All processing was performed using an Intel(R) Xeon(R) Gold 6130 CPU @ 2.10 GHz and a Tesla V100 GPU. Table 3 shows the average SDR for each condition.

Figure 9 also shows the calculation time for each iteration of each method. Performance improvements were confirmed in the methods of this embodiment (FastMVAE2, FastMVAE2_CPU, FastMVAE2_GPU). It was also confirmed that the method of this embodiment can achieve separation in the same calculation time as ILRMA when there are three or fewer sound sources, and can achieve separation in a shorter calculation time than ILRMA when there are three or more sound sources.

As described above, the signal analysis device according to the present embodiment learns an encoder that estimates a latent vector sequence using a spectrogram of a sound as an input based on the spectrogram and attribute class for each constituent sound, a classifier that identifies an attribute class indicating the attribute of the sound using the spectrogram of the sound as an input, and a decoder that generates the variance of the spectrogram of the sound using the latent vector sequence and the attribute class as an input. Then, the signal analysis device estimates a separation matrix and a scale parameter so as to optimize an objective function expressed using an observation signal in which each constituent sound is mixed, the latent vector sequence estimated for each constituent sound separated by a separation matrix by the learned encoder, the attribute class identified for each constituent sound separated by a separation matrix by the learned classifier, the variance of the spectrogram of the constituent sound generated by the learned decoder for each constituent sound, the scale parameter of the spectrogram of each constituent sound, the separation matrix for separating the mixed sound in which each constituent sound is mixed into each constituent sound in the time-frequency domain, and the signal obtained by separating the observation signal into each constituent sound. This reduces computational costs and enables accurate separation of each component sound from a mixed signal in which the components are mixed.

<Modification>
The present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the spirit and scope of the present invention.

For example, the case where the power spectrogram of the observed signal and the power spectrogram of the constituent sounds are calculated has been described as an example, but the present invention is not limited to this, and the amplitude spectrogram of the observed signal and the amplitude spectrogram of the constituent sounds may be calculated. In this case, the learning unit 32 learns an encoder that estimates a latent vector sequence using the amplitude spectrogram of the sound as input based on the amplitude spectrogram and attribute class of each constituent sound, a discriminator that identifies the attribute class using the amplitude spectrogram of the sound as input, and a decoder that generates the variance of the amplitude spectrogram of the sound using the latent vector sequence and the attribute class as input. In addition, the parameter estimation unit 36 estimates the separation matrix and scale parameters so as to optimize an objective function represented by an input of the observation signal, the latent vector sequence estimated by the trained encoder, the attribute class identified by the trained classifier, the amplitude spectrograms of the constituent sounds calculated from the variances of the amplitude spectrograms of the constituent sounds and the scale parameters generated by the trained decoder for each constituent sound, the scale parameters of the amplitude spectrograms of each constituent sound, a separation matrix for separating a mixed sound in which the constituent sounds are mixed in the time-frequency domain into each constituent sound, and a signal obtained by separating the observation signal into each constituent sound.

The order in which the parameters are updated can be determined arbitrarily and is not limited to the order shown in the above embodiment.

In addition, various processes that the CPU reads and executes software (programs) in each of the above embodiments may be executed by various processors other than the CPU. Examples of processors in this case include PLDs (Programmable Logic Devices) such as FPGAs (Field-Programmable Gate Arrays) whose circuit configuration can be changed after manufacture, and dedicated electrical circuits such as ASICs (Application Specific Integrated Circuits), which are processors with circuit configurations designed specifically to execute specific processes. In addition, the learning process and the signal analysis process may be executed by one of these various processors, or may be executed by a combination of two or more processors of the same or different types (for example, multiple FPGAs, and a combination of a CPU and an FPGA). In addition, the hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor elements.

In addition, in each of the above embodiments, the learning program and the signal analysis program are described as being pre-stored (installed) in the storage 14, but this is not limiting. The programs may be provided in a form stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The programs may also be downloaded from an external device via a network.

The following notes are further provided with respect to the above embodiment.

(Additional Note 1)
Memory,
at least one processor coupled to the memory;
Including,
The processor,
An encoder that estimates a latent vector sequence by taking a spectrogram of a sound as an input based on a spectrogram for each constituent sound and an attribute class indicating an attribute of the constituent sound, a classifier that identifies an attribute class indicating an attribute of the sound by taking the spectrogram of the sound as an input, and a decoder that generates a variance of the spectrogram of the sound by taking the latent vector sequence and the attribute class as input,
A signal analysis device configured to estimate the separation matrix and the scale parameter so as to optimize an objective function represented by an input of an observation signal in which each constituent sound is mixed, the latent vector sequence estimated for each constituent sound separated by the separation matrix by the trained encoder, the attribute class identified for each constituent sound separated by the separation matrix by the trained classifier, a spectrogram of the constituent sound calculated from a variance of the spectrogram of the constituent sound and a scale parameter generated by the trained decoder for each constituent sound, a scale parameter of the spectrogram of each constituent sound, a separation matrix for separating a mixed sound in which each constituent sound is mixed into each constituent sound in a time-frequency domain, and a signal obtained by separating the observation signal into each constituent sound.

(Additional Note 2)
A non-transitory storage medium storing a program executable by a computer to perform a signal analysis process,
The signal analysis process includes:
An encoder that estimates a latent vector sequence by taking a spectrogram of a sound as an input based on a spectrogram for each constituent sound and an attribute class indicating an attribute of the constituent sound, a classifier that identifies an attribute class indicating an attribute of the sound by taking the spectrogram of the sound as an input, and a decoder that generates a variance of the spectrogram of the sound by taking the latent vector sequence and the attribute class as input,
A non-transitory storage medium that estimates the separation matrix and the scale parameter so as to optimize an objective function represented by an observation signal in which each constituent sound is mixed, the latent vector sequence estimated for each constituent sound separated by the separation matrix by the trained encoder, the attribute class identified for each constituent sound separated by the separation matrix by the trained classifier, a spectrogram of the constituent sound calculated from the variance of the spectrogram of the constituent sound and a scale parameter generated by the trained decoder for each constituent sound, the scale parameter of the spectrogram of each constituent sound, a separation matrix for separating a mixed sound in which each constituent sound is mixed into each constituent sound in the time-frequency domain, and a signal obtained by separating the observation signal into each constituent sound.

11 CPU
14 Storage 15 Input unit 16 Display unit 24 Time-frequency expansion unit 30 Supervised learning unit 32 Learning unit 34 Sound source signal model storage unit 36 Parameter estimation unit 38 Output unit 40 Initial value setting unit 42 Separation matrix update unit 44 Latent variable class update unit 46 Scale parameter update unit 48 Convergence determination unit 100 Signal analysis device

Claims (8)

a learning unit that learns an encoder that estimates a latent vector sequence by using only a spectrogram of a sound as an input, based on a spectrogram for each constituent sound and an attribute class that indicates an attribute of the constituent sound; a classifier that uses the spectrogram of the sound as an input and identifies an attribute class that indicates an attribute of the sound; and a decoder that uses the latent vector sequence and the attribute class as input and generates a variance of the spectrogram of the sound;
a parameter estimation unit that estimates the separation matrix and the scale parameter so as to optimize an objective function represented by a spectrogram of the constituent sounds, the separation matrix, and a signal obtained by separating the observation signal into each constituent sound, the objective function being calculated from the variance of the spectrogram of the constituent sounds and the scale parameter of the spectrogram of each constituent sound, which are generated by the learned decoder for each constituent sound, and a signal obtained by separating the observation signal into each constituent sound; and
A signal analysis device comprising:
The system further includes a teacher learning unit that learns a teacher encoder that estimates a latent vector sequence using the spectrogram and the attribute class of a sound as input, based on the spectrogram and the attribute class of each component sound, and a teacher decoder that generates a variance of the spectrogram of the sound using the latent vector sequence and the attribute class as input,
The learning unit is a signal analysis device that learns the encoder and the decoder so that the output of the encoder corresponds to the output of the learned teacher encoder, and so that the output of the decoder corresponds to the output of the learned teacher decoder. a learning unit that learns an encoder that estimates a latent vector sequence by using only a spectrogram of a sound as an input, based on a spectrogram for each constituent sound and an attribute class that indicates an attribute of the constituent sound; a classifier that uses the spectrogram of the sound as an input and identifies an attribute class that indicates an attribute of the sound; and a decoder that uses the latent vector sequence and the attribute class as input and generates a variance of the spectrogram of the sound;
a parameter estimation unit that estimates the separation matrix and the scale parameter so as to optimize an objective function represented by a spectrogram of the constituent sounds, the separation matrix, and a signal obtained by separating the observation signal into each constituent sound, the objective function being calculated from the variance of the spectrogram of the constituent sounds and the scale parameter of the spectrogram of each constituent sound, which are generated by the learned decoder for each constituent sound, and a signal obtained by separating the observation signal into each constituent sound; and
A signal analysis device comprising:
The signal analysis device, wherein the encoder and the classifier are an integrated neural network, and the encoder and the classifier share some layers. a learning unit that learns an encoder that estimates a latent vector sequence by using only a spectrogram of a sound as an input, based on a spectrogram for each constituent sound and an attribute class that indicates an attribute of the constituent sound; a classifier that uses the spectrogram of the sound as an input and identifies an attribute class that indicates an attribute of the sound; and a decoder that uses the latent vector sequence and the attribute class as input and generates a variance of the spectrogram of the sound;
a parameter estimation unit that estimates the separation matrix and the scale parameter so as to optimize an objective function represented by a spectrogram of the constituent sounds, the separation matrix, and a signal obtained by separating the observation signal into each constituent sound, the objective function being calculated from the variance of the spectrogram of the constituent sounds and the scale parameter of the spectrogram of each constituent sound, which are generated by the learned decoder for each constituent sound, and a signal obtained by separating the observation signal into each constituent sound; and
A signal analysis device comprising:
The learning unit is
a learning criterion for evaluating the output of the encoder and the output of the decoder;
Mutual information of the decoder output and the attribute classes;
a reconstruction criterion for evaluating the spectrogram generated using the output of the decoder to which the output of the encoder and the output of the classifier are input;
a class identification criterion for evaluating an output of the classifier using as input the spectrogram generated using the output of the decoder using as input the output of the encoder and the output of the classifier;
The signal analysis apparatus trains the encoder, the classifier, and the decoder so as to optimize a criterion including: The parameter estimation unit
an initial value setting unit that sets initial values to the separating matrix, the latent vector sequence, the attribute class, and the scale parameter;
a separation matrix update unit that updates the separation matrix so as to increase the objective function based on a spectrogram of the observed signal, the latent vector sequence, the attribute class, the scale parameter, and the separation matrix that have been previously updated or have initial values set;
a latent variable class update unit that updates the latent vector sequence to the latent vector sequence obtained by using an output of the encoder to which a spectrogram of each component sound obtained by using the observation signal and the separation matrix is input, and updates the attribute class by using an output of a classifier to which a spectrogram of each component sound obtained by using the observation signal and the separation matrix is input;
a scale parameter update unit that updates the scale parameter so as to increase the objective function based on a spectrogram of the observed signal and the updated latent vector sequence, the attribute class, the scale parameter, and the separating matrix; a convergence determination unit that determines whether a predetermined convergence condition is satisfied, and repeats the update by the separating matrix update unit, the update by the latent variable class update unit, and the update by the scale parameter update unit until the convergence condition is satisfied;
The signal analysis device according to any one of claims 1 to 3, comprising: a learning unit learns an encoder that estimates a latent vector sequence by taking only the spectrogram of a sound as an input, based on a spectrogram for each constituent sound and an attribute class indicating an attribute of the constituent sound; a classifier that uses the spectrogram of the sound as an input to identify an attribute class indicating an attribute of the sound; and a decoder that uses the latent vector sequence and the attribute class as input to generate a variance of the spectrogram of the sound;
a parameter estimation unit receives as input an observation signal in which constituent sounds are mixed, the latent vector sequence estimated for each constituent sound separated by the trained encoder using a separation matrix for separating a mixed sound in which constituent sounds are mixed in a time-frequency domain into each constituent sound, and the attribute class identified by the trained classifier for each constituent sound separated by the separation matrix, and estimates the separation matrix and the scale parameter so as to optimize an objective function expressed using a spectrogram of the constituent sounds, the separation matrix, and a signal obtained by separating the observation signal into each constituent sound, the objective function being calculated for each constituent sound from a variance of the spectrogram of the constituent sound and a scale parameter of the spectrogram of each constituent sound, which are generated by the trained decoder,
The teacher learning unit further includes learning a teacher encoder that estimates a latent vector sequence using the spectrogram and the attribute class of a sound as input based on the spectrogram and the attribute class for each component sound, and a teacher decoder that generates a variance of the spectrogram of the sound using the latent vector sequence and the attribute class as input,
A signal analysis method in which the learning unit learns the encoder and the decoder so that the output of the encoder corresponds to the output of the trained teacher encoder, and so that the output of the decoder corresponds to the output of the trained teacher decoder. a learning unit learns an encoder that estimates a latent vector sequence by taking only the spectrogram of a sound as an input, based on a spectrogram for each constituent sound and an attribute class indicating an attribute of the constituent sound; a classifier that uses the spectrogram of the sound as an input to identify an attribute class indicating an attribute of the sound; and a decoder that uses the latent vector sequence and the attribute class as input to generate a variance of the spectrogram of the sound;
a parameter estimation unit receives as input an observation signal in which constituent sounds are mixed, the latent vector sequence estimated for each constituent sound separated by the trained encoder using a separation matrix for separating a mixed sound in which constituent sounds are mixed in a time-frequency domain into each constituent sound, and the attribute class identified by the trained classifier for each constituent sound separated by the separation matrix, and estimates the separation matrix and the scale parameter so as to optimize an objective function expressed using a spectrogram of the constituent sounds, the separation matrix, and a signal obtained by separating the observation signal into each constituent sound, the objective function being calculated for each constituent sound from a variance of the spectrogram of the constituent sound and a scale parameter of the spectrogram of each constituent sound, which are generated by the trained decoder,
A signal analysis method in which the encoder and the classifier are an integrated neural network, and the encoder and the classifier share some layers. a learning unit learns an encoder that estimates a latent vector sequence by taking only the spectrogram of a sound as an input, based on a spectrogram for each constituent sound and an attribute class indicating an attribute of the constituent sound; a classifier that uses the spectrogram of the sound as an input to identify an attribute class indicating an attribute of the sound; and a decoder that uses the latent vector sequence and the attribute class as input to generate a variance of the spectrogram of the sound;
a parameter estimation unit receives as input an observation signal in which constituent sounds are mixed, the latent vector sequence estimated for each constituent sound separated by the trained encoder using a separation matrix for separating a mixed sound in which constituent sounds are mixed in a time-frequency domain into each constituent sound, and the attribute class identified by the trained classifier for each constituent sound separated by the separation matrix, and estimates the separation matrix and the scale parameter so as to optimize an objective function expressed using a spectrogram of the constituent sounds, the separation matrix, and a signal obtained by separating the observation signal into each constituent sound, the objective function being calculated for each constituent sound from a variance of the spectrogram of the constituent sound and a scale parameter of the spectrogram of each constituent sound, which are generated by the trained decoder,
The learning unit is
a learning criterion for evaluating the output of the encoder and the output of the decoder;
Mutual information of the decoder output and the attribute classes;
a reconstruction criterion for evaluating the spectrogram generated using the output of the decoder to which the output of the encoder and the output of the classifier are input;
a class identification criterion for evaluating an output of the classifier using as input the spectrogram generated using the output of the decoder using as input the output of the encoder and the output of the classifier;
The signal analysis method further comprises training the encoder, the classifier, and the decoder to optimize a criterion including: A signal analysis program for causing a computer to function as each unit of the signal analysis device according to any one of claims 1 to 4.