JP7293162B2 - Signal processing device, signal processing method, signal processing program, learning device, learning method and learning program - Google Patents

Description

Application of Article 30, Paragraph 2 of the Patent Law IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING VOL. 13, NO. 4 Date of issue June 13, 2019 https://arxiv. org/abs/2001.08378 https://arxiv. org/pdf/2001.08378. pdf Posted on website January 23, 2020

The present invention relates to a signal processing device, a signal processing method, a signal processing program, a learning device, a learning method, and a learning program.

SpeakerBeam is known as a technique for extracting the speech of a target speaker from a mixed speech signal obtained from speeches of a plurality of speakers (see, for example, Non-Patent Document 1). For example, the method described in Non-Patent Document 1 converts a mixed speech signal into the frequency domain and extracts the speech of the target speaker from the mixed speech signal in the frequency domain. The output of the auxiliary NN is input to the adaptive layer provided in the middle part of the main NN, and the target speech included in the mixed speech signal in the frequency domain is input. It estimates and outputs the speech signal of the person.

K. Zmolikova, M. Delcroix, K. Kinoshita, T. Higuchi, A. Ogawa, and T. Nakatani, “Speaker-aware neural network based beamformer for speaker extraction in speech mixtures,” in Proc. of Interspeech'17, 2017 , pp. 2655-2659.

However, the conventional method has a problem that it may not be possible to accurately extract the speech signal of the target speaker from the mixed speech signal. For example, when the features of the audio signals included in the mixed audio signal are similar, the method described in Non-Patent Document 1 may not provide sufficient accuracy. For example, features of speech signals obtained from speech of multiple speakers of the same sex may resemble each other.

In order to solve the above-described problems and achieve the object, the signal processing device includes a first transforming unit that transforms a time-domain speech signal obtained from an utterance of a target speaker into an adaptive feature amount; A second conversion unit that converts a multi-channel time-domain mixed speech signal obtained by recording the voice with a plurality of microphones into a pre-adaptive feature value by a neural network, and using the adaptation feature value and a third conversion unit that converts the post-adaptation feature amount obtained by adapting the pre-adaptation feature amount to the target speaker into information for output by a neural network having one or more layers. do.

According to the present invention, it is possible to accurately extract the speech signal of the target speaker from the mixed speech signal.

FIG. 1 is a diagram illustrating a configuration example of a signal processing device according to the first embodiment. FIG. 2 is a diagram showing an example of arrangement of microphones and speakers. FIG. 3 is a flow chart showing the processing flow of the signal processing device according to the first embodiment. FIG. 4 is a flowchart showing the processing flow of the first auxiliary NN. FIG. 5 is a flowchart showing the processing flow of the main NN. FIG. 6 is a diagram illustrating a configuration example of a signal processing device according to the second embodiment. FIG. 7 is a flow chart showing the processing flow of the signal processing device according to the second embodiment. FIG. 8 is a flow chart showing the processing flow of the second auxiliary NN. FIG. 9 is a flow chart showing the processing flow of the main NN. FIG. 10 is a diagram showing a configuration example of a learning device according to the third embodiment. FIG. 11 is a flow chart showing the processing flow of the learning device according to the third embodiment. FIG. 12 shows experimental data. FIG. 13 is a diagram showing experimental results. FIG. 14 is a diagram showing experimental results. FIG. 15 is a diagram showing experimental results. FIG. 16 is a diagram showing experimental results. FIG. 17 is a diagram of an example of a computer that executes a program;

Hereinafter, embodiments of a signal processing device, a signal processing method, a signal processing program, a learning device, a learning method, and a learning program according to the present application will be described in detail based on the drawings. In addition, this invention is not limited by embodiment described below.

[First Embodiment]
FIG. 1 is a diagram illustrating a configuration example of a signal processing device according to the first embodiment. As shown in FIG. 1, the signal processing device 10 has a first conversion unit 111 and an integration unit 112 as processing units for executing the first auxiliary NN 11 . The signal processing device 10 also has a second transforming unit 121 , an adapting unit 122 and a third transforming unit 123 as processing units for executing the main NN 12 . The signal processing device 10 also stores parameters such as weights and biases of each neural network as model information 15 . Here, the specific values of the parameters stored as the model information 15 may be obtained by pre-learning in a learning device or learning method, which will be described later.

Here, the main NN 12 is a neural network for extracting the speech signal of the target speaker from the mixed speech signal. The first auxiliary NN 11 is a neural network for obtaining adaptation features for adapting the main NN 12 to the target speaker.

Here, the convolution block is a set of layers for performing one-dimensional convolution, normalization, and the like. Also, the encoder is a neural network that maps the speech signal to a predetermined feature space, that is, converts the speech signal into a feature amount vector. Conversely, the decoder is a neural network that maps features in a predetermined feature space to the space of speech signals, that is, converts feature vectors into speech signals. The encoder and decoder may have a similar structure as the convolution block.

The configuration of the convolution block (1-D Conv), encoder and decoder is described in reference 1 (Y. Luo and N. Mesgarani, “Conv-TasNet: Surpassing ideal time-frequency magnitude masking for speech separation,” IEEE/ACM Trans. ASLP, vol. 27, no. 8, pp. 1256-1266, 2019.). Alternatively, the audio signal in the time domain may be obtained by the method described in Reference 1. Also, each feature amount in the following description shall be represented by a vector.

The first conversion unit 111 converts a time-domain speech signal obtained from an utterance of the target speaker into a feature quantity for adaptation. That is, the first transforming unit 111 receives an input of a time-domain audio signal and outputs an adaptation feature amount. In the example of FIG. 1, the first conversion unit 111 is assumed to be implemented by a neural network, and the neural network converts the speech signal in the time domain into a feature quantity for adaptation. In the following description, the neural network used in the first conversion unit 111 will be called the first neural network. The first neural network is part of the first auxiliary NN11. In the example of FIG. 1, the first neural network comprises an encoder and a convolution block. The adaptive feature amount can be said to be an embedding vector of the target speaker.

Note that the first conversion unit 111 is not limited to a neural network as in the configuration example of FIG. You may use the method to do.

In addition, the integration unit 112 averages the adaptation feature amount for a plurality of time frames. If the speech signal of the target speaker given as an input is a short utterance, it may be averaged over all time frames. , a time interval longer than the time frame that is the processing unit of the first conversion unit. In other words, the integration unit 112 receives an input of the feature amount for adaptation before averaging, and outputs an averaged feature amount for adaptation. Note that the integration unit 112 may be configured by a plurality of fully connected layers.

The second conversion unit 121 converts a multi-channel time-domain mixed speech signal obtained by recording sounds from a plurality of sound sources with a plurality of microphones into a pre-adaptive feature value using a neural network. That is, the second transform unit 121 receives an input of a multi-channel time-domain audio signal and outputs a pre-adaptation feature amount. In the following description, the neural network used in the second conversion section 121 will be called a second neural network. A second neural network is part of the main NN 12 . In the example of FIG. 1, the second neural network is provided with encoders and convolution blocks.

The adaptation unit 122 converts the pre-adaptation feature quantity into the post-adaptation feature quantity adapted to the target speaker using the averaged feature quantity for adaptation. In other words, it receives the input of the adaptation feature amount and the pre-adaptation feature amount, and outputs the post-adaptation feature amount. The adaptation unit 122 can adapt to the target speaker in the same manner as the conventional speaker beam. For example, the adaptation unit 122 can obtain the post-adaptation feature quantity by calculating the element-wise product of the adaptation feature quantity and the pre-adaptation feature quantity, which are vectors of the same number of dimensions. can.

Here, the adaptation unit 122 is realized as a layer in the neural network, that is, as an adaptation layer. Looking at the main NN 12 as a whole, as shown in FIG. 1, the adaptation layer is inserted between the first and second convolutional blocks following the encoder.

The third conversion unit 123 converts the post-adaptation feature quantity into information for output by a neural network. That is, the third transforming unit 123 receives an input of the post-adaptation feature amount and outputs information for output as an estimation result. The output information is information corresponding to the speech signal of the target speaker in the input mixed speech, and may be the speech signal itself or data in a predetermined format from which the speech signal can be derived. good too. In the following description, the neural network used by the third conversion unit 123 is called a third neural network. A third neural network is part of the main NN 12 . In the example of FIG. 1, the third neural network comprises one or more convolution blocks and decoders.

Here, the third transform unit 123 can obtain an estimation result from the intermediate feature amount output from the encoder of the second transform unit 121 and the intermediate feature amount output from the convolution block of the third transform unit 123. . Also, since adaptation to the target speaker is performed, the third conversion unit 123 can not only separate the mixed speech signal for each speaker, but also extract and output the speech signal of the target speaker.

Using FIG. 2, a method of recording a mixed sound that is the source of a mixed sound signal will be described. FIG. 2 is a diagram showing an example of arrangement of microphones and speakers. The microphone array 30 has microphones 301 , 302 , 303 and 304 . Also, the speaker 41 is the target speaker. Also, speaker 42 is a non-target speaker.

Each microphone in microphone array 30 records the voice of both speaker 41 and speaker 42 . As a result, the audio signal of the audio recorded by the microphone array 30 can be treated as an audio signal for each channel corresponding to each microphone. In a first embodiment, it is assumed that mixed audio signals obtained from audio recorded by a microphone array comprising at least two microphones are used. Note that the mixed speech signal may include not only the speech produced by the speaker's speech, but also background noise and the like.

On the other hand, the speech signal of the target speaker is obtained by recording only the speech of the speaker 41 who is the target speaker. Also, in that case, the number of microphones may be one. That is, the target speaker's speech signal may be single-channel. Furthermore, the position of the speaker 41 may be different when recording to obtain the mixed speech signal and when recording to obtain the speech signal of the target speaker.

FIG. 3 is a flow chart showing the processing flow of the signal processing device according to the first embodiment. As shown in FIG. 3, the signal processing device 10 receives an input of a target speaker's speech signal and a mixed speech signal (step S11).

The signal processing device 10 executes the first auxiliary NN 11 (step S12). The signal processing device 10 also executes the main NN 12 (step S13). Here, the signal processing device 10 may execute the first auxiliary NN 11 and the main NN 12 in parallel. However, since the main NN12 uses the output of the first auxiliary NN11, the execution of the main NN12 is not completed until the execution of the first auxiliary NN11 is completed.

FIG. 4 is a flowchart showing the processing flow of the first auxiliary NN. As shown in FIG. 4, the first conversion unit 111 converts the input speech signal of the target speaker in the time domain into a feature for adaptation (step S121). Next, the integration unit 112 integrates and outputs the adaptive feature amount for the time frame (step S122).

FIG. 5 is a flowchart showing the processing flow of the main NN. As shown in FIG. 5, first, the second conversion unit 121 converts the input mixed speech signal in the time domain into a mixed speech feature quantity (step S131). The adaptation unit 122 acquires post-adaptation feature values obtained by adapting the mixed speech feature values to the target speaker using the integrated feature values for adaptation (step S132). The third conversion unit 123 converts the post-adaptation feature amount into an audio signal and outputs the audio signal (step S133).

As described above, the first transforming unit 111 transforms the time-domain speech signal obtained from the speech of the target speaker into the feature quantity for adaptation. In addition, the second conversion unit 121 converts a multi-channel time-domain mixed speech signal obtained by recording sounds from a plurality of sound sources with a plurality of microphones into a pre-adaptive feature amount using a second neural network. . Further, the third transforming unit 123 outputs the post-adaptation feature amount obtained by adapting the pre-adaptation feature amount to the target speaker using the adaptation feature amount by a third neural network having one or more layers. Convert to information for Thus, the mixed audio signal input to the signal processing device 10 is multi-channel. Therefore, the mixed audio signal contains information about the space in which the audio was recorded. As a result, according to the first embodiment, the speech signal of the target speaker can be extracted with higher accuracy than when a single-channel mixed speech signal is input.

[Second embodiment]
In the second embodiment, the IPD (Inter-microphone phase difference) feature amount is further used to acquire spatial information. In the description of the embodiments below, the same reference numerals are given to the parts having the same functions as those of the already described embodiments, and the description thereof will be omitted as appropriate.

The IPD feature amount is an example of information on the phase difference between microphones corresponding to each channel of the mixed audio signal. The angle Φ for calculating the elements of the IPD feature quantity is calculated as shown in Equation (1).

where Y _i,t,f are the coefficients corresponding to microphone i in the short-time Fourier transform (STFT) of the mixed speech signal with time index t and frequency index f. Furthermore, the IPD feature quantity is calculated as in Equation (2). where F is the number of frequency bins. Also, ∠ represents a phase represented by a complex number.

Note that the STFT window size and shift width for obtaining the IPD feature amount may be determined according to the encoder of the main NN 12 .

FIG. 6 is a diagram illustrating a configuration example of a signal processing device according to the second embodiment. As shown in FIG. 6, the signal processing device 10a has a fourth conversion section 132 for executing the second auxiliary NN13. The signal processing device 10a also has a coupling unit 122a. The signal processing device 10a also stores parameters such as weights and biases of each neural network as model information 15a.

The fourth conversion unit 132 converts the spatial information about the phase difference between the microphones corresponding to each channel of the mixed audio signal into a spatial information feature amount. In other words, the fourth conversion unit 132 receives an input of spatial information and outputs a spatial information feature amount. For example, spatial information is an IPD feature. In addition, the neural network forming the fourth transform unit 132 is provided with layers for convolution blocks and upsampling. The spatial information feature amount can be said to be a feature amount obtained by up-sampling the feature amount obtained by the convolution block and then converting it by the convolution block.

The combining unit 122a combines the post-adaptation feature quantity output by the adaptation unit 122 and the spatial information feature quantity. The combining unit 122a may simply combine so that each element of the spatial information feature amount that is a vector follows each element of the post-adaptation feature amount that is a vector. The third conversion unit 123 converts the post-adaptation feature amount obtained by combining the spatial information feature amounts into information for output.

FIG. 7 is a flow chart showing the processing flow of the signal processing device according to the second embodiment. As shown in FIG. 7, the signal processing device 10a receives input of the target speaker's speech signal, mixed speech signal, and spatial information (step S21).

The signal processing device 10a executes the first auxiliary NN 11 (step S22). The signal processing device 10a also executes the second auxiliary NN 13 (step S23). The signal processing device 10a also executes the main NN 12 (step S24). The flow of processing in which the signal processing device 10a executes the first auxiliary NN 11 is the same as that described with reference to FIG.

FIG. 8 is a flow chart showing the processing flow of the second auxiliary NN. As shown in FIG. 8, the fourth conversion unit 132 converts the input spatial information into a spatial information feature amount (step S231). Then, the fourth conversion unit 132 up-samples the spatial information feature quantity and outputs it (step S232).

FIG. 9 is a flow chart showing the processing flow of the main NN. As shown in FIG. 9, first, the second conversion unit 121 converts the input mixed speech signal in the time domain into mixed speech features (step S241). The adapting unit 122 acquires the post-adaptation feature amount by adapting the mixed speech feature amount to the target speaker using the integrated feature amount for adaptation (step S242).

Here, the combining unit 122a combines the spatial information feature quantity with the post-adaptation feature quantity (step S243). The third transforming unit 123 transforms the post-adaptation feature amount in which the spatial information feature amount has been combined into an audio signal and outputs the audio signal (step S244).

As described above, the fourth conversion unit 132 converts the spatial information regarding the phase difference between the microphones corresponding to each channel of the mixed audio signal into the spatial information feature quantity. Also, the third transforming unit 123 transforms the post-adaptation feature amount obtained by combining the spatial information feature amounts into information for output. In this way, the signal processing device 10a can extract the speech of the target speaker by using the feature amount that makes the spatial information clearer. As a result, according to the second embodiment, the speech signal of the target speaker can be extracted with higher accuracy.

The adaptation layer realized by the adaptation unit 122 extracts the feature amount of the target speaker's speech from the feature amount of the mixed speech signal, using the feature amount obtained from the target speaker's speech signal as a clue. Furthermore, in the second embodiment, spatial information is used in the layer on the output side of the adaptive layer, so that correction can be made in consideration of the direction of each voice in the mixed voice. In other words, in the second embodiment, if the post-adaptation feature amount includes a sound in an essentially unnecessary direction, the feature amount of the audio signal with higher separation performance is obtained by removing the feature related to the sound. It is considered possible.

Furthermore, in the example of FIG. 6, the spatial information feature amount is input to the layer on the output side of the adaptation layer. On the other hand, an embodiment is also conceivable in which the spatial information feature amount is input to a layer closer to the input side than the adaptive layer. However, since the adaptation layer selects a speaker based on spectral information, the spatial information features input to the layer on the input side from the adaptation layer may adversely affect speaker selection. Conceivable. This is also reflected in the experimental results presented later.

[Third embodiment]
In the third embodiment, a learning device that performs learning processing for generating the model information 15 of the signal processing device 10 of the first embodiment will be described. FIG. 10 is a diagram showing a configuration example of a learning device according to the third embodiment.

As shown in FIG. 10, the learning device 20 executes the first auxiliary NN11 and the main NN12 on the learning data, like the signal processing device 10 of the first embodiment. For example, the learning data is data containing a mixed speech signal and the speech signal of the target speaker included in the mixed speech signal as a correct answer.

The first conversion unit 111, the second conversion unit 121, and the third conversion unit 123 perform the same processing as in the first embodiment. Further, the updating unit 24 updates the first neural network, the second neural Update the parameters of the network and the third neural network. For example, the update unit 24 updates the parameters by error backpropagation. The updating unit 24 updates the model information 25, which is the parameter of each neural network.

Here, the update unit 24 is adapted to optimize the signal-to-noise ratio between the estimation result of the speech signal of the target speaker indicated by the information for output and the correct answer of the speech signal of the target speaker. The parameters can be updated so that the ability to discriminate the target speaker's speech signal by the feature quantity is improved. In this case, the updating unit 24 updates the parameters so that the loss calculated as in equation (3) is optimized. In other words, the learning device 20 performs multi-task learning so that both of the two tasks of speech recognition and speaker identification are closer to correct answers.

As shown in equation (3), the loss function is the weighted sum of the loss related to the output of the main NN12 and the loss related to the output of the first auxiliary NN11. For example, as shown in the first term of equation (3), the loss related to the output of the main NN 12 is the audio signal ^x _s (right above x) output from the main NN, and the loss included in the training data. is the signal-to-noise ratio (SiSNR) of the target speaker's speech signal to the correct answer _xs . In addition, the loss related to the output of the first auxiliary NN 11 is expressed using the discriminating ability in the task of speaker discrimination to discriminate "whether or not the speaker of the input speech signal belongs to the target speaker". . For example, the second term of equation (3) converts the speaker label l ^s and the target speaker's feature quantity e ^s (output of the first auxiliary NN 11) using the matrix W, and applies the softmax function σ(・) The loss related to the output of the first auxiliary NN 11 is expressed by the cross entropy (CE) with the result obtained by multiplying the weight (scaling parameter) α.

FIG. 11 is a flow chart showing the processing flow of the learning device according to the third embodiment. As shown in FIG. 11, the learning device 20 receives input of the target speaker's speech signal and mixed speech signal (step S31). Each audio signal input to the learning device 20 is learning data whose correct answer is known.

The learning device 20 executes the first auxiliary NN 11 (step S32). Also, the learning device 20 executes the main NN 12 (step S33). Here, the updating unit 24 updates the model information 25 so that the loss is optimized (step S34).

If the predetermined condition is satisfied, the learning device 20 determines that convergence has occurred (step S35, Yes) and ends the process. On the other hand, if the predetermined condition is not satisfied, the learning device 20 determines that convergence has not occurred (step S35, No) and returns to step S32 to repeat the process. For example, the conditions are that a predetermined number of iterations has been reached, that the loss function value has become equal to or less than a predetermined threshold value, that the parameter update amount (differential value of the loss function value, etc.) has become equal to or less than a predetermined threshold value, etc. is.

As described above, the first transforming unit 111 converts a time-domain speech signal obtained from an utterance of the target speaker into a feature value for adaptation using a first neural network having one or more layers. Convert to The second conversion unit 121 converts multi-channel time-domain mixed speech signals obtained by recording sounds from a plurality of sound sources with a plurality of microphones into the same number of layers as the number of layers included in the first neural network. is converted into pre-adaptive features by a second neural network with The third transforming unit 123 converts the post-adaptation feature quantity obtained by adapting the pre-adaptation feature quantity to the target speaker using the feature quantity for adaptation, by a third neural network having one or more layers, to output Convert to information. The update unit 24 updates the first neural network, the second neural network, and the Update the parameters of the third neural network. As a result, according to the third embodiment, it is possible to improve the accuracy of extracting the speech signal of the target speaker.

The update unit 24 optimizes the signal-to-noise ratio between the estimation result of the speech signal of the target speaker indicated by the information for output and the correct answer of the speech signal of the target speaker, and also updates the adaptive feature amount. The parameters are updated so that the ability to discriminate the target speaker's speech signal by is improved. This improves the accuracy of not only the NN for speech extraction but also the NN for adaptation to the target speaker. As a result, according to the third embodiment, it is possible to improve the accuracy of extracting the speech signal of the target speaker.

[Experimental result]
Here, the results of experiments conducted to compare the embodiment with the conventional method will be described. FIG. 12 shows experimental data. 13 to 16 are diagrams showing experimental results. In the experiment, two types of multi-channel mixed speech WSJ (MC-WSJ0-2 mix) and CSJ (CSJ-2mix) shown in FIG. 12 were used. #Spks is the number of speakers, #F is the number of female speakers, #M is the number of male speakers, and #Mixture is the number of mixed utterances.

13 and 14 are the results of evaluating the speech signal of the target speaker extracted by each method by SDR (signal-to-distortion ratio). (7) in FIG. 13 corresponds to the estimation method of the first embodiment. (9) in FIG. 13 corresponds to the estimation method of the second embodiment. In addition, (8) in FIG. 13 corresponds to the case where the spatial information feature amount is input to the layer on the input side from the adaptation layer in the estimation method of the second embodiment. FF indicates mixed voices of female voices. MM indicates a mixed voice of male voices. Also, FM indicates a mixture of female and male voices. As shown in FIG. 13, embodiments show high accuracy for mixed speech, especially for speakers of different genders. Also, the method (9) is more accurate when the gender of the speaker is the same.

FIG. 14(5) corresponds to the estimation method of the first embodiment, and is the result when the loss function during learning according to the third embodiment does not include the loss related to the output of the first auxiliary NN11. On the other hand, (6) is the result when the loss function during learning includes the loss (SI-loss) related to the output of the first auxiliary NN 11 . As shown in FIG. 14, the first embodiment exhibits higher accuracy than the conventional method, and introducing SI-loss further improves the accuracy. In particular, the introduction of SI-loss greatly improved the accuracy in the case of FF.

FIG. 15 shows the degree of SDR improvement in each case of FF, MM, and FM. As shown in FIG. 15, according to the method of the embodiment (TD-SpkBeam, TD-SpkBeam+SI-loss), the SDR often exceeds 0, improving the accuracy. FIG. 16 shows the SDR according to the number of speakers in the training data. As shown in FIG. 16, according to the method of the embodiment (TD-SpkBeam, TD-SpkBeam+SI-loss), the SDR is greatly improved especially when the number of speakers exceeds 100.

[System configuration, etc.]
Also, each component of each device illustrated is functionally conceptual, and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution and integration of each device is not limited to the illustrated one, and all or part of them can be functionally or physically distributed and integrated in arbitrary units according to various loads and usage conditions. Can be integrated and configured. Furthermore, all or any part of each processing function performed by each device is realized by a CPU (Central Processing Unit) and a program analyzed and executed by the CPU, or hardware by wired logic can be realized as

Further, among the processes described in the present embodiment, all or part of the processes described as being automatically performed can be performed manually, or the processes described as being performed manually can be performed manually. All or part of this can also be done automatically by known methods. In addition, information including processing procedures, control procedures, specific names, and various data and parameters shown in the above documents and drawings can be arbitrarily changed unless otherwise specified.

[program]
As one embodiment, the signal processing apparatus 10 can be implemented by installing a signal processing program for executing the above-described audio signal extraction processing as package software or online software in a desired computer. For example, the information processing device can function as the signal processing device 10 by causing the information processing device to execute the above signal processing program. The information processing apparatus referred to here includes a desktop or notebook personal computer. In addition, information processing devices include smart phones, mobile communication terminals such as mobile phones and PHS (Personal Handyphone Systems), and slate terminals such as PDAs (Personal Digital Assistants).

Further, the signal processing device 10 can be implemented as a signal processing server device that uses a terminal device used by a user as a client and provides the client with the service related to the above signal processing. For example, the signal processing server device is implemented as a server device that receives a mixed speech signal as an input and provides a signal processing service for extracting the speech signal of the target speaker. In this case, the signal processing server device may be implemented as a web server, or may be implemented as a cloud that provides services related to the above signal processing through outsourcing.

FIG. 17 is a diagram of an example of a computer that executes a program; The computer 1000 has a memory 1010 and a CPU 1020, for example. Computer 1000 also has hard disk drive interface 1030 , disk drive interface 1040 , serial port interface 1050 , video adapter 1060 and network interface 1070 . These units are connected by a bus 1080 .

The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012 . The ROM 1011 stores a boot program such as BIOS (Basic Input Output System). Hard disk drive interface 1030 is connected to hard disk drive 1090 . A disk drive interface 1040 is connected to the disk drive 1100 . A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100 . Serial port interface 1050 is connected to mouse 1110 and keyboard 1120, for example. Video adapter 1060 is connected to display 1130, for example.

The hard disk drive 1090 stores an OS 1091, application programs 1092, program modules 1093, and program data 1094, for example. That is, a program that defines each process of the signal processing device 10 is implemented as a program module 1093 in which computer-executable code is described. Program modules 1093 are stored, for example, on hard disk drive 1090 . For example, the hard disk drive 1090 stores a program module 1093 for executing processing similar to the functional configuration of the signal processing device 10 . Note that the hard disk drive 1090 may be replaced by an SSD.

Also, setting data used in the processing of the above-described embodiment is stored as program data 1094 in the memory 1010 or the hard disk drive 1090, for example. Then, the CPU 1020 reads out the program modules 1093 and program data 1094 stored in the memory 1010 and the hard disk drive 1090 to the RAM 1012 as necessary and executes them.

Note that the program modules 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090, but may be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, program modules 1093 and program data 1094 may be stored in another computer connected via a network (LAN (Local Area Network), WAN (Wide Area Network), etc.). Program modules 1093 and program data 1094 may then be read by CPU 1020 through network interface 1070 from other computers.

10, 10a Signal processing device 11 First auxiliary NN
12 main NN
15, 15a, 25 model information 20 learning device 24 update unit 30 microphone array 41, 42 speaker 111 first conversion unit 112 integration unit 121 second conversion unit 122 adaptation unit 122a combining unit 123 third conversion unit 132 fourth conversion unit 301, 302, 303, 304 Microphones

Claims (8)

a first conversion unit that converts a time-domain speech signal obtained from an utterance of a target speaker into a feature quantity for adaptation;
a second conversion unit that converts a multi-channel time-domain mixed speech signal obtained by recording sounds from a plurality of sound sources with a plurality of microphones into a pre-adaptive feature value by a neural network;
a third conversion unit that converts the post-adaptation feature amount obtained by adapting the pre-adaptation feature amount to the target speaker using the adaptation feature amount into information for output by a neural network;
A signal processing device comprising: further comprising a fourth conversion unit that converts information about a phase difference between microphones corresponding to each channel of the mixed audio signal into a spatial information feature amount;
2. The signal processing apparatus according to claim 1, wherein the third conversion unit converts the post-adaptation feature amount obtained by combining the spatial information feature amounts into the information for output. A signal processing method performed by a signal processing device, comprising:
a first transformation step of transforming a time-domain speech signal obtained from an utterance of a target speaker into a feature quantity for adaptation;
a second conversion step of converting a multi-channel time-domain mixed speech signal obtained by recording sounds from a plurality of sound sources with a plurality of microphones into a pre-adaptive feature amount by a neural network;
a third conversion step of converting the post-adaptation feature quantity obtained by adapting the pre-adaptation feature quantity to the target speaker using the adaptation feature quantity into information for output by a neural network;
A signal processing method comprising: A signal processing program for causing a computer to function as the signal processing device according to claim 1 or 2. a first conversion unit that converts a time-domain speech signal obtained from an utterance of a target speaker into a feature quantity for adaptation;
a second conversion unit that converts a multi-channel time-domain mixed speech signal obtained by recording sounds from a plurality of sound sources with a plurality of microphones into a pre-adaptive feature value by a neural network;
a third conversion unit that converts the post-adaptation feature amount obtained by adapting the pre-adaptation feature amount to the target speaker using the adaptation feature amount into information for output by a neural network;
The neural network used in the second conversion unit and the third an update unit characterized by updating parameters of a neural network used in the conversion unit;
A learning device characterized by comprising: The update unit optimizes a signal-to-noise ratio between the estimation result of the speech signal of the target speaker indicated by the information for output and the correct answer of the speech signal of the target speaker, and 6. The learning apparatus according to claim 5, wherein said parameter is updated so as to improve the ability to discriminate the speech signal of said target speaker by means of the feature quantity for adaptation. A computer implemented learning method comprising:
a first transformation step of transforming a time-domain speech signal obtained from an utterance of a target speaker into a feature quantity for adaptation;
a second conversion step of converting a multi-channel time-domain mixed speech signal obtained by recording sounds from a plurality of sound sources with a plurality of microphones into a pre-adaptive feature amount by a neural network;
a third conversion step of converting the post-adaptation feature quantity obtained by adapting the pre-adaptation feature quantity to the target speaker using the adaptation feature quantity into information for output by a neural network;
The neural network used in the second transformation step and the third an update step characterized by updating the parameters of the neural network used in the conversion step;
A learning method comprising: A learning program for causing a computer to function as the learning device according to claim 5 or 6.

Images