JP7368779B2 - Audio signal conversion model learning device, audio signal conversion device, audio signal conversion model learning method and program - Google Patents

Description

Application of Article 30, Paragraph 2 of the Patent Act On April 5, 2019, the web address of “WaveCycleGAN2: Time-domain Neural Post-filter for Speech Waveform Generation” is https://arxiv. org/abs/1904.02892 and https://arxiv. org/pdf/1904.02892v1. Published in pdf.

The present invention relates to an audio signal conversion model learning device, an audio signal conversion device, an audio signal conversion model learning method, and a program.

Input information, such as voice generation using a parametric vocoder method (see Non-Patent Document 1) and statistical voice quality conversion (see Non-Patent Document 2), is said to have the potential to expand human communication abilities and physical functions. Research is being conducted on technology to generate desired audio from. For example, voice generation technology using a parametric vocoder method is useful for assisting people with physical disabilities (see Non-Patent Documents 3 and 4) and language education support (see Non-Patent Documents 3 and 4) because of its ease of system construction and high versatility. 5 and 6), and applications to amusement (see Non-Patent Document 7) have been widely studied.

H. Zen, A. Senior, and M. Schuster, “Statistical parametric speech synthesis using deep neural networks,” 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.7962-7966, 2013. T. Toda, A.W. Black, and K. Tokuda, “Voice conversion based on maximum-likelihood estimation of spectral parameter trajectory,” IEEE Transactions on Audio, Speech and Language Processing (TASLP), vol.15, no.8, pp. 2222-2235, 2007. J. Yamagishi, C. Veaux, S. King, and S. Renals, “Speech synthesis techniques for individuals with vocal disabilities: Voice banking and reconstruction,” Acoustical Science and Technology, vol.33, no.1, pp.1- 5, 2012. K. Tanaka, T. Toda, G. Neubig, S. Sakti, and S. Nakamura, “A hybrid approach to electrolaryngeal speech enhancement based on noise reduction and statistical excitation generation,” IEICE Transactions on Information and Systems, vol.97, no.6, pp.1429-1437, 2014. D. Felps, H. Bortfeld, and R. Gutierrez-Osuna, “Foreign accent conversion in computer assisted pronunciation training,” Speech Communication, vol.51, no.10, pp.920-932, 2009. S. Aryal and R. Gutierrez-Osuna, “Can voice conversion be used to reduce non-native accents?,” 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.7879-7883, 2014. K. Kobayashi, T. Toda, and S. Nakamura, “Intra-gender statistical singing voice conversion with direct waveform modulation using log-spectral differential,” Speech Communication, vol.99, pp.211-220, 2018. J. Zhu, T. Park, P. Isola, and A.A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” 2017 IEEE International Conference on Computer Vision(ICCV), pp.2242-2251, 2017 .

However, there has been a problem in that the voices generated using the above-mentioned conventional techniques and the like have a large difference from the voices actually uttered by humans. One of the causes of the difference from human voices is excessive smoothing of the generated features. As a method for suppressing such excessive smoothing, GAN (Generative Adversarial Networks), which is one of the machine learning methods, has been proposed. As a method using GAN, for example, SEGAN (Speech Enhancement Generative Adversarial Network) has been proposed (see Non-Patent Document 8). However, with the methods using SEGAN that have been proposed so far, there is a problem that learning cannot be achieved if there is a target speech waveform with the same amplitude spectrum but a different phase spectrum in the training data. It was difficult to use data to reduce the difference between vocal sounds actually uttered by humans. Further, such problems are not limited to humans, but also occur in the generation of sounds emitted by animals.

In view of the above circumstances, an object of the present invention is to provide a technique for generating sounds that are closer to sounds emitted by animals.

One aspect of the present invention is learning to convert an input signal, which is an input audio signal, into an audio signal with a natural signal degree that indicates the degree of similarity to a natural signal, which is the sound actually emitted by an animal, than the input signal. a learning unit that obtains a model using a machine learning method; a first generating section that outputs a forward-transformed signal which is a signal with a high degree of natural signal; a first identification section that identifies whether the input signal is a forward-transformed signal or a natural signal; a second generation unit that outputs an inversely transformed signal having a lower natural signal level than the audio signal by executing an inverse transform process that lowers the natural signal level of the audio signal; and an input signal is prepared in advance. a second identification unit that identifies whether the signal is a pre-synthesized signal that is a synthesized signal or an inversely transformed signal, based on the identification results of the first identification unit and the second identification unit. This is an audio signal conversion model learning device, which is a learning method based on the following.

According to the present invention, it is possible to generate sounds that are closer to sounds emitted by animals.

FIG. 1 is an explanatory diagram illustrating an overview of an audio signal generation system 100 according to an embodiment. FIG. 1 is an explanatory diagram illustrating an overview of an audio signal conversion model learning device 1 in an embodiment. 5 is a flowchart illustrating an example of the flow of forward conversion signal identification processing in the embodiment. The 1st flowchart which shows an example of the flow of forward transformation learning processing in an embodiment. The 2nd flowchart which shows an example of the flow of forward transformation learning processing in an embodiment. 5 is a flowchart illustrating an example of the flow of forward conversion signal identification learning processing in the embodiment. 5 is a flowchart illustrating an example of the flow of inversely transformed signal identification processing in the embodiment. 5 is a flowchart illustrating an example of the flow of inverse transformation learning processing in the embodiment. 5 is a flowchart illustrating an example of the flow of inversely transformed signal identification learning processing in the embodiment. 1 is a flowchart illustrating an example of the flow of processing executed by the audio signal conversion model learning device 1 in the embodiment. FIG. 1 is a diagram illustrating an example of a hardware configuration of an audio signal conversion model learning device 1 in an embodiment. The figure which shows an example of the functional structure of the control part 10 in embodiment. FIG. 2 is a diagram showing an example of a hardware configuration of an audio signal conversion device 2 in an embodiment. The figure which shows an example of the functional structure of the control part 20 in embodiment. 5 is a flowchart showing an example of the flow of processing executed by the audio signal conversion device 2 in the embodiment. The first diagram showing an example of the experimental results of the first experiment. The second diagram showing an example of the experimental results of the first experiment. The figure which shows an example of the experimental result of a 2nd experiment. The figure which shows an example of the experimental result of a 3rd experiment.

(Embodiment)
An outline of the audio signal generation system 100 according to the embodiment will be described using FIGS. 1 and 2. FIG. 1 is an explanatory diagram illustrating an overview of an audio signal generation system 100 according to an embodiment. The audio signal generation system 100 generates a synthesized audio signal (hereinafter referred to as "synthesized signal") that has a low degree of similarity (hereinafter referred to as "natural signal degree") to a natural signal (hereinafter referred to as "unnatural signal"). (referred to as "synthetic signal"). Natural signals are sounds actually made by humans. That is, the audio signal generation system 100 converts the input unnatural composite signal into a natural composite signal that is a composite signal having a higher degree of natural signal than the input unnatural composite signal. Note that converting an unnatural composite signal to a natural composite signal is equivalent to generating a natural composite signal based on the unnatural composite signal.

The audio signal generation system 100 includes an audio signal conversion model learning device 1 and an audio signal conversion device 2. The audio signal conversion model learning device 1 uses machine learning to obtain a trained model (hereinafter referred to as "audio signal conversion model") that generates a natural composite signal based on an unnatural composite signal. To simplify the explanation below, performing machine learning will be referred to as learning. Note that performing machine learning means suitably adjusting the values of parameters in a machine learning model. In the following description, learning to satisfy A means that the values of parameters in the machine learning model are adjusted so as to satisfy A. A represents a predetermined condition.

The audio signal conversion model learning device 1 uses a natural signal and a synthesized signal as input, and performs cyclic adversarial learning (Cycle GAN) using an audio waveform discriminator and an audio feature discriminator in learning the discriminator. Learn an audio signal conversion model. The audio waveform discriminator is a classifier that identifies whether or not an audio signal is a natural signal based on the waveform of the audio signal used for learning (hereinafter referred to as "audio waveform"). The audio feature discriminator is a classifier that acquires information that satisfies a predetermined condition from an audio signal used for learning as an audio feature, and identifies whether the audio signal is a natural signal based on the acquired audio feature. Hereinafter, CycleGAN using a voice waveform classifier and a voice feature classifier will be referred to as convolution CycleGAN. Note that the audio feature amount is, for example, a phase spectrum of an audio signal. Note that, as described later, the natural signal and the composite signal input to the audio signal conversion model learning device 1 may be stored in advance in a storage unit included in the audio signal conversion model learning device 1.

Convolution CycleGAN is a neural network that uses different discriminators to learn the audio waveform and audio feature of the audio signal used for learning. In general, a neural network that learns features of data used for learning using different classifiers for each feature is called a convolutional neural network. Therefore, convolutional CycleGAN is both a neural network that is a modification of CycleGAN, and a neural network that is a modification of a convolutional neural network.

FIG. 2 is an explanatory diagram illustrating an overview of the audio signal conversion model learning device 1 in the embodiment. The audio signal conversion model learning device 1 includes a first generation section 110, a first identification section 120, a first input determination section 130, a second generation section 150, a second identification section 160, and a second input determination section 170. The first generation unit 110, the first identification unit 120, the second generation unit 150, and the second identification unit 160 are functional units that perform learning. In the audio signal conversion model learning device 1, the first generation unit 110, the first identification unit 120, the first input determination unit 130, the second generation unit 150, the second identification unit 160, and the second input determination unit 170 cooperate. and execute CycleGAN.

The first generation unit 110 performs forward conversion processing on the input audio signal. The forward conversion process is a process that improves the natural signal level of the input audio signal. The first generation unit 110 outputs the audio signal after the forward conversion process as a forward conversion signal. The first generation unit 110 learns based on the identification result of the first identification unit 120, the details of which will be described later. Through learning, the first generation unit 110 learns to further improve the natural signal level through forward transformation processing.

A specific example of learning that further improves the natural signal degree by forward transformation processing is to reduce the value of a loss function, which is a function that shows a larger value as the probability that the identification result of the first identification unit 120 is incorrect is lower. This is a process of suitably adjusting parameter values.

The first identification unit 120 identifies whether the input audio signal is a natural signal or a forward-transformed signal. The first identification unit 120 learns based on this identification result. The first identification unit 120 includes an audio waveform identification unit 121 , an audio feature identification unit 122 , an integrated identification unit 123 , and a first determination unit 140 .

The audio signal input to the first identification unit 120 is input to the audio waveform identification unit 121 . The audio signal input to the audio waveform identification unit 121 is an audio signal determined by a first input determining unit 130, which will be described in detail later, and is a natural signal or a forward conversion signal. The audio waveform identification unit 121 identifies whether the audio signal input to the first identification unit 120 is a natural signal or a forward-converted signal based on the audio waveform of the input audio signal. The audio waveform identifier 121 is an example of an audio waveform identifier.

The audio signal input to the first identification unit 120 is input to the audio feature identification unit 122 . That is, the audio signal input to the audio feature amount identifying section 122 is the same as the audio signal input to the audio waveform identifying section 121. The audio feature identification unit 122 acquires audio features based on the input audio signal. The audio feature identifying unit 122 identifies whether the audio signal input to the first identifying unit 120 is a natural signal or a forward-converted signal based on the acquired audio feature. The audio feature identification unit 122 is an example of an audio feature identifier.

The integrated identification unit 123 determines whether the audio signal input to the first identification unit 120 is a natural signal or a forward-converted signal based on the identification result of the audio waveform identification unit 121 and the identification result of the audio feature amount identification unit 122. identify whether The identification result of the integrated identification unit 123 is the identification result of the first identification unit 120. The identification result of the integrated identification unit 123 is output to the first determination unit 140.

The first determination unit 140 determines whether the identification result of the integrated identification unit 123 is correct based on the determination result of the first input determination unit 130.

The speech waveform identification section 121, the speech feature amount identification section 122, and the integrated identification section 123 learn based on the determination result of the first determination section 140. The speech waveform identification section 121, the speech feature amount identification section 122, and the integrated identification section 123 learn to further improve the accuracy of identification through learning. A specific example of learning that further improves the accuracy of identification is to increase the value of a loss function, which is a function that takes a larger value as the probability that the identification result of the integrated identification unit 123 is incorrect is lower. This is a suitable adjustment process.

The first input determining unit 130 determines whether the audio signal input to the first identifying unit 120 is a forward-transformed signal or a natural signal.

When the first input determination unit 130 determines a natural signal as the audio signal input to the first identification unit 120, one natural signal belonging to the natural signal group described in the center column of FIG. is input. The natural signal group is a set of natural signals prepared in advance for learning. Note that the composite signal group shown in the center column of FIG. 2 is a set of composite signals prepared in advance for learning. Hereinafter, the combined signal belonging to the combined signal group will be referred to as a pre-combined signal.

When the first input determining unit 130 determines a composite signal as the composite signal to be input to the first identifying unit 120 , the forward transformed signal is input to the first identifying unit 120 .

The second generation unit 150 performs inverse transformation processing on the input audio signal. When a forward conversion signal is input as an audio signal, an inverse conversion process is performed on the acquired forward conversion signal. When a natural signal is input as an audio signal, an inverse transformation process is performed on the acquired natural signal. The inverse conversion process is a process that reduces the natural signal intensity of the input audio signal. The second generation unit 150 outputs the audio signal after the inverse transformation process as an inverse transformation signal. The second generation unit 150 learns based on the identification result of the second identification unit 160, the details of which will be described later. Through learning, the second generation unit 150 learns to further reduce the natural signal intensity through inverse transformation processing.

A specific example of learning in which the natural signal degree is further reduced by inverse transformation processing is to reduce the value of a loss function, which is a function that shows a larger value as the probability that the identification result of the second identification unit 160 is incorrect is lower. This is a process of suitably adjusting parameter values.

The second identification unit 160 identifies whether the input audio signal is an inversely transformed signal or a pre-synthesized signal. The second identification unit 160 learns based on the identification result of the second identification unit 160. The second identification unit 160 includes an audio waveform identification unit 161 , an audio feature identification unit 162 , an integrated identification unit 163 , and a second determination unit 180 .

The audio waveform identification unit 161 determines whether the audio signal input to the second identification unit 160 is an inversely converted signal or a pre-synthesized signal based on the audio waveform of the audio signal input to the second identification unit 160. identify The audio waveform identifier 161 is an example of an audio waveform identifier.

The audio feature identifying unit 162 determines whether the audio signal input to the second identifying unit 160 is an inversely converted signal or a pre-synthesized signal based on the audio feature of the audio signal input to the second identifying unit 160. identify. The audio feature identification unit 162 is an example of an audio feature identifier.

The integrated identification unit 163 determines whether the audio signal input to the second identification unit 160 is a combination of an inversely converted signal and a pre-synthesized signal, based on the identification result of the audio waveform identification unit 161 and the identification result of the audio feature amount identification unit 162. Identify which one. The identification result of the integrated identification unit 163 is the identification result of the second identification unit 160. The identification result of the integrated identification unit 163 is output to the second determination unit 180.

The second determination unit 180 determines whether the identification result of the second identification unit 160 is correct based on the determination result of the second input determination unit 170.

The speech waveform identification section 161, the speech feature amount identification section 162, and the integrated identification section 163 learn based on the determination result of the second determination section 180. The speech waveform identification section 161, the speech feature amount identification section 162, and the integrated identification section 163 learn to further improve the accuracy of identification through learning. A specific example of learning that further improves the accuracy of identification is to increase the value of a loss function, which is a function that takes a larger value as the probability that the identification result of the integrated identification unit 163 is incorrect is lower. This is a suitable adjustment process.

The second input determining unit 170 determines whether the audio signal to be input to the second generating unit 150 is a forward-transformed signal or a natural signal. The second input determining unit 170 also determines whether the audio signal to be input to the second identifying unit 160 is an inversely transformed signal or a pre-synthesized signal.

The first generation unit 110, first identification unit 120, second generation unit 150, and second identification unit 160 work together to learn to reduce the objective function L expressed by the following formula. . That is, the objective function L is a loss function used when the first generation unit 110, first identification unit 120, second generation unit 150, and second identification unit 160 learn.

H1 represents self-identical loss. More specifically, H1 is expressed by the following formula (18).

The sum of H2 to H9 represents the adversarial loss. Note that D _xwave represents a discriminator that identifies what kind of signal the audio signal x is based on the waveform of the audio signal x. Note that D _ywave represents a discriminator that identifies what kind of signal the audio signal y is based on the waveform of the audio signal y. Note that D _xmsp represents a discriminator that identifies what kind of signal the audio signal x is based on the audio feature amount of the audio signal x. Note that D _ymsp represents a classifier that identifies what kind of signal the audio signal y is based on the audio feature amount of the audio signal y. Hereinafter, for ease of explanation, the discriminator will be represented by the symbol D. Note that Dmsp(A) is a function that outputs the probability of whether or not A is the target audio feature. Furthermore, log(1-Dmsp(A)) is a function that outputs the probability that A is not the target audio feature.

F(A) means a process in which a fast Fourier transform matrix windowed using a Hanning window is convolved with A, and then a Mel filter is convolved with the absolute value of A after the convolution.

The term λ _cyc of L _cyc in the objective function represents the weight. _Lcyc is a hyperparameter in learning. G _x→y is a mapping that converts the audio signal x into the audio signal y. The audio signal y has a higher degree of natural signal than the audio signal x. D _y represents a discrimination function for determining whether the input audio signal y is a natural signal or a synthetic signal. G _y→x is a mapping that converts the audio signal y into the audio signal x. D _x represents a discriminant function for determining whether the input audio signal x is a natural signal or a synthetic signal.

L _adv represents an objective function in adversarial learning. That is, L _adv represents an adversarial loss. An adversarial loss is a value expressed by a loss function in adversarial learning. L _id represents an identity mapping. The identity mapping exists in the objective function L so that the objective function L does not change when the input to the mapping G _x→y is not the audio signal x but the audio signal y. The value of the identity mapping L _id represents the identity mapping loss.

L1 represents a loss function in adversarial learning performed by the first generation unit 110 and the first identification unit 120 in cooperation. L2 represents a loss function in adversarial learning performed by the second generation unit 150 and the second identification unit 160 in cooperation. L3 is a function representing a cyclic consistent loss in CycleGAN. That is, in L3, mapping G _{x → y} and mapping G _{y →} This is a function that indicates whether there is a one-to-one correspondence.

In this way, the objective function L is a function represented by a function representing an adversarial loss, a function representing a consistent loss, and a function representing an identity mapping loss.

Here, an example of the flow of each of the forward transform signal identification process, forward transform learning process, forward transform signal identification learning process, inverse transform signal identification process, inverse transform learning process, and inverse transform signal identification learning process will be described. The forward conversion signal identification process is a process in which the first identification unit 120 identifies whether the input audio signal is a natural signal or a forward conversion signal. The forward transformation learning process is a process learned by the first generation unit 110. The forward conversion signal identification learning process is a process that the first identification unit 120 learns. The inversely transformed signal identification process is a process in which the second identification unit 160 identifies whether the input audio signal is an inversely transformed signal or a pre-synthesized signal. The inverse transformation learning process is a process learned by the second generation unit 150. The inverse transformation signal identification learning process is a process that the second identification unit 160 learns.

FIG. 3 is a flowchart illustrating an example of the flow of forward conversion signal identification processing in the embodiment. The audio waveform identification unit 121 acquires the audio signal input to the first identification unit 120, and determines whether the audio signal input to the first identification unit 120 is a natural signal or a forward conversion signal based on the acquired audio waveform. It is determined whether there is one (step S101). Next, the audio feature identifying unit 122 acquires the audio feature of the audio signal input to the first identifying unit 120, and based on the acquired audio feature, the audio signal input to the first identifying unit 120 is determined to be a natural signal. and a forward conversion signal (step S102). Next, the integrated identification unit 123 analyzes the audio signal input to the first identification unit 120 according to a predetermined rule determined in advance based on the identification result of the audio waveform identification unit 121 and the identification result of the audio feature amount identification unit 122. It is determined whether the signal is a natural signal or a forward-transformed signal (step S103). The identification result of the integrated identification unit 123 in step S103 is output to the first determination unit 140.

FIG. 4 is a first flowchart showing an example of the flow of forward transformation learning processing in the embodiment. The first input determining unit 130 determines the audio signal input to the first identifying unit 120 as a forward conversion signal (step S201). Next, the first generation unit 110 acquires one composite signal from the composite signal group, and generates a forward transformation signal by performing a forward transformation process on the acquired composite signal (step S202). Next, the first generation unit 110 outputs the generated forward conversion signal to the first identification unit 120 (step S203). Next, the first identification unit 120 performs forward conversion signal identification processing on the acquired audio signal (step S204). That is, the processes from step S101 to step S103 are executed. Next, the first determination unit 140 determines whether the identification result of the first identification unit 120 is correct by comparing it with the determination result of the first input determination unit 130 (step S205). Next, the first generation unit 110 learns to further improve the natural signal level through forward transformation processing based on the determination result of the first determination unit 140 (step S206). Specifically, the first generation unit 110 learns to make the objective function L smaller.

FIG. 5 is a second flowchart showing an example of the flow of forward transformation learning processing in the embodiment. Hereinafter, processes similar to those shown in FIG. 3 or 4 will be designated by the same reference numerals as those in FIG. 3 or 4, and a description thereof will be omitted.

The second generation unit 150 outputs an inversely transformed signal (step S301). Next, the first generation unit 110 acquires the inversely transformed signal outputted by the second generating unit 150, and generates a forwardly transformed signal by performing a forward transformation process on the acquired inversely transformed signal (step S302). . Next, the processes from step S203 to step S206 are executed.

FIG. 6 is a flowchart illustrating an example of the flow of forward conversion signal identification learning processing in the embodiment. Hereinafter, processes similar to those shown in FIGS. 3 to 5 will be designated by the same reference numerals as in FIGS. 3 to 5, and a description thereof will be omitted.

The first input determining unit 130 determines whether the audio signal input to the first identifying unit 120 is a natural signal or a forward-converted signal (step S401). Next, the processes of step S204 and step S205 are executed. Next, the first identification unit 120 learns to further improve the accuracy of identification (step S402). Specifically, the first identification unit 120 learns to make the objective function L larger. More specifically, the speech waveform identification section 121 and the speech feature amount identification section 122 learn to make the objective function L larger.

FIG. 7 is a flowchart illustrating an example of the flow of inversely transformed signal identification processing in the embodiment. The audio waveform identification unit 161 acquires the audio waveform of the audio signal input to the second identification unit 160, and based on the acquired audio waveform, the audio signal input to the second identification unit 160 is divided into an inversely converted signal and a pre-synthesized signal. (Step S501). Next, the audio feature identification unit 162 acquires the audio signal input to the second identification unit 160, and based on the acquired audio feature, the audio signal input to the second identification unit 160 is pre-synthesized with the inversely transformed signal. It is determined which one is the signal (step S502). Next, the integrated identification unit 163 analyzes the audio signal input to the second identification unit 160 according to a predetermined rule determined in advance based on the identification result of the audio waveform identification unit 161 and the identification result of the audio feature amount identification unit 162. It is determined whether the signal is an inversely transformed signal or a pre-synthesized signal (step S503). The identification result of the integrated identification unit 163 in step S503 is output to the second determination unit 180.

FIG. 8 is a flowchart illustrating an example of the flow of inverse transformation learning processing in the embodiment. The second input determining unit 170 determines the audio signal input to the second identifying unit 160 as an inversely transformed signal (step S601). Next, the second generation unit 150 generates an inversely transformed signal by acquiring the forwardly transformed signal and performing an inversely transforming process on the acquired forwardly transformed signal (step S602). Next, the second generation unit 150 outputs the generated inversely transformed signal to the second identification unit 160 (step S603). Next, the second identification unit 160 performs inversely transformed signal identification processing on the acquired audio signal (step S604). That is, the processes from step S401 to step S403 are executed. Next, the second determination unit 180 determines whether the identification result of the second identification unit 160 is correct by comparing it with the determination result of the second input determination unit 170 (step S605). Next, the second generation unit 150 learns to further improve the natural signal level through inverse transformation processing based on the determination result of the second determination unit 180 (step S606). Specifically, the second generation unit 150 learns to make the objective function L smaller. Note that even when the second generation unit 150 acquires a natural signal and generates an inversely transformed signal, the processes from step S602 to step S606 are similarly performed.

FIG. 9 is a flowchart illustrating an example of the flow of inversely transformed signal identification learning processing in the embodiment. Hereinafter, processes similar to those shown in FIG. 7 or 8 will be given the same reference numerals as those in FIG. 7 or 8, and a description thereof will be omitted.

The second input determining unit 170 determines whether the audio signal input to the second identifying unit 160 is a natural signal or an inversely transformed signal (step S701). Next, the processes of step S604 and step S605 are executed. Next, the second identification unit 160 learns to further improve the accuracy of identification (step S702). Specifically, the second identification unit 160 learns to make the objective function L larger. More specifically, the speech waveform identification section 161 and the speech feature amount identification section 162 learn to make the objective function L larger.

FIG. 10 is a flowchart showing an example of the flow of processing executed by the audio signal conversion model learning device 1 in the embodiment. In FIG. 10, an example of the flow of subsequent processing will be described using an example in which step S201 is performed. Further, in FIG. 10, an example of the flow of processing will be described using as an example the case where the processing in step S601 is performed. Hereinafter, processes similar to those shown in FIGS. 3 to 9 will be designated by the same reference numerals as those shown in FIGS. 3 to 9, and a description thereof will be omitted.

Starting from step S201, processing is executed in the order of step S202, step S203, step S204, step S205, step S206, step S402, step S601, step S602, step S604, step S605, step S606, and step S702. After step S702, it is determined whether the termination condition is satisfied (step S801). The termination condition is, for example, that the number of times of learning exceeds a predetermined number of times. Whether or not the termination condition is satisfied is determined, for example, by the management unit 102, which will be described later.

If the termination condition is satisfied (step S801: YES), the process ends. On the other hand, if the end condition is not satisfied (step S801: NO), the process of step S301 is executed. Next, the process of step S302 is executed. After step S302, the process returns to step S203.

Note that the processing in step S206 and the processing in step S402 may be executed in reverse order. Note that the processing in step S606 and the processing in step S702 may be executed in reverse order.

Note that if the first input determining unit 130 executes a process of determining the audio signal input to the first identifying unit 120 as a natural signal instead of the process of step S201, the processes of steps S602 to S302 are not executed. . In such a case, the process ends after the process in FIG. 6 is executed.

Note that if the second input determining unit 170 determines the audio signal input to the second identifying unit 160 to be a natural signal instead of the process in step S601, the processes in steps S602 to S604 and step S606 processing is not executed.

In this way, the audio signal conversion model learning device 1 performs the forward conversion signal identification process, the forward conversion learning process, the forward conversion signal identification learning process, the inverse conversion signal identification process, the inverse conversion learning process, and the inverse conversion signal identification learning process. Obtain an audio signal conversion model with a higher degree of natural signal each time it is trained.

FIG. 11 is a diagram showing an example of the hardware configuration of the audio signal conversion model learning device 1 in the embodiment.
The audio signal conversion model learning device 1 includes a control unit 10 including a processor 91 such as a CPU (Central Processing Unit) and a memory 92 connected via a bus, and executes a program. The audio signal conversion model learning device 1 functions as a device including a control section 10, an input section 11, an interface section 12, a storage section 13, and an output section 14 by executing a program. More specifically, processor 91 reads a program stored in storage unit 13 and stores the read program in memory 92 . When the processor 91 executes the program stored in the memory 92, the audio signal conversion model learning device 1 functions as a device including a control section 10, an input section 11, an interface section 12, a storage section 13, and an output section 14. do.

The control unit 10 controls the operations of various functional units included in the audio signal conversion model learning device 1. The control unit 10 executes, for example, a forward transformation signal identification process, a forward transformation learning process, a forward transformation signal identification learning process, an inverse transformation signal identification process, an inverse transformation learning process, and an inverse transformation signal identification learning process.

The input unit 11 includes input devices such as a mouse, a keyboard, and a touch panel. The input unit 11 may be configured as an interface that connects these input devices to its own device. The input unit 11 receives input of various information regarding the own device. The input unit 11 receives, for example, an input instructing to start learning. The input unit 11 receives, for example, input of a composite signal to be added to the composite signal group. The input unit 11 receives, for example, an input of a natural signal to be added to the natural signal group.

The interface unit 12 includes a communication interface for connecting the own device to an external device. The interface unit 12 communicates with an external device via wire or wireless. The external device may be, for example, a storage device such as a USB (Universal Serial Bus) memory. For example, when the external device outputs a composite signal, the interface unit 12 acquires the composite signal output by the external device through communication with the external device. When the external device outputs a natural signal, for example, the interface unit 12 acquires the natural signal output by the external device through communication with the external device.

The interface unit 12 is configured to include a communication interface for connecting the own device to the audio signal conversion device 2 . The interface unit 12 communicates with the audio signal conversion device 2 via wire or wireless. The interface unit 12 outputs the audio signal conversion model to the audio signal conversion device 2 through communication with the audio signal conversion device 2 .

The storage unit 13 is configured using a non-temporary computer-readable storage medium device such as a magnetic hard disk device or a semiconductor storage device. The storage unit 13 stores various information regarding the audio signal conversion model learning device 1. The storage unit 13 stores, for example, a group of natural signals in advance. The storage unit 13 stores a composite signal group in advance, for example. The storage unit 13 stores, for example, a composite signal and a natural signal input via the input unit 11 or the interface unit 12. The storage unit 13 stores, for example, the identification result of the first identification unit 120.

The storage unit 13 stores, for example, the identification result of the second identification unit 160. The storage unit 13 stores, for example, the determination result of the first determination unit 140. The storage unit 13 stores, for example, the determination result of the second determination unit 180. The storage unit 13 stores, for example, the determination result of the first input determination unit 130. The storage unit 13 stores, for example, the determination result of the second input determination unit 170. The storage unit 13 stores, for example, an audio signal conversion model.

The output unit 14 outputs various information. The output unit 14 includes a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, and an organic EL (Electro-Luminescence) display. The output unit 14 may be configured as an interface that connects these display devices to its own device. The output unit 14 outputs, for example, information input to the input unit 11.

FIG. 12 is a diagram showing an example of the functional configuration of the control unit 10 in the embodiment. The control unit 10 includes a managed unit 101 and a management unit 102. The managed unit 101 includes a first generation unit 110, a first identification unit 120, a first input determination unit 130, a first determination unit 140, a second generation unit 150, a second identification unit 160, a second input determination unit 170, and A second determination unit 180 is provided. The managed unit 101 performs forward transformation signal identification processing, forward transformation learning processing, forward transformation signal identification learning processing, inverse transformation signal identification processing, inverse transformation learning processing, and An audio signal conversion model is obtained by executing the inverse conversion signal identification learning process. Specifically, the audio signal conversion model is a trained model representing forward conversion processing by the first generation unit 110.

The management unit 102 controls the operation of the managed unit 101. For example, the management unit 102 controls the execution of each of forward transformation signal identification processing, forward transformation learning processing, forward transformation signal identification learning processing, inverse transformation signal identification processing, inverse transformation learning processing, and inverse transformation signal identification learning processing by the managed unit 101. Control timing.

The management unit 102 controls, for example, the operations of the input unit 11, the interface unit 12, the storage unit 13, and the output unit 14. The management unit 102 reads out various information from the storage unit 13, for example, and outputs it to the managed unit 101. The management unit 102 acquires information input to the input unit 11, for example, and outputs it to the managed unit 101. For example, the management unit 102 acquires information input to the input unit 11 and records it in the storage unit 13. The management unit 102 acquires information input to the interface unit 12, for example, and outputs it to the managed unit 101. Information input to the management unit 102, for example, the interface unit 12, is acquired and recorded in the storage unit 13. The management unit 102 causes the output unit 14 to output information input to the input unit 11, for example.

The management unit 102 records the identification result of the first identification unit 120 in the storage unit 13, for example. The management unit 102 records the identification result of the second identification unit 160 in the storage unit 13, for example. The storage unit 13 records the determination result of the first determination unit 140 in the storage unit 13, for example. The storage unit 13 records the determination result of the second determination unit 180 in the storage unit 13, for example. The storage unit 13 records the determination result of the first input determination unit 130 in the storage unit 13, for example. The storage unit 13 records, for example, the determination result of the second input determination unit 170 in the storage unit 13.

FIG. 13 is a diagram showing an example of the hardware configuration of the audio signal conversion device 2 in the embodiment.
The audio signal conversion device 2 includes a control unit 20 including a processor 93 such as a CPU and a memory 94 connected via a bus, and executes a program. The audio signal conversion device 2 functions as a device including a control section 20, an input section 21, an interface section 22, a storage section 23, and an output section 24 by executing a program. More specifically, the processor 93 reads the program stored in the storage unit 23 and stores the read program in the memory 94. When the processor 93 executes the program stored in the memory 94, the audio signal conversion device 2 functions as a device including a control section 20, an input section 21, an interface section 22, a storage section 23, and an output section 24.

The control unit 20 controls the operations of various functional units included in the audio signal conversion device 2. The control unit 20 converts the unnatural composite signal into a natural composite signal using, for example, the audio signal conversion model obtained by the audio signal conversion model learning device 1.

The input unit 21 includes input devices such as a mouse, a keyboard, and a touch panel. The input unit 21 may be configured as an interface that connects these input devices to its own device. The input unit 21 receives input of various information regarding the own device. The input unit 21 receives, for example, an input instructing the start of processing for converting an unnatural composite signal into a natural composite signal. The input unit 21 receives, for example, an input of an unnatural composite signal to be converted.

The interface unit 22 includes a communication interface for connecting the own device to an external device. The interface unit 22 communicates with an external device via wire or wireless. The external device is, for example, an output destination of the natural composite signal. In such a case, the interface unit 22 outputs the natural composite signal to the external device through communication with the external device. The external device used to output the natural synthetic signal is, for example, an audio output device such as a speaker.

The external device may be, for example, a storage device such as a USB memory that stores an audio signal conversion model. For example, when the external device stores an audio signal conversion model and outputs the audio signal conversion model, the interface unit 22 acquires the audio signal conversion model through communication with the external device.

The external device is, for example, an output source of the unnatural composite signal. In such a case, the interface unit 22 obtains the unnatural composite signal from the external device through communication with the external device.

The interface unit 22 includes a communication interface for connecting the own device to the audio signal conversion model learning device 1 . The interface unit 22 communicates with the audio signal conversion model learning device 1 via wire or wireless. The interface unit 22 acquires the audio signal conversion model from the audio signal conversion model learning device 1 through communication with the audio signal conversion model learning device 1 .

The storage unit 23 is configured using a non-temporary computer-readable storage medium device such as a magnetic hard disk device or a semiconductor storage device. The storage unit 23 stores various information regarding the audio signal conversion device 2. The storage unit 13 stores, for example, an audio signal conversion model acquired via the interface unit 22.

The output unit 24 outputs various information. The output unit 24 is configured to include a display device such as a CRT display, a liquid crystal display, an organic EL display, or the like. The output unit 24 may be configured as an interface that connects these display devices to its own device. The output unit 24 outputs the information input to the input unit 21, for example.

FIG. 14 is a diagram showing an example of the functional configuration of the control unit 20 in the embodiment. The control unit 20 includes a conversion target acquisition unit 201, a conversion unit 202, and an audio signal output control unit 203.

The conversion target acquisition unit 201 acquires an unnatural composite signal to be converted. The conversion target acquisition unit 201 acquires, for example, an unnatural composite signal input to the input unit 21. The conversion target acquisition unit 201 acquires, for example, an unnatural composite signal input to the interface unit 22.

The conversion unit 202 converts the conversion target acquired by the conversion target acquisition unit 201 into a natural synthetic signal using an audio signal conversion model. The natural synthesized signal is output to the audio signal output control section 203.

The audio signal output control section 203 controls the operation of the interface section 22. The audio signal output control section 203 controls the operation of the interface section 22 to cause the interface section 22 to output a natural composite signal.

FIG. 15 is a flowchart illustrating an example of the flow of processing executed by the audio signal conversion device 2 in the embodiment. The control unit 20 acquires the unnatural composite signal input to the interface unit 22 (step S901). Next, the control unit 20 converts the unnatural composite signal into a natural composite signal using the audio signal conversion model stored in the storage unit 23 (step S902). Next, the control unit 20 controls the operation of the interface unit 22 to output the natural composite signal to the output destination (step S903). The output destination is, for example, an external device such as a speaker.

(Experimental result)
The experimental results of a comparison experiment (hereinafter referred to as "first experiment") between the audio signal conversion model obtained by the audio signal generation system 100 and the audio signal conversion model obtained by other learning methods are shown in FIGS. 16 and 17. show.

The first experiment was conducted using 437 sentences included in a dataset of Japanese voices uttered by female narrators. Out of 437 sentences in the Japanese speech dataset, 407 sentences (approximately 1 hour) were used to obtain the speech signal conversion model. Thirty sentences (four minutes) out of 437 sentences in the Japanese speech dataset were used to obtain a five-level MOS (Mean Opinion Score) evaluation of the naturalness of the sound quality. The audio sampling rate was 22.05kHz. There were 10 subjects. Each subject evaluated 30 and 20 randomly selected sentences for each learning method.

FIG. 16 is a first diagram showing an example of the experimental results of the first experiment. FIG. 17 is a second diagram showing an example of the experimental results of the first experiment. The horizontal axes in FIGS. 16 and 17 indicate a method for obtaining an audio signal conversion model. The vertical axes in FIGS. 16 and 17 indicate a five-level MOS evaluation regarding naturalness of sound quality. The dotted horizontal axis in FIGS. 16 and 17 represents the evaluation results of natural speech.

“SPSS” refers to a DNN (Deep Neural Network) text-to-speech synthesis (SPSS) method. “GANv” indicates a correction method on audio feature amounts. “V1” shows a method using a downsampling module for a convolutional neural network.

“V2” indicates a method of obtaining an audio signal conversion model by the audio signal generation system 100 including a first simple identification unit in place of the first identification unit 120 and a second simple identification unit in place of the second identification unit 160. . The first simple identification unit includes an audio waveform identification unit 121, and does not include an audio feature identification unit 122 and an integrated identification unit 123, and determines whether the input audio signal is a natural signal or a forward conversion signal based on the waveform of the input audio signal. It is a discriminator that identifies The second simple identification unit includes an audio waveform identification unit 161, and does not include an audio feature identification unit 162 and an integrated identification unit 163. This is a discriminator that identifies the

“V2msp” indicates a method for obtaining an audio signal conversion model by the audio signal generation system 100 including a third simple identification unit in place of the first identification unit 120 and a fourth simple identification unit in place of the second identification unit 160. . The third simple identification section includes an audio waveform identification section 121, an audio feature amount identification section 122, and an integrated identification section 123. The audio feature identification unit 122 included in the third simple identification unit uses the mel spectrogram of the audio signal to be identified as the feature used for identification. The fourth simple identification section includes an audio waveform identification section 161, an audio feature amount identification section 162, and an integrated identification section 163. The audio feature identification unit 162 included in the fourth simple identification unit uses the mel spectrogram of the audio signal to be identified as the feature used for identification.

“V2ph” indicates a method for obtaining an audio signal conversion model by the audio signal generation system 100 including a fifth simple identification unit in place of the first identification unit 120 and a sixth simple identification unit in place of the second identification unit 160. . The fifth simple identification section includes an audio waveform identification section 121, an audio feature amount identification section 122, and an integrated identification section 123. The audio feature identification unit 122 included in the fifth simple identification unit uses the phase spectrum of the audio signal to be identified as the feature used for identification. The sixth simple identification unit includes an audio waveform identification unit 161, an audio feature identification unit 162, and an integrated identification unit 163. The audio feature identification unit 162 included in the sixth simple identification unit uses the phase spectrum of the audio signal to be identified as the feature used for identification.

“V2mfcc” indicates a method of obtaining an audio signal conversion model by the audio signal generation system 100 including a seventh simple identification unit in place of the first identification unit 120 and an eighth simple identification unit in place of the second identification unit 160. . The seventh simple identification section includes an audio waveform identification section 121, an audio feature amount identification section 122, and an integrated identification section 123. The audio feature identification unit 122 included in the seventh simple identification unit uses the Mel frequency cepstral coefficient of the audio signal to be identified as the feature used for identification. The eighth simple identification section includes an audio waveform identification section 161 , an audio feature amount identification section 162 , and an integrated identification section 163 . The audio feature identification unit 162 included in the eighth simple identification unit uses the Mel frequency cepstral coefficient of the audio signal to be identified as the feature used for identification.

16 and 17 show that V1 significantly improves the sound quality compared to SPSS. Note that improving sound quality means increasing the natural signal level. 16 and 17 show that V1 improves the sound quality over GANv. Figures 16 and 17 show that V2 improves the sound quality over SPSS. Figures 16 and 17 show that V2 does not improve the sound quality over V1. This is because V2 generates a voice with more noise than V1. Figures 16 and 17 show that V2msp and V2mfcc have higher MOS ratings than V1, V2, V2ph, SPSS and GANv.

Furthermore, FIGS. 16 and 17 show that the p value of the two-sided Mann-Whitney test is 0.05 or more for V2msp and V2mfcc. This shows that the audio signal converted by V2msp and V2mfcc has no statistical difference from the natural signal. FIGS. 16 and 17 show that V2ph is a voice containing noise and has a lower MOS evaluation than V2. The results shown in FIGS. 16 and 17 suggest that it is effective to use a speech waveform discriminator (that is, speech waveform discriminators 121 and 161) and speech feature quantity discriminators 122 and 162. Note that in FIGS. 16 and 17, "V2msp", "V2ph", and "V2mfcc" are examples of processing for converting audio using the audio signal conversion model obtained by the audio signal generation system 100.

FIG. 18 shows the experimental results of a comparison experiment (hereinafter referred to as "second experiment") between the audio signal conversion model obtained by the audio signal generation system 100 and the audio signal conversion model obtained by another learning method.

The second experiment was conducted using 13,100 sentences included in the English speech dataset LJSpeech (see Reference 1). Forty sentences out of 13,100 sentences in the English speech dataset were used to obtain a five-point MOS rating for naturalness of sound quality. The audio sampling rate was 22.05kHz. There were 14 subjects. Each subject evaluated 15 sentences for each learning method. In the second experiment, spectral distortion was also calculated.

FIG. 18 is a diagram showing an example of the experimental results of the second experiment. FIG. 18 shows the least significant difference (LSD: Least squared distance) and MOS evaluation results for each learning method. WORLD is the method described in Reference 2. Griffin-Lim is a method described in Reference 3. Open WaveNet is a method described in Reference 4. WaveGlow is a method described in Reference 5.

Reference 1: “The LJ Speech Dataset” [online] [searched on March 30, 2020], Internet <URL: https://keithito.com/LJ-Speech-Dataset/>
Reference 2: M. Morise, F. Yokomori, and K. Ozawa, “WORLD: a vocoder-based high-quality speech synthesis system for real-time applications,” IEICE Transactions on Information and Systems, vol.99, no. 7, pp.1877-1884, 2016.
Reference 3: D. Griffin and J. Lim, “Signal estimation from modified short-time Fourier transform,” IEEE Transactions on Audio, Speech and Language Processing (TASLP), vol.32, no.2, pp. 236-243 , 1984.
Reference 4: Ryuichi Yamamoto et al. “WaveNet vocoder” [online] [searched on March 30, 2020], Internet <URL: https://doi.org/10.5281/zenodo.1472609>
Reference 5: R. Prenger, R. Valle, and B. Catanzaro, “WaveGlow: A flow-based generative network for speech synthesis,” 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.3617- 3621, 2019.

Figure 18 shows that Griffin-Lim has the lowest LSD. FIG. 18 shows that WORLD's LSD is highly distorted because it is a parametric vocoder. On the other hand, FIG. 18 shows that there is no difference in MOS evaluation between Griffin-Lim and WORLD.

FIG. 18 shows that when WaveGlow and open WaveNet are compared, open WaveNet has a larger LSD. On the other hand, FIG. 18 shows that when WaveGlow and open WaveNet are compared, WaveGlow has a higher MOS evaluation. These results indicate that LSD around 4 is likely not to affect MOS evaluation. Figure 18 shows that V2msp has the highest LSD and MOS ratings.

Note that in FIG. 18, "Recorded" indicates the recorded audio itself. “Recorded” is the target audio itself as the converted audio. Therefore, there is no LSD value corresponding to "Recorded".

The audio signal generation system 100 converts an input waveform into a waveform with high natural signal strength. Therefore, the audio signal generation system 100 is capable of converting the input audio into audio whose bandwidth has been restored, even if, for example, audio whose bandwidth has been reduced compared to the original audio (degraded audio) is input. I can do it. This means that the audio signal generation system 100 has expanded the band.

FIG. 19 shows the experimental results of a comparison experiment (hereinafter referred to as "third experiment") between the audio signal conversion model obtained by the audio signal generation system 100 and the audio signal conversion model obtained by another learning method.

In the third experiment, 50 sentences (3000 sentences in total) from each of 60 randomly selected speakers out of 109 speakers included in the English speech dataset VCTK (see Reference 6) were used to obtain a speech signal conversion model. was used for. In the third experiment, two men and two women were randomly selected from the remaining speakers, and then two sentences (total of 8 sentences) were randomly selected for each selected speaker, and the MUSHRA test regarding sound quality was conducted. It was conducted.

Reference 6: Ryuichi Yamamoto et al. “WaveNet vocoder” [online] [searched on March 30, 2020], Internet <URL: https://doi.org/10.5281/zenodo.1472609>

FIG. 19 is a diagram showing an example of the experimental results of the third experiment. The vertical axis in FIG. 19 shows the test results of the MUSHRA test. The horizontal axis in FIG. 19 indicates the method to be evaluated. "48" on the horizontal axis of FIG. 19 indicates natural speech sampled at 48 kHz. “16to48” on the horizontal axis in FIG. 19 indicates audio whose band has been expanded by the audio signal generation system 100.

“8to48” on the horizontal axis in FIG. 19 indicates audio whose band has been expanded by the audio signal generation system 100. “8to16to48” on the horizontal axis in FIG. 19 indicates audio whose band has been expanded by the audio signal generation system 100. The differences between "16to48", "8to48" and "8to16to48" are as follows.

In "16to48", "16" is input to the audio signal conversion device 2 as degraded audio, the audio signal conversion model obtained by the audio signal generation system 100 is applied to "16", and the band of "16" is expanded to 48kHz. shows the resulting audio. “16” indicates audio obtained by downsampling audio sampled at 48 kHz to 16 kHz.

In "8to16to48", "8" is input to the audio signal conversion device 2 as degraded audio, the audio signal conversion model obtained by the audio signal generation system 100 is applied to "8", and the band of "8" is expanded to 48kHz. shows the resulting audio. This shows the audio resulting from the band “16” being extended to 48 kHz. "8" indicates audio obtained by downsampling audio sampled at 48 kHz to 16 kHz.

In "8to16to48", "8" is input to the audio signal converter 2 as degraded audio and converted to "16", and then "16" is input as degraded audio to the audio signal converter 2 and converted to "48". This shows the voice. "48" indicates audio sampled at 48 kHz.

"16" on the horizontal axis of FIG. 19 indicates natural speech downsampled to 16 kHz. "4" on the horizontal axis of FIG. 19 indicates natural speech downsampled to 4 kHz.

FIG. 19 shows that "16to48" has a small difference from the original sound. FIG. 19 shows that "8to48" is significantly degraded compared to the original sound. The reason for the deterioration is that in audio, information is aggregated below 16 kHz, so downsampling to 8 kHz significantly reduced the amount of information, and learning did not go well. FIG. 19 shows that "8to16to48" has higher sound quality than "8to48".

The audio signal generation system 100 of the embodiment configured as described above uses not only one but both of the audio waveform and the audio feature of the audio signal, and performs forward conversion signal identification processing, forward conversion learning processing, and forward conversion signal identification learning. An audio signal conversion model is obtained by performing processing, inverse conversion signal identification processing, inverse conversion learning processing, and inverse conversion signal identification learning processing. Therefore, the audio signal generation system 100 configured in this manner can generate an audio signal with a higher degree of naturalness than when an audio signal conversion model is obtained using only audio waveforms. That is, the audio signal generation system 100 configured in this manner can generate sounds that are closer to the sounds produced by humans. Note that a method for obtaining an audio signal conversion model using only audio waveforms is, for example, SEGAN (Speech Enhancement Generative Adversarial Network).

Furthermore, the audio signal generation system 100 of the embodiment configured as described above includes a forward transform signal identification process, a forward transform learning process, a forward transform signal identification learning process, an inverse transform signal identification process, an inverse transform learning process, and an inverse transform signal An audio signal conversion model is obtained by executing the discrimination learning process. Therefore, the audio signal generation system 100 can generate a sound that is closer to human speech than when an audio signal conversion model is obtained only by a convolutional neural network using audio waveforms and audio features.

The audio signal generation system 100 according to the embodiment configured as described above includes forward transformation signal identification processing, forward transformation learning processing, forward transformation signal identification learning processing, inverse transformation signal identification processing, inverse transformation learning processing, and inverse transformation signal identification learning. Use processing. Therefore, even if the alignment of the audio signal used for learning is low, it is possible to generate a voice close to the voice uttered by a human. Therefore, the audio signal generation system 100 has the effect that the application situations are not limited more than SEGAN (Speech Enhancement Generative Adversarial Networks) (see Reference 7), which is effective only when alignment is high. Note that high alignment means that the difference between the learning audio signal and the user's ideal audio signal that the audio signal generation system 100 wants to output is small. A highly aligned learning audio signal is, for example, an audio signal obtained by superimposing noise on a computer on audio recorded in an ideal environment to simulate audio in a noisy environment, and then removing noise. A learning audio signal with low alignment is a synthesized speech generated by text-to-speech synthesis or speech conversion. Since the length of such audio signals also differs from signal to signal, alignment is also poor in this respect.

Reference 7: S. Pascual et al., “SEGAN: Speech enhancement generative adversarial network,” 2017 Annual Conference of the International Speech Communication Association (INTERSPEECH), pp.3642-3646, 2017.

(Modified example)
Note that the method by which the audio signal generation system 100 generates the audio signal conversion model does not necessarily need to be convolution CycleGAN. The method by which the audio signal generation system 100 generates the audio signal conversion model (hereinafter referred to as "model generation method") may be any method as long as it satisfies the following learning method conditions.

The learning method conditions include a first condition. The first condition is that the model generation method generates a forward transformed signal, which is a signal with a higher natural signal level than the audio signal, by executing a forward transform process, which is a transformation that increases the natural signal level, on the input audio signal. The condition is that the method uses the first generator that outputs the output.

The learning method condition includes a second condition. The second condition is that the model generation method is a method using a first discriminator that identifies whether the input signal is a forward-transformed signal or a natural signal.

The learning method condition includes a third condition. The third condition is that the model generation method outputs an inversely transformed signal with a lower natural signal intensity than the forward transformed signal by executing inverse transformation processing, which is a transformation that lowers the natural signal intensity, on the input signal. The condition is that the method uses a generator of

The learning method condition includes a fourth condition. The fourth condition is that the model generation method uses a second discriminator to identify whether the input signal is a pre-synthesized signal, which is a signal prepared in advance and a synthesized signal, or an inversely transformed signal. The condition is that the method is used. Note that the composite signal read out from the composite signal group by the second identification unit 160 is an example of a pre-composed signal.

The learning method conditions include a fifth condition. The fifth condition is that the model generation method includes a first generator, a first discriminator, a second generator, and a second discriminator. The condition is that learning is performed based on the identification results.

The learning method conditions may further include the following weak classifier conditions. The weak classifier condition includes a condition that at least one of the first classifier and the second classifier is trained using a voice waveform classifier and a voice feature classifier. Therefore, the model generation method is a method using, for example, a third generator different from the first generator and the second generator, and a third discriminator different from the first discriminator and the second discriminator. You can.

This is an example of a first generator of the first generation unit 110. The first identification unit 120 is an example of a first identifier. The second generation unit 150 is an example of a second generator. The second identification unit 160 is an example of a second identifier.

Note that as long as the method for generating the audio signal conversion model satisfies at least the first to fifth conditions, the audio signal generation system 100 will be able to recognize human voices even if the alignment of the audio signal used for learning is low. It is possible to generate sounds close to .

Note that the audio waveform identification unit 121 and the audio waveform identification unit 161 may identify the audio signal based on a frequency spectrum converted based on a perceptual measure of pitch. A perceptual measure of pitch is, for example, the Mel scale. The frequency spectrum transformed based on the perceptual measure of pitch is, for example, a spectrum expressed by Mel frequency cepstral coefficients. The frequency spectrum may be, for example, a phase spectrum. The frequency spectrum may be an amplitude spectrum. The frequency spectrum transformed based on a perceptual measure of pitch may be, for example, a mel spectrogram. In this way, information on human perception can also be used to generate speech based on the perceptual scale of pitch, so that the audio signal generation system 100 can generate speech that is closer to the speech produced by humans.

Note that the audio signal conversion model learning device 1 does not necessarily need to learn a learning model that converts an input audio signal into an audio signal with a sound similar to the human voice. The audio signal conversion model learning device 1 may learn a learning model that converts an input audio signal into an audio signal that is similar to the audio of an animal other than a human, such as a dog or a cat. In such a case, the audio signal conversion device 2 converts the input voice into an audio signal similar to the voice of an animal other than a human. As mentioned above, animals in this embodiment include humans.

Note that the unnatural signal and the natural synthetic signal are preferably audio signals of the same type of animal, but do not necessarily have to be the same.

Note that the managed section 101 is an example of a learning section. Note that the unnatural signal is an example of an input signal.

The audio signal conversion model learning device 1 may be implemented using a plurality of information processing devices communicably connected via a network. In this case, each functional unit included in the audio signal conversion model learning device 1 may be distributed and implemented in a plurality of information processing devices. For example, the first generation unit 110, the first identification unit 120, the second generation unit 150, and the second identification unit 160 may be implemented in different information processing devices.

The audio signal conversion device 2 may be implemented using a plurality of information processing devices communicatively connected via a network. In this case, each functional unit included in the audio signal conversion device 2 may be distributed and implemented in a plurality of information processing devices.

Note that all or part of each function of the audio signal generation system 100 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). good. The program may be recorded on a computer-readable recording medium. The computer-readable recording medium is, for example, a portable medium such as a flexible disk, magneto-optical disk, ROM, or CD-ROM, or a storage device such as a hard disk built into a computer system. The program may be transmitted via a telecommunications line.

Although the embodiments of the present invention have been described above in detail with reference to the drawings, the specific configuration is not limited to these embodiments, and includes designs within the scope of the gist of the present invention.

100...Audio signal generation system, 1...Audio signal conversion model learning device, 2...Audio signal conversion device, 10...Control unit, 11...Input unit, 12...Interface unit, 13...Storage unit, 14...Output unit, 101... Managed section, 102... Management section, 110... First generation section, 120... First identification section, 121... Audio waveform identification section, 122... Audio feature amount identification section, 123... Integrated identification section, 130... First input determination 140...First determination unit, 150...Second generation unit, 160...Second identification unit, 161...Speech waveform identification unit, 162...Speech feature identification unit, 163...Integrated identification unit, 170...Second input determination 180... Second determination unit, 20... Control unit, 21... Input unit, 22... Interface unit, 23... Storage unit, 24... Output unit, 201... Conversion target acquisition unit, 202... Conversion unit, 203... Audio signal Output control unit, 91...Processor, 92...Memory, 93...Processor, 94...Memory

Claims (8)

A trained model that converts an input signal, which is an input audio signal, into an audio signal with a higher natural signal degree, which indicates the degree of similarity to a natural signal, which is the sound actually emitted by an animal, than the input signal is used in machine learning. Learn how to get it,
Equipped with
The machine learning method outputs a forward conversion signal, which is a signal with a higher natural signal level than the audio signal, by performing a forward conversion process on the input audio signal, which is a conversion that increases the natural signal level. a first generation unit; a first identification unit that identifies whether the input signal is a forward conversion signal or a natural signal; and an inverse conversion that lowers the natural signal level of the input audio signal. a second generation unit that outputs an inversely transformed signal having a lower natural signal intensity than the audio signal by performing processing; and a pre-synthesized signal in which the input signal is a previously prepared signal and is a synthesized signal. and a second identification unit that identifies which one is the inversely transformed signal, and the second identification unit learns based on the identification results of the first identification unit and the second identification unit,
Audio signal conversion model learning device. The machine learning method is a cyclic adversarial learning (CycleGAN: Cycle Generative Adversarial Networks) method,
The audio signal conversion model learning device according to claim 1. At least one of the first identification unit and the second identification unit includes an audio waveform discriminator that identifies whether or not the audio signal is a natural signal based on the waveform of the audio signal used for learning; learning using a voice feature discriminator that acquires a voice feature amount that is information that satisfies a condition, and identifies whether or not the voice signal is a natural signal based on the acquired voice feature amount;
The audio signal conversion model learning device according to claim 1 or 2. the audio waveform discriminator is a frequency spectrum of the audio signal transformed based on a perceptual measure of pitch;
The audio signal conversion model learning device according to claim 3. A trained model that converts an input signal, which is an input audio signal, into an audio signal with a higher natural signal degree, which indicates the degree of similarity to a natural signal, which is the sound actually emitted by an animal, than the input signal is used in machine learning. The machine learning method includes a learning unit that obtains a signal having a higher natural signal level than the input audio signal by performing forward conversion processing, which is a conversion that increases the natural signal level, on the input audio signal. a first generation unit that outputs a forward conversion signal, which is a signal; a first identification unit that identifies whether the input signal is a forward conversion signal or a natural signal; a second generation unit that outputs an inversely transformed signal having a lower natural signal strength than the audio signal by executing an inverse transform process that is a transformation that lowers the signal strength, and the input signal is a previously prepared signal; A second identification unit that identifies whether the synthesized signal is a pre-synthesized signal or an inversely transformed signal, the method is such that the second identification unit learns based on the identification results of the first identification unit and the second identification unit. a conversion unit that converts the input audio signal using the learned model obtained by a certain audio signal conversion model learning device;
An audio signal conversion device comprising: A trained model that converts an input signal, which is an input audio signal, into an audio signal with a higher natural signal degree, which indicates the degree of similarity to a natural signal, which is the sound actually emitted by an animal, than the input signal is used in machine learning. learning steps, how to get
has
The machine learning method outputs a forward conversion signal, which is a signal with a higher natural signal level than the audio signal, by performing a forward conversion process on the input audio signal, which is a conversion that increases the natural signal level. a first generation unit; a first identification unit that identifies whether the input signal is a forward conversion signal or a natural signal; and an inverse conversion that lowers the natural signal level of the input audio signal. a second generation unit that outputs an inversely transformed signal having a lower natural signal intensity than the audio signal by performing processing; and a pre-synthesized signal in which the input signal is a previously prepared signal and is a synthesized signal. and a second identification unit that identifies which one is the inversely transformed signal, and the second identification unit learns based on the identification results of the first identification unit and the second identification unit,
Audio signal conversion model learning method. A program for causing a computer to function as the audio signal conversion model learning device according to any one of claims 1 to 4. A program for causing a computer to function as the audio signal conversion device according to claim 5.

Images