JP7109004B2 - Estimation device, estimation method, and program - Google Patents

Description

Application of Article 30, Paragraph 2 of the Patent Act 1. On December 14, 2017, the Institute of Electronics, Information and Communication Engineers Technical Research Report IEICE Technical Report Vol. 117 No. 368 SP2017-55-SP2017-65 Speech Pages 17 to 20 disclose the technology related to "estimation device, estimation method, program, and onomatopoeia generation device". 2. Technology related to "estimation device, estimation method, program, and onomatopoeia generation device" dated February 27, 2018 in the 2018 Spring Research Presentation Meeting of the Acoustical Society of Japan Published about. 3. On April 15, 2018, at the 2018 IEEE International Conference on Acoustics, Speech, and Signal Processing ICASSP2018, technology related to "estimation device, estimation method, program, and onomatopoeia generation device" was disclosed. 4. On August 29, 2018, the technology related to "estimation device, estimation method, program, and onomatopoeia generation device" was published in the CD-ROM of the Acoustical Society of Japan 2018 Autumn Research Presentation. ５．２０１８年１１月５日付けで、ＴＡＭＰＥＲＥ ＵＮＩＶＥＲＳＩＴＹ ＯＦ ＴＥＣＨＮＯＬＯＧＹ ＴＵＴＣＲＩＳ Ｒｅｓｅａｒｃｈ Ｐｏｒｔａｌ Ｐｒｏｃｅｅｄｉｎｇｓ ｏｆ ｔｈｅ Ｄｅｔｅｃｔｉｏｎ ａｎｄ Ｃｌａｓｓｉｆｉｃａｔｉｏｎ ｏｆ Ａｃｏｕｓｔｉｃ Ｓｃｅｎｅｓ ａｎｄ Ｅｖｅｎｔｓ ２０１８ Ｗｏｒｋｓｈｏｐ（ＤＣＡＳＥ２０１８）において、「系列信号を形容する言語表現の生成方法、言語表現The technology related to the method of searching for series signals by

The present invention relates to an estimating device, an estimating method, and a program .

Attention is focused on environmental sound recognition. Objects treated in recognition have various characteristics. However, an appropriate method for expressing it has not been established. In addition, the demand for human-computer interaction is increasing with the progress of information technology. As examples of onomatopoeia generation systems, a method of dividing a sound waveform into segments and assigning phonemes to each segment (for example, Patent Document 1) and a conversion method by Connectionist Temporal Classification (CTC) (for example, Patent Document 2) are known. there is

Kazushi Ishihara, Yasushi Tsubota, Hiroshi G. Okuno. “Automatic Transformation of Environmental Sounds into Sound-Imitation Words Based on Japanese Syllable Structure” In Proc. EUROSPEECH, pp. 3185-3188, 2003. Koichi Miyazaki, Tomoki Hayashi, Kazuya Takeda, Tomoki Toda, "Conversion from Acoustic Events to Onomatopoeic Expressions Based on CTC", 2017 Autumn Meeting Proceedings, pp.19-20, Acoustical Society of Japan, 2017 September

Two types of problems are pointed out with respect to the method of Patent Document 1 described above. The first is the segmentation problem. That is, it is difficult to determine the segment corresponding to the phoneme of the onomatopoeia from the sound waveform. The second is the phoneme ambiguity problem. That is, the most suitable phoneme for each segment is not unique and depends on the listener. In the above Patent Document 2, acoustic signals are converted into onomatopoeia end-to-end using a neural network, and the segmentation problem is attempted to be solved by learning. However, the phoneme ambiguity problem is not considered.

An object of the present invention made in view of this point is to provide an estimation device, an estimation method, and a program capable of improving the accuracy of associating two pieces of data.

In order to solve the above problems, the estimation device according to the present invention includes:
An estimating device that associates first data with second data that is different from the first data,
a first feature extraction unit that extracts a feature amount in a predetermined feature space from the first data;
a data generation unit that generates second data from the feature amount in the feature space;
including
By associating the feature space and the space expressing the second data by the data generator,
expressing the first data and the second data in the same feature space ;
The first data is one of signal data and character data expressing or explaining the signal data, and the second data is signal data and character data expressing or explaining the signal data. characterized by being the other .

In order to solve the above problems, the estimation method according to the present invention includes:
An estimation method for associating second data, which is data different from the first data, with the first data,
a step of extracting a feature amount in a predetermined feature space from the first data by a first feature extraction unit;
a step of generating second data from the feature amount in the feature space by a data generation unit;
including
By associating the feature space and the space expressing the second data by the data generator,
further comprising expressing the first data and the second data in the same feature space;
The first data is one of signal data and character data expressing or explaining the signal data, and the second data is signal data and character data expressing or explaining the signal data. the other.

In order to solve the above problems, the program according to the present invention is
A computer functions as the estimation device.

According to the estimation device, estimation method, and program according to the present invention, it is possible to improve the accuracy of associating two pieces of data.

1 is a functional block diagram showing an onomatopoeia generating device of this embodiment; FIG. It is a figure which shows the flowchart which an onomatopoeia production|generation apparatus performs. It is a figure which shows the content of the evaluation experiment using an onomatopoeia generation apparatus. FIG. 11 is a first diagram showing the results of evaluation experiments; It is a second diagram showing the results of evaluation experiments. It is a third diagram showing the results of evaluation experiments. FIG. 10 is a first diagram showing subjective evaluation in evaluation experiments; FIG. 10 is a second diagram showing subjective evaluation in the evaluation experiment; 1 is a first diagram showing text-based retrieval in the prior art; FIG. Fig. 2 is a second diagram showing text-based retrieval in the prior art; 1 is a schematic diagram of an estimation device of this embodiment; FIG. It is a functional block diagram of the estimation device of this embodiment. FIG. 4 is a diagram showing an onomatopoeia generation model by Encoder-Decoder; It is a figure which shows the flowchart which an estimation apparatus performs. FIG. 4 is a first diagram showing processing executed by a learning unit of the estimation device; FIG. 11 is a second diagram showing processing executed by the learning unit of the estimation device; It is a figure which shows the process which the series latent feature extraction part of an estimation apparatus performs. FIG. 10 is a first diagram showing subjective evaluation in an evaluation experiment using an estimating device; FIG. 10 is a second diagram showing subjective evaluation in an evaluation experiment using the estimating device; FIG. 11 is a first diagram showing search results by the estimating device; It is a second diagram showing search results by the estimating device.

[Onomatopoeia generation method]
In this embodiment, an onomatopoeia generation device G is disclosed. The onomatopoeia generator G uses a generative model to derive a many-to-many relationship between the onomatopoeia and the acoustic signal. The onomatopoeia generating device G of this embodiment uses the Sequence-to-Sequence model disclosed in Non-Patent Document 3 below as a reference.
[Non-Patent Document 3] Ilya Sutskever, Oriol Vinyals, Quoc V. Le. “Sequence to Sequence Learning with Neural Networks.” In Z. Ghahramani, M. Welling, C. Cortes, ND Lawrence, and KQ Weinberger, editors, Advances in Neural Information Processing Systems 27, pp.3104-3112. Curran Associates, Inc., 2014.

Although each function of the onomatopoeia generation device G will be described in detail below, it is not intended to exclude other functions that the onomatopoeia generation device G has.

The processing executed by the onomatopoeia generation device G is executed by one or more processors. A processor may include one or more memories that store programs and information during operation for various processes. Memory includes volatile memory and non-volatile memory. The memory includes memory separate from the processor and memory internal to the processor. The processor includes a general-purpose processor that loads a specific program and executes a specific function, and a dedicated processor that specializes in specific processing.

FIG. 1 is a functional block diagram of the onomatopoeia generator G. As shown in FIG. As shown in FIG. 1, the onomatopoeia generation device G has an encoder 1 and a decoder 2. In FIG. The encoder 1 and decoder 2 are composed of recurrent neural networks such as RNN, Bi-directional LSTM (Long Short-term Memory). The configuration by Bi-directional LSTM is explained below.

The model used by the onomatopoeia generator G is as follows.

のテキストを生成する。 The processing executed by the onomatopoeia generation device G will be described with reference to the flowchart of FIG. At step S1, the encoder 1 extracts the latent variable z of the acoustic signal from the acoustic feature quantity X using the Bi-directional LSTM. The acoustic features may be, for example, Mel-frequency cepstrum coefficients, power spectrum by Fourier transform, and the like. A latent variable may be a set of numerical values. In step S2, the decoder 2 uses a two-layer LSTM for the extracted latent variable z to generate an onomatopoeia that maximizes the probability distribution p(l|z).

to generate the text of

The onomatopoeia generation device G uses the above model and performs training according to the following two types of proposed methods.
Vanilla Training: Minimize the cross-entropy at each time step for each onomatopoeia label.
Multi-task training: Applying the softmax function to the output layer, minimizing the square error with the probability distribution of the first phoneme calculated from the onomatopoeia label (sub-task) and the vanilla training task (main-task) Learn to multitask.

An evaluation experiment was conducted using a model to which the above two types of proposed methods are applied and a CTC model. In the experiment, data of RWCP Real Environment Speech/Sound Database (RWCP-SSD) provided by RWCP (Real World Computing Partnership) of New Information Processing Development Organization of Technology Research Association was used. The content of the experiment is as shown in FIG. The metrics are word error rate (WER) and average phoneme error rate (MPER). The word error rate (WER) is the rate at which generated words are not included in the test data, and is calculated by the following formula.
WER = number of word errors/number of test data

The average phoneme error rate (MPER) is the average of the best phoneme error rates of all test data and is calculated by the following equation.
MPER = (number of insertion errors + number of deletion errors + number of replacement errors) / teacher label phoneme sequence length

As a result of the experiment, WER and MPER are shown in FIGS. 4 and 5. FIG. FIG. 4 is a graph for 512 cells. At this time, the values of WER and MPER converged in about 15 epochs. In addition, the Sequence-to-Sequence model (" Seq2Seq”) showed a similar learning curve.

As shown in Fig. 5, both the two proposed approaches showed lower WER and MPER compared to CTC.

A comparison of the onomatopoeic words generated by the two proposed methods and the CTC model is shown in FIG. Both the Sequence-to-Sequence model and the CTC model output valid onomatopoeias for attenuated sounds. The two proposed methods output favorable results compared to the CTC model, especially for repetitive sounds.

Furthermore, subjective evaluation was obtained as to whether or not the word was accepted as an onomatopoeia. As shown in FIG. 7, the ratio of acceptable onomatopoeic words was relatively high in the two proposed methods. A comparison was also made by Scheffe's method. As shown in FIG. 8, the two proposed methods showed relatively high results. Error bars in FIG. 8 indicate 95% confidence intervals.

From the above experimental results, it was confirmed that the two proposed methods generate more valid onomatopoeia than the conventional method.

[Acoustic signal search method]
In the following, a method is disclosed for performing acoustic signal retrieval based on onomatopoeia queries using some of the above onomatopoeia generation models.

Conventionally, as a search method for acoustic signals, a search based on class classification (for example, applause, gunshot, soccer goal, etc.) is performed. However, it is difficult to pre-cover the classes with the required granularity. Another search method in the prior art is a text-based search for onomatopoeic words (eg, ton-ton, pi, etc.). In this method, as shown in FIG. 9, onomatopoeia tags are added in advance to all search target acoustic signals. The onomatopoeic tag is a text or a string of symbols. As a text, words that directly imitate the timbre such as "tonton", words related to the mode of sound such as sizzle, growl, etc. is useful. The symbol string is useful, for example, a phoneme symbol string used for phonetic symbols. The onomatopoeic tag is added manually or automatically based on the acoustic signal. In this search method, as shown in FIG. 10, the degree of similarity as a character string is calculated between the onomatopoeia query specified by the user and each onomatopoeia tag.

Such text-based retrieval has the following two problems. The first is that many different sounds are associated with the same text, resulting in a large number of search results that cannot be ranked. For example, there are countless sounds to which the onomatopoeic tag "ton" can be added. Text-based retrieval makes it difficult to rank them. Second, it is necessary to assign onomatopoeic tags in advance to a large number of sounds to be searched, which is generally difficult (no uniqueness, processing time or manpower required). When assigning text automatically using a program, it is difficult to tag a sufficient number while maintaining accuracy.

In order to solve the problem of the searching method as described above, an estimating device S is disclosed here that performs an estimating method as schematically shown in FIG. The estimation device S extracts latent variables from onomatopoeic queries, which are series data, by an encoder using a neural network. The estimating device S searches for an acoustic signal, which is series data, based on the acoustic signal-onomatopoeia divergence (the divergence between the acoustic signal and the onomatopoeia) defined by the latent variable. Details will be described later.

12 is a functional block diagram of the estimation device S. FIG. The estimating device S has a signal latent feature extractor 12 , a latent feature accumulator 13 , a sequence latent feature extractor 14 , a learner 15 and an estimator 16 . Although the acoustic signal storage unit 11 is provided outside the estimation device S in FIG. 12 , the estimation device S may further include the acoustic signal storage unit 11 . The processes performed by the signal latent feature extraction unit 12, the sequence latent feature extraction unit 14, the learning unit 15, and the estimation unit 16 are performed by one or more processors. A processor may include one or more memories that store programs and information during operation for various processes. Memory includes volatile memory and non-volatile memory. The memory includes memory separate from the processor and memory internal to the processor. The processor includes a general-purpose processor that loads a specific program and executes a specific function, and a dedicated processor that specializes in specific processing. The signal latent feature extractor 12 or the sequence latent feature extractor 14 corresponds to the feature extractor of the present invention. The estimator 16 corresponds to the data generator of the present invention.

The acoustic signal storage unit 11 and the latent feature storage unit 13 are composed of semiconductor memory, magnetic memory, or the like, for example. The acoustic signal storage unit 11 includes an acoustic signal database. The latent feature accumulation unit 13 includes a latent variable database.

Although each function of the estimator S is described in detail below, it is not intended to exclude other functions that the estimator S may have.

The signal latent feature extraction unit 12 of this embodiment extracts the first feature amount using a predetermined feature amount extraction process. The predetermined feature amount extraction process may be any process, but in the present embodiment, it is a process learned to represent feature amounts in the same latent space (also referred to as feature space) using learning data. The predetermined feature amount extraction process is, for example, an onomatopoeia generation model (see FIG. 13) by an encoder-decoder described in the following document.
Shota Igawa, Kunio Kashino, "Acoustic signal retrieval based on latent features with onomatopoeia as a query", 2018 Autumn Conference Proceedings, Acoustical Society of Japan, September 2018

Specifically, referring to the flowchart of FIG. 14, the signal latent feature extraction unit 12 acquires the acoustic signal x as the first data set from the acoustic signal storage unit 11 in step S11. A set of data in the present embodiment refers to a set of data that does not include different types of data in each set. For example, one data set consists only of text representing onomatopoeia, and another data set consists only of acoustic signals. However, it should be noted that the present invention works both when each set does not contain heterogeneous data and when each set contains multiple types of data. At step S12, the signal latent feature extraction unit 12 encodes the acquired acoustic signal x into a fixed-length vector in the intermediate layer (13L in FIG. 13). In this intermediate layer, a latent variable containing features related to both the acoustic signal and the onomatopoeia is extracted as the first feature amount. The signal latent feature extractor 12 regards this latent variable as the latent variable zx inherent in the input acoustic signal _x .

The onomatopoeia generation model used for extracting latent variables is not limited to the one described above, and other models may be used.

After the latent variable _zx is extracted, the latent feature accumulation unit 13 accumulates the latent variable _zx in step S13.

The learning unit 15 learns mapping from the first and second data to a common latent space, which is necessary for extracting latent features. The learning unit 15 can perform off-line learning in advance before data to be subjected to estimation processing such as search and onomatopoeia generation is given. Specifically, in step S14, as shown in FIG. 15, the learning unit 15 uses the _onomatopoeia generation model by the encoder-decoder described above. In addition, the decoder learns the transformation from the latent variable _zx to the onomatopoeic word l. For these learning, a pair of an acoustic signal x and an onomatopoeic word l linked in advance is used as teacher data. Here, the conversion processing using the decoder corresponds to associating a latent feature, that is, one point on the latent space with a certain onomatopoeic word. This fact can be used to learn the conversion process (encoder) from onomatopoeia to latent features. That is, in step _S15 , the learning unit 15, as shown in FIG. to the latent variable _zx . The encoder may have the same structure as the encoder of the onomatopoeia generation model by the encoder-decoder described above. In this way, the estimating device S can learn the mapping from the onomatopoeia to the latent space by learning the correspondence between the latent space and the space expressing the onomatopoeia using a large amount of data.

A series latent feature extraction unit 14 extracts a second feature quantity from the second data set. Specifically, in step S16, as shown in FIG. 17, the sequence latent feature extraction unit 14 extracts latent Extract variables. The series latent feature extraction unit 14 regards the extracted latent variable as the latent variable _zl inherent in the onomatopoeia.

As described above, the estimator S extracts the computable latent variable _zx and the latent variable _zl as latent variables existing in the same latent space.

Consider again the subject of the present invention. When an acoustic signal is converted into text and a search is performed based on the converted text, there is a problem that the converted text cannot be uniquely determined. Therefore, as described above, the estimating device S expresses the latent variables extracted from the acoustic signal that is the query and the latent variables extracted from the text stored in the database in the same latent space.

Returning to the description of the flowchart, in step S17, the estimation unit 16 compares the extracted latent variable _zx and latent variable _zl , and associates the onomatopoeic query with the acoustic signal based on the comparison result. For example, the estimation unit 16 calculates the distance D( _x , _l ) between the latent variable zx and the latent variable zl by any method using any distance measure such as squared error. Any distance calculation method may be used. The estimator 16 similarly performs distance calculation for each sample and ranks the acoustic signals based on the calculated distances. For example, the estimation unit 16 may present the acoustic signal with the shortest distance to the user. Thus, the estimating device S can obtain the degree of divergence between the two latent variables, and as a result, can obtain search results that match the characteristics of the acoustic signal itself.

An evaluation experiment was conducted using the estimation device S described above. In the evaluation experiment, two subjective evaluations were obtained. A first subjective evaluation is an evaluation of the degree of matching of the search signal to the query. The results are shown in FIG.

As shown in FIG. 18, most of the answers were "relatively true" for both the database of this embodiment and the external database. Of the respondents, 58.7% answered that the database of this embodiment and 39.7% answered that the search results matched the query.

The second subjective evaluation was obtained by comparing the retrieval method of this embodiment with a conventional text-based retrieval method. The results are shown in FIG. As a result, the search method of this embodiment was evaluated higher than the text-based search method. The significance level is 1%. Therefore, it can be said that the search method of this embodiment outputs more appropriate search results for the query.

20 and 21 show search results obtained by the search method of this embodiment. 20 and 21 are diagrams obtained by transforming three or more dimensions into two dimensions based on the principal components. As shown in FIG. 20, onomatopoeic words with the same classification tag are distributed close to each other even in the two-dimensional latent space. Also, as shown in FIG. 21, there are many double consonants (Q) on the lower left side. In addition, there are many long sounds (H) or repetitions on the right side. In this way, onomatopoeias with similar tendencies are distributed nearby.

As a result of the above two evaluation experiments, the following was shown. That is, the estimating device S correctly responds to the query and operates. The estimating device S has the generalization ability to search databases to which onomatopoeia tags are not attached. The estimating device S can obtain more relevant search results compared to text-based searches.

According to the above embodiment, the estimating device S extracts a feature amount in a predetermined feature space from the first data, and associates the feature space with the space representing the second data, thereby extracting the first data and the second data are expressed in the same feature space. This can improve the accuracy of associating the two data.

According to the above embodiment, the series latent feature extracting unit 14 extracts the feature amount as a query when receiving the input of the second data. The estimation unit 16 also uses the extracted feature amount to calculate the degree of divergence between the first data and the second data. The estimating unit 16 may use any index such as squared error as the measure of the distance. As a result, it is possible to perform distance calculation for the input query, so that user convenience can be further improved.

According to the above embodiment, the first data and the second data are series data. Also, the first data is one of an audio signal or text. Also, the second data is the other of an audio signal or text. As a result, various types of data can be associated, thereby further improving user convenience.

Although the present invention has been described with reference to drawings and embodiments, it should be noted that a person skilled in the art can easily make various variations and modifications based on this disclosure. Therefore, it should be noted that these variations and modifications are included within the scope of the present invention. For example, functions included in each means or each step can be rearranged so as not to be logically inconsistent, and multiple means or steps can be combined into one or divided. .

For example, in the acoustic signal search method described above, the estimation device S searches for acoustic signals using onomatopoeic words as queries. However, in other embodiments, on the contrary, the estimator S may search for onomatopoeia using the acoustic signal as a query. The estimating device S may use data other than onomatopoeia (such as moving image data or character data) as a query to retrieve data other than sound signals (such as cartoon data or moving image data).

In the acoustic signal search method described above, the estimating device S searches the series data of the acoustic signal from the series data of the onomatopoeia query. However, in other embodiments, the estimating device S may search for sequence data from data that is not sequence data (for example, image data or audio data based on speech such as mouth imitation). And vice versa. In yet another embodiment, the estimator S may retrieve non-sequence data from non-sequence data.

When the onomatopoeia generating device G or the estimating device S is configured by a computer, a program describing the processing content for realizing each function is stored in a storage means inside or outside the computer, and the central processing of the computer is performed. A computer can be operated by reading and executing this program by a device (CPU). In addition, such programs can be distributed by selling, assigning, or lending portable recording media such as DVDs or CD-ROMs. , and can be distributed by transferring the program from a server to another computer via a network. Also, a computer that executes such a program can once store, for example, a program recorded on a portable recording medium or a program transferred from a server in its own storage means. Also, as another embodiment of this program, the computer may read the program directly from a portable recording medium and execute processing according to the program. In addition, it is also possible to sequentially execute processing according to the received program.

G Onomatopoeia generation device 1 Encoder 2 Decoder S Estimation device 11 Acoustic signal storage unit 12 Signal latent feature extraction unit 13 Latent feature storage unit 14 Sequence latent feature extraction unit 15 Learning unit 16 Estimation unit

Claims (8)

An estimating device that associates first data with second data that is different from the first data,
a first feature extraction unit that extracts a feature amount in a predetermined feature space from the first data;
a data generation unit that generates second data from the feature amount in the feature space;
including
By associating the feature space and the space expressing the second data by the data generator,
expressing the first data and the second data in the same feature space ;
The first data is one of signal data and character data expressing or explaining the signal data, and the second data is signal data and character data expressing or explaining the signal data. An estimating device, characterized in that it is the other . The estimation device according to claim 1,
The first feature extraction unit, upon receiving the input of the second data, extracts a feature amount as a query,
The estimating device, wherein the second data is generated using the degree of divergence between the first data and the same type of data as the second data, which is calculated using the feature amount. The estimation device according to claim 1,
A second feature extraction unit that extracts from the second data a feature amount in a predetermined feature space learned by the second data output by the data generation unit,
The first data and the second data are expressed in the same feature space by associating the space expressing the data by the first feature extraction unit and the second feature extraction unit. , the estimator. In the estimation device according to any one of claims 1 to 3,
The estimating device, wherein the first data is one of an acoustic signal or text and the second data is the other of an acoustic signal or text. In the estimation device according to any one of claims 1 to 4,
further comprising a learning unit, the learning unit comprising:
Using pre-linked pairs of acoustic signals and onomatopoeias as training data, the encoder learns the conversion of input data from acoustic signals to latent variables using an onomatopoeia generation model by Encoder-Decoder. learning the conversion from latent variables to onomatopoeia,
estimation device. In the estimation device according to any one of claims 1 to 5,
The estimation device, wherein the feature amount is a latent variable. An estimation method for associating second data, which is data different from the first data, with the first data,
the first a step of extracting a feature amount in a predetermined feature space from the first data by a feature extraction unit;
a step of generating second data from the feature amount in the feature space by a data generation unit;
including
By associating the feature space and the space expressing the second data by the data generator,
further comprising expressing the first data and the second data in the same feature space;fruit,
The first data is one of signal data and character data expressing or explaining the signal data, and the second data is signal data and character data expressing or explaining the signal data. on the other hand,estimation method. A program for causing a computer to function as the estimation device according to any one of claims 1 to 6 .

Images