JP7208846B2 - Acoustic feature quantity conversion device, acoustic feature quantity conversion method, and acoustic feature quantity conversion program - Google Patents

Description

The present invention relates to an acoustic feature conversion device, an acoustic model learning device, an acoustic recognition device, an acoustic feature conversion method, and an acoustic feature conversion program.

For example, in Patent Document 1, even when training data for learning an acoustic model is insufficient, pseudo training data is generated in order to ensure performance at a practical level in acoustic recognition. , discloses a technique for adapting an acoustic model to a task targeted for acoustic recognition (hereinafter referred to as a recognition target task). Here, the recognition target task is a task whose acoustic features such as the physical state of the sound source, the transmission environment of the sound 1, and the noise type are different from those of the original acoustic model.

In general, the performance of acoustic recognition depends on the completeness of training data with respect to the complexity of the task to be recognized. The complexity of the task is how complex the various changes that the aforementioned acoustic features such as the physical state of the sound source, sound transmission environment, noise type, etc. can take.

In other words, when the training data for the recognition target task is insufficient, it is often the case that sufficient recognition accuracy cannot be obtained even if the acoustic model is adapted to the recognition target task as it is. Therefore, usually, after recording training data under a sufficient number of acoustic conditions, the training data is padded by data augmentation to form a training data group with the desired comprehensiveness. Training data sets are used to adapt acoustic models.

JP 2016-186515 A

However, like the above-described conventional technique, forming a learning data group with desired comprehensiveness requires a great deal of economic and time costs. Moreover, a learning data group with desired comprehensiveness requires a huge data storage capacity, and a large amount of computation time is required for adaptation of the acoustic model to the recognition target task. In addition, the acoustic model becomes more complex depending on the complexity of the task to be recognized, increasing the data storage capacity and computational time required for training.

In other words, a learning data group having the desired comprehensiveness to ensure sufficient acoustic recognition accuracy has a large amount of data, requires time and effort to record the data set, and requires a huge data storage capacity. Since the learning model becomes complicated, there is a problem that the learning time increases.

SUMMARY OF THE INVENTION The present invention has been made in view of the above-described problems, and has an object to obtain sufficient acoustic recognition accuracy even with a small amount of learning data.

In order to solve this problem, in the present invention, as one means, the acoustic feature value conversion device includes a sound source separation unit that separates an acoustic signal whose sound source is the target device from the acoustic signal acquired by the acoustic signal acquisition unit. , an acoustic feature quantity extraction unit for extracting an acoustic feature quantity from the acoustic signal separated by the sound source separation unit, and an acoustic feature quantity normalization unit or an acoustic feature quantity extension unit. The acoustic feature amount normalization unit uses an acoustic feature amount normalization model based on a correlation between an acoustic signal having the equipment as a sound source and an operation parameter of the equipment to operate the equipment acquired by the operation parameter acquisition unit. Based on the parameters, a normalized acoustic feature quantity is generated from the acoustic feature quantity extracted by the acoustic feature quantity extraction unit, which is the acoustic feature quantity at the time of the specific operating parameter of the device. The acoustic feature amount extension unit uses an acoustic feature amount extension model based on a correlation between an acoustic signal having the equipment as a sound source and an operation parameter of the equipment to determine the specific operation of the equipment acquired by the operation parameter acquisition unit. Based on the parameters, an expanded acoustic feature quantity is generated from the acoustic feature quantity extracted by the acoustic feature quantity extracting unit, which is the acoustic feature quantity at the time of any operating parameter of the device.

According to the present invention, for example, even with a small amount of learning data, sufficient acoustic recognition accuracy can be obtained.

FIG. 1 is a diagram exemplifying a functional configuration diagram of an acoustic feature quantity conversion device, an acoustic model learning device, and an acoustic recognition device according to the first embodiment. FIG. 2 is a diagram exemplifying the acoustic feature quantity conversion processing flow of the first embodiment. FIG. 3 is a diagram illustrating an acoustic model learning processing flow of the first embodiment; FIG. 4 is a diagram exemplifying the acoustic recognition processing flow of the first embodiment. FIG. 5 is a diagram exemplifying a functional configuration diagram of an acoustic feature value conversion device, an acoustic model learning device, and an acoustic recognition device according to the second embodiment. FIG. 6 is a diagram exemplifying the acoustic feature quantity conversion processing flow of the second embodiment. FIG. 7 is a diagram exemplifying the acoustic model learning processing flow of the second embodiment. FIG. 8 is a diagram exemplifying the acoustic recognition processing flow of the second embodiment. FIG. 9 is a diagram exemplifying a functional configuration diagram of an acoustic feature value conversion device, an acoustic model learning device, and an acoustic recognition device according to the third embodiment. FIG. 10 is a diagram exemplifying the acoustic feature quantity conversion processing flow of the third embodiment. FIG. 11 is a diagram exemplifying the acoustic model learning processing flow of the third embodiment. FIG. 12 is a diagram exemplifying the acoustic recognition processing flow of the third embodiment. FIG. 13 is a diagram exemplifying a functional configuration diagram of an acoustic feature value conversion device, an acoustic model learning device, and an acoustic recognition device according to the fourth embodiment. FIG. 14 is a diagram exemplifying the acoustic feature quantity conversion processing flow of the fourth embodiment. FIG. 15 is a diagram exemplifying the acoustic model learning processing flow of the fourth embodiment. FIG. 16 is a diagram exemplifying the acoustic recognition processing flow of the fourth embodiment. FIG. 17 is a diagram illustrating a computer that implements the acoustic feature value conversion device, the acoustic model learning device, and the acoustic recognition device.

Hereinafter, embodiments of the present invention will be described in detail based on the drawings. In this specification, the same reference numbers in each drawing indicate the same or similar configuration or processing, and in the description of the embodiments described later, only the differences from the previously described embodiments will be described, and the description later will be omitted. may occur. Further, each embodiment and each modification can be combined in whole or in part within the scope of the technical idea of the present invention and within the scope of matching.

In the following examples, by utilizing the fact that there is a correlation between the operating parameters of the device that is the task to be recognized and the observed input acoustic signal, feature amount data of the acoustic signal is extracted based on the operating parameters. Standardize. Here, the device is, for example, a motor, a press machine, or the like. The operating parameters are not limited to physical quantities such as the operating speed, operating cycle, pressure, and flow rate of the equipment, but also include parameters obtained from sensors and control devices attached to the equipment, such as equipment control signals. .

Assuming that the amount of training data in the recognition target task is not sufficient to train the acoustic model with sufficient recognition accuracy, we improve the recognition accuracy even with a small amount of training data, Reduce the time and effort required to record datasets, reduce the amount of data storage required, lighten machine learning models, and reduce training time. In the following embodiments, the normalization of the feature amount data of the acoustic signal, the learning of the acoustic model, and the acoustic recognition are generally performed according to the following flow of (1) to (5).

(1) Assuming that there is a correlation between an input acoustic signal emitted from a device serving as a task to be recognized and an operating parameter of this device, acoustic features extracted from the input acoustic signal and the operating parameter are A transducer is generated that converts the standardized acoustic feature value based on an arbitrary operating parameter of the device. Here, the operating parameters are, for example, the operating speed of equipment, flow rate, pressure, or other physical quantities.
(2) Using the generated converter, the learning data is converted into the normalized acoustic feature quantity.
(3) Learning an acoustic model using the converted normalized acoustic feature amount.
(4) Generate an acoustic recognizer using the trained acoustic model.
(5) Using the generated transducer, the input acoustic signal is converted into a normalized acoustic feature amount, and acoustic recognition of the input acoustic signal is performed using the acoustic recognizer.

In the first embodiment, when there is a correlation between an input acoustic signal observed from a device serving as a task to be recognized and an operating parameter of this device, the correlation is used to extract an input from the input acoustic signal and the operating parameter. Acoustic signals are converted into normalized acoustic features normalized to specific operating parameters. An example of a correlation is an increase in the spectral peaks of an acoustic signal as a function of equipment operating speed. Further, an acoustic model is trained based on learning data composed of the converted normalized acoustic feature quantity, and the acoustic model recognizes the normalized acoustic feature quantity converted by the acoustic feature quantity conversion device.

<Structures of Acoustic Feature Transformation Apparatus, Acoustic Model Learning Apparatus, and Acoustic Recognition Apparatus of Embodiment 1>
FIG. 1 is a diagram exemplifying a functional configuration diagram of an acoustic feature quantity conversion device, an acoustic model learning device, and an acoustic recognition device according to the first embodiment. FIG. 1 illustrates a configuration in which the acoustic model learning device 2 and the acoustic recognition device 3 share the acoustic feature quantity conversion device 1 . However, FIG. 1 is only for convenience of explanation, and the acoustic model learning device 2 and the acoustic recognition device 3 may be separate devices each including a separate acoustic feature value conversion device 1 . Alternatively, as shown in FIG. 1, the acoustic model learning device 2 and the acoustic recognition device 3 may be separate devices or an integrated device sharing the acoustic feature quantity conversion device 1 .

The acoustic feature quantity conversion device 1 of Example 1 includes a sound source separation unit 12, an acoustic feature quantity extraction unit 13, and an acoustic feature quantity normalization unit 14, as shown in FIG. The acoustic signal acquisition unit 10 shown in FIG. 1 is a sound collecting device such as a microphone that collects sounds emitted by a device serving as a recognition target task, as will be described later. The operating parameter acquisition unit 11 acquires the operating parameters of the device serving as the recognition target task, as will be described later.

FIG. 1 illustrates the case where the acoustic feature quantity conversion device 1 includes all of the sound source separation unit 12, the acoustic feature quantity extraction unit 13, and the acoustic feature quantity normalization unit 14. A configuration that does not include either or both of the extraction units 13 is also possible. Further, FIG. 1 illustrates a configuration in which the output of the sound source separation unit 12 is input to the acoustic feature amount extraction unit 13, and the output of the acoustic feature amount extraction unit 13 is input to the acoustic feature amount normalization unit 14. Alternatively, the output of the sound source separation unit 12 can be input to the acoustic feature amount normalization unit 14 , and the output of the acoustic feature amount normalization unit 14 can be used as the input of the acoustic feature amount extraction unit 13 .

Further, as shown in FIG. 1, the acoustic model learning device 2 of the first embodiment includes, in addition to each component of the acoustic feature quantity conversion device 1, a normalized acoustic feature quantity storage unit 21, an acoustic model learning unit 15, and acoustic model storage unit 22 . At least one of the normalized acoustic feature amount storage unit 21 and the acoustic model storage unit 22 may be configured as a storage unit independent of the acoustic model learning device 2 .

Further, the acoustic recognition device 3 of Example 1 includes a acoustic recognition unit 16 in addition to each component of the acoustic feature quantity conversion device 1, as shown in FIG.

<Acoustic Feature Quantity Conversion Processing of Embodiment 1>
FIG. 2 is a diagram exemplifying the acoustic feature quantity conversion processing flow of the first embodiment. Acoustic feature quantity conversion processing of the first embodiment is executed by the acoustic feature quantity conversion device 1 at the time of acoustic feature quantity conversion.

In FIG. 2, first, in step S10, the acoustic feature quantity conversion device 1 receives an input acoustic signal through the acoustic signal acquisition unit 10, and acquires data such as an operating parameter, such as an operating speed, through the operating parameter acquisition unit 11. is entered. An input acoustic signal is acquired by a sound collecting device such as a single or multiple microphones. At this time, the input acoustic signal may be acquired in real time using a microphone or the like, or may be acquired off-line with data recorded in a storage device such as a hard disk drive using a recording device such as a computer or a voice recorder. good too.

Further, the operation parameters are obtained, for example, from sensors installed in the equipment that is the task to be recognized, from control signals output by the control device of the equipment, and from input by the operator. , can be estimated from other observed values such as acoustic signals by a program or device. In addition to the operating speed, the operating parameters are not limited to physical quantities that indicate the operating state of the device, such as pressure, flow rate, and temperature. etc.

Subsequently, in step S11, the sound source separation unit 12 extracts the recognition target from the input acoustic signal input in step S10, which includes noise such as noise, reverberation, and sound from a sound source other than the device of the recognition target task, for example. A post-separation acoustic signal is calculated by extracting and separating only the acoustic signal emitted from the device of the task.

Subsequently, in step S12, the acoustic feature quantity extraction unit 13 calculates a digital acoustic feature quantity from the analog post-separation acoustic signal calculated in step S11. The acoustic feature quantity calculated at this time is, for example, a spectrogram, a mel feature quantity, a mel-cepstrum feature quantity, or the like.

Subsequently, in step S13, the acoustic feature quantity normalization unit 14 initializes the acoustic feature quantity normalization model based on the operating parameters. Specifically, the acoustic feature amount standardization unit 14 utilizes the fact that there is a correlation between the acoustic signal emitted when the device is in operation and the operating parameter, and uses the acoustic feature amount and the operating parameter data of the device to determine the operating parameters. A normalized feature quantity normalized by a parameter is calculated. Correlations between acoustic signals and operating parameters used at this time include, for example, correlations that can be calculated by simple arithmetic calculations such as proportionality, inverse proportions, and proportionality to squares, and correlations that can be obtained by statistical calculations or machine learning.

Here, an example in which the operating parameter is the operating speed will be described. Let s be the input sound signal, x be the input operating speed, and s′ be the normalized sound signal. The acoustic feature amount normalized model Fα normalized to the signal s′ is represented, for example, by the following equation (1).

At this time, the specific operating speed at the time of normalization is, for example, specified by the operator or automatically generated from the distribution of operating speeds of the learning data. Also, the standardization at this time is the non-dimensionalization of speed.

<Acoustic model learning process of the first embodiment>
FIG. 3 is a diagram illustrating an acoustic model learning processing flow of the first embodiment; The acoustic model learning process of the first embodiment is executed by the acoustic model learning device 2 during acoustic model learning. Steps S10 to S12 in FIG. 3 are the same as in FIG.

In FIG. 3, in step S14 subsequent to step S12, the acoustic model learning device 2 uses the acoustic feature quantity normalized model generated using the acoustic feature quantity conversion device 1 to obtain the input input in step S10. Acoustic features are standardized for acoustic signals. In step S<b>14 , the normalized acoustic feature amount normalized by the acoustic feature amount normalization model is stored in the normalized acoustic feature amount storage unit 21 .

Subsequently, in step S15, the acoustic model learning unit 15 learns the acoustic model using the normalized acoustic feature amount stored in the normalized acoustic feature amount storage unit 21 as learning data. Known methods such as Gaussian mixture models, support vector machines, and neural networks are used for learning. The acoustic model learned by the acoustic model learning unit 15 is stored in the acoustic model storage unit 22 .

<Acoustic Recognition Processing of Embodiment 1>
FIG. 4 is a diagram exemplifying the acoustic recognition processing flow of the first embodiment. The acoustic recognition processing of the first embodiment is executed by the acoustic recognition device 3 during acoustic recognition. Steps S10 to S12 in FIG. 4 are the same as in FIGS.

In FIG. 4, in step S14 subsequent to step S12, the acoustic recognition device 3 uses the acoustic feature quantity normalized model generated using the acoustic feature quantity conversion device 1 to input the input in step S10 of FIG. Acoustic feature quantity normalization of the input acoustic signal is performed.

In step S16 following step S14, the acoustic recognition unit 16 performs identification using the normalized acoustic features normalized by the acoustic feature conversion device 1 and the acoustic model stored in the acoustic model storage unit 22. .

2, the acoustic model learning process of the first embodiment shown in FIG. 3, and the acoustic recognition process of the first embodiment shown in FIG. The execution order of step S13 or step S14 and step S12 can be interchanged. At this time, for example, the separated acoustic signal output from the sound source separation unit 12 is input to the acoustic feature value normalization unit 14, and the normalized acoustic signal normalized by the operating speed, for example, is output. The normalized acoustic signal is input to the acoustic feature amount extraction unit 13, and the normalized acoustic feature amount is calculated.

As described above, according to the first embodiment, even if sufficient learning data for the recognition target task cannot be prepared, the data amount is effectively increased by normalizing the data using the operating parameters, and the accuracy is improved. A good learning model can be learned. Also, at the time of identification, similar data normalization is performed to improve identification accuracy. It also reduces the time and money costs required to record datasets, reduces the required storage capacity, makes machine learning models lighter, and reduces the time cost of training.

The second embodiment differs from the first embodiment in that the sound source separation unit and the acoustic feature amount extraction unit of the acoustic feature amount conversion device each refer to the operating parameters in the same manner as the acoustic feature amount normalization unit.

In the first embodiment, the operating parameters are referred to only in the acoustic feature amount normalization unit, but by referring to the operating parameters in the sound source separation unit and the acoustic feature amount extraction unit as well, it is possible to further increase the number of parameters necessary for data set recording. It is possible to reduce the time and money costs, reduce the required storage capacity, lighten the machine learning model, and reduce the time cost of learning. In the following, differences from the first embodiment described above will be mainly described.

<Structures of Acoustic Feature Transformation Apparatus, Acoustic Model Learning Apparatus, and Acoustic Recognition Apparatus of Embodiment 2>
FIG. 5 is a diagram exemplifying a functional configuration diagram of an acoustic feature value conversion device, an acoustic model learning device, and an acoustic recognition device according to the second embodiment. The acoustic feature quantity conversion device 1B of Example 2 includes a sound source separation unit 17, an acoustic feature quantity extraction unit 18, and an acoustic feature quantity normalization unit 14, as shown in FIG.

Further, as shown in FIG. 5, the acoustic model learning device 2B of the second embodiment includes, in addition to each component of the acoustic feature quantity conversion device 1B, a normalized acoustic feature quantity storage unit 21, an acoustic model learning unit 15, and acoustic model storage unit 22 .

Further, as shown in FIG. 5, the acoustic recognition device 3B of the second embodiment includes a acoustic recognition unit 16 in addition to each component of the acoustic feature quantity conversion device 1B.

The second embodiment differs from the first embodiment in that the sound source separation unit 17 and the acoustic feature value extraction unit 18 of the acoustic feature value conversion device 1B refer to the operating parameters acquired from the operating parameter acquisition unit 11. be.

Here, in the second embodiment, all of the sound source separation unit 17, the acoustic feature amount extraction unit 18, and the acoustic feature amount normalization unit 14 do not need to refer to the operating parameters. At least one or more processing units out of the unit 18 and the acoustic feature amount normalization unit 14 should refer to the operating parameters.

When referring to operation parameters, the sound source separation unit 17, the acoustic feature amount extraction unit 18, and the acoustic feature amount normalization unit 14 each use, for example, an acoustic signal with a device as a sound source and an operating parameter of the device, as will be described later. A neural network configured based on correlation with parameters may be used.

On the other hand, the sound source separation unit 17 becomes a processing unit including functions equivalent to those of the sound source separation unit 12 of the first embodiment when the operation parameter is not referred to. Further, the acoustic feature amount extraction unit 18 becomes a processing unit including processing functions equivalent to those of the acoustic feature amount extraction unit 13 of the first embodiment when the operating parameters are not referred to.

<Acoustic Feature Transformation Processing of Embodiment 2>
FIG. 6 is a diagram exemplifying the acoustic feature quantity conversion processing flow of the second embodiment. Acoustic feature quantity conversion processing of the second embodiment is executed by the acoustic feature quantity conversion device 1B at the time of acoustic feature quantity conversion. Step S10 in FIG. 6 is the same as in FIG.

In FIG. 6, in step S17 subsequent to step S10, the sound source separation unit 17 performs sound source separation based on the operating parameters obtained in step S10. Specifically, for example, consider a case where devices of the same model are operating side by side, and sound source separation based on sound source direction is impossible. At this time, if each device operates at a different operating speed, sound source separation can be performed by solving the combination problem of the operating speed of each device that is a sound source and the acoustic signal that can be emitted from each device. can be done. A well-known method such as a simple iterative calculation that tries all possible combinations or machine learning can be used as a method for performing sound source separation.

In step S18 following step S17, the acoustic feature amount extraction unit 18 extracts an acoustic feature amount with reference to the operating parameters acquired in step S10. Specifically, for example, (A) setting a bandpass filter, (B) forming a filter bank, and (C) changing a parameter of Fourier transform.

(A) The setting of the bandpass filter will be explained. If the operating speed is known, the band-pass filter can be set by using the fact that it is possible to estimate to some extent which frequency band contains the effective feature amount. The effective frequency band estimation for the operating parameters at this time is, for example, using parameters set manually by humans, using a simple arithmetic model, using a neural network, or the like.

(B) Formation of a filter bank will be described. As with the setting of the band-pass filter, if the operating speed is known, it is possible to estimate to some extent which frequency band contains the effective feature amount. filter bank can be formed. The formation of the filter bank at this time is, for example, one using parameters set manually by a person, one using a simple arithmetic model, one using a neural network, or the like.

(C) Change of parameters for Fourier transform will be described. Using the fact that it is possible to estimate how effective features are generated in an acoustic signal in accordance with operating parameters, it is possible to set Fourier transform parameters such as window size and shift length, for example. The parameters at this time are, for example, those using parameters set manually by humans, those using a simple arithmetic model, those using a neural network, and the like.

With the above (A) to (C), in addition to the effects obtained in Example 1, such as, for example, robust sound identification against noise is possible, and the compression efficiency of the sound signal is improved, It can reduce the time and money costs required to record datasets, reduce the required storage capacity, make machine learning models lighter, and reduce the time costs required for learning.

In step S19 following step S18, the acoustic feature quantity normalization unit 14 initializes the acoustic feature quantity normalization model. The acoustic feature quantity normalization unit 14 utilizes the fact that there is a correlation between an acoustic signal emitted when a device serving as a task to be recognized is in operation and an operating parameter, and uses the acoustic feature quantity and the operating parameter data of the device to determine the operating parameters. A normalized feature quantity normalized by a parameter is calculated.

At this time, the correlation between the acoustic signal to be used and the operating parameter includes, for example, correlation that can be expressed by simple arithmetic calculation such as proportionality, inverse proportion, and proportionality to the square, and correlation that can be obtained by statistical calculation or machine learning. .

For example, let s be the input acoustic signal, x be the input operating parameter, and s′ be the normalized acoustic signal. The acoustic feature amount normalization model Fα to be normalized to the post-acoustic signal s′ is represented, for example, by the following equation (2).

At this time, the specific operating parameters to be normalized are, for example, specified by the operator or automatically generated from the distribution of the operating parameters of the learning data. In addition, the standardization at this time is the non-dimensionalization of the operating parameters.

<Acoustic model learning process of the second embodiment>
FIG. 7 is a diagram exemplifying the acoustic model learning processing flow of the second embodiment. The acoustic model learning process of the second embodiment is executed by the acoustic model learning device 2B during acoustic model learning. Steps S10 and S17 to S18 in FIG. 7 are the same as in FIG.

In FIG. 7, in step S20 subsequent to step S18, the acoustic model learning device 2B uses the acoustic feature quantity normalized model generated using the acoustic feature quantity conversion device 1B and the operating parameters input in step S10. Then, the input acoustic signal input in step S10 is normalized. In step S<b>20 , the normalized acoustic feature amount normalized by the acoustic feature amount normalization model is stored in the normalized acoustic feature amount storage unit 21 .

In step S15 following step S20, the acoustic model learning unit 15 learns the acoustic model using the normalized acoustic feature amount stored in the normalized acoustic feature amount storage unit 21 as learning data. Known methods such as Gaussian mixture models, support vector machines, and neural networks are used for learning. The acoustic model learned by the acoustic model learning unit 15 is stored in the acoustic model storage unit 22 .

<Acoustic Recognition Processing of Embodiment 2>
FIG. 8 is a diagram exemplifying the acoustic recognition processing flow of the second embodiment. The acoustic recognition processing of the second embodiment is executed by the acoustic recognition device 3B during acoustic recognition. Steps S10, S17-S18 in FIG. 8 are the same as in FIGS.

In FIG. 8, in step S20 subsequent to step S18, the acoustic recognition device 3B uses the acoustic feature quantity normalized model generated using the acoustic feature quantity conversion device 1B to convert the operating parameters input in step S10 into Based on this, the input acoustic signal is normalized.

In step S16 subsequent to step S20, the acoustic recognition unit 16 identifies using the normalized acoustic features normalized by the acoustic feature conversion device 1B and the acoustic model stored in the acoustic model storage unit 22. I do.

6, the acoustic model learning process of the second embodiment shown in FIG. 7, and the acoustic recognition process of the second embodiment shown in FIG. , step S19 or step S20, and step S18 can be changed in order of execution, as in the first embodiment. At this time, for example, the separated acoustic signal output from the sound source separation unit 17 is input to the acoustic feature amount normalization unit 14, and the normalized acoustic signal or the like normalized by the operating parameter is output. The normalized acoustic signal is input to the acoustic feature amount extraction unit 18, and the normalized acoustic feature amount is output.

As described above, according to the second embodiment, even if sufficient learning data for a recognition target task cannot be prepared, in addition to normalizing data by operating parameters, operating parameters are also used for sound source separation and acoustic feature extraction. By referring to , it is possible to more effectively improve identification accuracy, reduce the time and money costs required for recording datasets, reduce the required storage capacity, reduce the weight of machine learning models, Reduce the time cost of learning.

The embodiment 3 differs from the embodiments 1 and 2 in that the acoustic feature amount conversion device 1C refers to the operating parameters and expands the acoustic feature amount instead of normalizing it. By performing data augmentation, we improve the completeness of the dataset and improve the recognition accuracy of the trained acoustic model. In the following, differences from the above-described second embodiment will be mainly described.

<Structures of Acoustic Feature Transformation Apparatus, Acoustic Model Learning Apparatus, and Acoustic Recognition Apparatus of Embodiment 3>
FIG. 9 is a diagram exemplifying a functional configuration diagram of an acoustic feature value conversion device, an acoustic model learning device, and an acoustic recognition device according to the third embodiment. The acoustic feature quantity conversion device 1C of Example 3 includes a sound source separation unit 17, an acoustic feature quantity extraction unit 18, and an acoustic feature quantity expansion unit 19, as shown in FIG.

Further, as shown in FIG. 9, the acoustic model learning device 2C of the third embodiment includes, in addition to each component of the acoustic feature quantity conversion device 1C, an expanded acoustic feature quantity storage unit 23, an acoustic model learning unit 15, and An acoustic model storage unit 22 is included.

Moreover, as shown in FIG. 9, the acoustic recognition device 3C of Example 3 includes a acoustic recognition unit 16 in addition to each component of the acoustic feature quantity conversion device 1C.

The third embodiment is different from the second embodiment in that, in the acoustic feature quantity conversion apparatus 1C, the acoustic feature quantity expansion unit 19 expands the data instead of normalizing the acoustic feature quantities.

Here, in the third embodiment, it is not necessary for all of the sound source separation unit 17, the acoustic feature amount extraction unit 18, and the acoustic feature amount extension unit 19 to refer to the operating parameters, and the sound source separation unit 17, the acoustic feature amount extraction unit 18, and at least one or more of the acoustic feature amount extension units 19 refer to the operating parameters.

When referring to operation parameters, the sound source separation unit 17, the acoustic feature amount extraction unit 18, and the acoustic feature amount expansion unit 19 each use, for example, an acoustic signal with the device as the sound source and the device's operation parameter It may be one using a neural network configured based on the correlation with.

On the other hand, the sound source separation unit 17 becomes a processing unit including functions similar to those of the sound source separation unit 12 of the first embodiment when the operation parameter is not referred to. Further, the acoustic feature quantity extraction unit 18 becomes a processing unit including processing functions similar to those of the acoustic feature quantity extraction unit 13 of the first embodiment when the operating parameters are not referred to. In addition, when the acoustic feature amount extending unit 19 does not refer to the operating parameters, the acoustic feature amount extending unit 19 becomes a processing unit that performs data extension without using the operating parameters.

<Acoustic Feature Transformation Processing of Embodiment 3>
FIG. 10 is a diagram exemplifying the acoustic feature quantity conversion processing flow of the third embodiment. Acoustic feature quantity conversion processing of the third embodiment is executed by the acoustic feature quantity conversion device 1C at the time of acoustic feature quantity conversion. Steps S10 and S17 to S18 in FIG. 10 are the same as in FIG.

In FIG. 10, in step S21 subsequent to step S18, the acoustic feature amount expansion unit 19 uses the acoustic feature amount extracted by the acoustic feature amount extraction unit 18 to convert the acoustic feature amount into an acoustic feature amount with a plurality of operating parameters. By transforming and expanding into quantities, an expanded acoustic feature quantity group is generated. The acoustic feature amount extension unit 19 utilizes the fact that there is a correlation between the acoustic signal emitted when the device is in operation and the operating parameter.

Correlations between acoustic signals and operating parameters used at this time include, for example, correlations that can be expressed by simple arithmetic calculations such as proportionality, inverse proportions, and squared proportions, and correlations that can be obtained by statistical calculations or machine learning.

Here, assuming that the acoustic signal or acoustic feature quantity at a specific operating parameter x is sx, the acoustic signal or acoustic feature quantity sα at an arbitrary operating parameter α is obtained by the acoustic feature quantity extension model F′x as follows: It is shown like Formula (3).

By using the above equation (3), in the acoustic feature quantity conversion device 1C, the acoustic feature quantity expansion unit 19 converts the input acoustic feature quantity output from the acoustic feature quantity extraction unit 18 into a plurality of arbitrary operating parameters. It converts and expands the acoustic features to generate an expanded acoustic feature group.

<Acoustic model learning process of the third embodiment>
FIG. 11 is a diagram exemplifying the acoustic model learning processing flow of the third embodiment. The acoustic model learning process of the third embodiment is executed by the acoustic model learning device 2C during acoustic model learning. Steps S10 and S17-S18 in FIG. 11 are the same as in FIG.

In FIG. 11, in step S21 subsequent to step S18, the acoustic model learning device 2C uses the acoustic feature conversion device 1C to convert and expand the acoustic feature quantity into acoustic feature quantity with a plurality of operating parameters. For example, when the operating parameter is the operating speed, the acoustic feature amount extension unit 19 converts the acoustic feature amount extracted by the acoustic feature amount extracting unit 18 into acoustic feature amounts at a plurality of operating speeds. And by expanding, an expanded acoustic feature quantity group is generated and stored in the expanded acoustic feature quantity storage unit 23 .

Subsequently, in step S22, the acoustic model learning unit 15 learns the acoustic model using the extended acoustic feature amount stored in the extended acoustic feature amount storage unit 23 as learning data. Known techniques such as Gaussian mixture models, support vector machines, and neural networks are used for learning. The acoustic model learned by the acoustic model learning unit 15 is stored in the acoustic model storage unit 22 .

<Acoustic Recognition Processing of Embodiment 3>
FIG. 12 is a diagram exemplifying the acoustic recognition processing flow of the third embodiment. The acoustic recognition processing of the third embodiment is executed by the acoustic recognition device 3C during acoustic recognition. Steps S10 and S17-S18 in FIG. 12 are the same as in FIGS.

In FIG. 12, in step S21, the acoustic recognition device 3C uses the acoustic feature quantity conversion device 1C to convert and expand the acoustic feature quantities into acoustic feature quantities with a plurality of operating parameters. Specifically, the acoustic feature amount extension unit 19 converts the acoustic feature amount from the acoustic feature amount extracted by the acoustic feature amount extraction unit 18 to the acoustic feature amount at a plurality of operating speeds, for example, when the operating speed is the operating parameter. By transforming and expanding into quantities, an expanded acoustic feature quantity group is generated.

In step S23 subsequent to step S21, the acoustic recognition unit 16 performs identification using the extended acoustic features extended by the acoustic feature conversion device 1C and the acoustic model stored in the acoustic model storage unit 22. conduct. The acoustic recognition unit 16 identifies each of the acoustic features extended by the acoustic feature conversion device 1, and integrates the results to improve the accuracy of identification.

10, the acoustic model learning process of the third embodiment shown in FIG. 11, and the acoustic recognition process of the third embodiment shown in FIG. The execution order of step S18 and step S21 can be interchanged. At that time, for example, the separated acoustic signal output from the sound source separation unit 17 is input to the acoustic feature amount extension unit 19, and an extended acoustic signal group extended by the operating parameters is output, and the extended acoustic signal group is It is input to the acoustic feature amount extraction unit 18, and an extended acoustic feature amount group is output.

Further, although the acoustic feature quantity is expanded for each of the acoustic model learning process of the third embodiment shown in FIG. 11 and the acoustic recognition process of the third embodiment shown in FIG. It is not necessary to extend the acoustic features in both the acoustic recognition device 3C. At least one of the acoustic model learning device 2C and the acoustic recognition device 3C needs to extend the acoustic features. Just do it.

As described above, according to the third embodiment, when sufficient learning data for a task to be recognized cannot be prepared, the amount of data is padded by expanding the data using the operating parameters, and the same data expansion is performed at the time of identification. By performing, the identification accuracy is improved. It also reduces the time and money costs required to record datasets.

Compared with the first embodiment, the fourth embodiment discloses an acoustic feature value conversion device that refers to a plurality of operating parameters, and an acoustic model learning device and an acoustic recognition device using the acoustic feature value conversion device. Differences from the first embodiment will be mainly described below.

Based on the premise that there is a correlation between the acoustic signal emitted when the equipment is in operation and the operating parameter, in the first embodiment, a single operating parameter is referred to, whereas in the fourth embodiment, the acoustic By referring to a plurality of operating parameters by the feature conversion device, data standardization with higher compression efficiency, for example, becomes possible. As a result, according to the fourth embodiment, it is possible to further reduce the required storage capacity, reduce the weight of the machine learning model, and reduce the time cost required for learning.

<Structures of Acoustic Feature Transformation Apparatus, Acoustic Model Learning Apparatus, and Acoustic Recognition Apparatus of Embodiment 4>
FIG. 13 is a diagram exemplifying a functional configuration diagram of an acoustic feature value conversion device, an acoustic model learning device, and an acoustic recognition device according to the fourth embodiment. As shown in FIG. 13, the acoustic feature quantity conversion device 1D of the fourth embodiment includes a sound source separation unit 12, an acoustic feature quantity extraction unit 13, an acoustic feature quantity normalization unit 112, a sound source data storage unit 24, and a sound source model learning unit 113. , and a sound source model storage unit 25 .

Further, as shown in FIG. 13, the acoustic model learning device 2D of the fourth embodiment includes, in addition to each component of the acoustic feature quantity conversion device 1D, a normalized acoustic feature quantity storage unit 21, an acoustic model learning unit 15, and acoustic model storage unit 22 .

Further, as shown in FIG. 13, the acoustic recognition device 3D of Example 4 includes a acoustic recognition unit 16 in addition to each component of the acoustic feature quantity conversion device 1D.

In the fourth embodiment, two types of operating parameters acquired respectively from the operating parameter acquiring unit 110 and the operating parameter acquiring unit 111 are referred to. However, the number of operating parameters to be acquired and referenced is not limited to two, and at least two types of operating parameters may be referenced.

<Acoustic Feature Transformation Processing of Embodiment 4>
FIG. 14 is a diagram exemplifying the acoustic feature quantity conversion processing flow of the fourth embodiment. The acoustic feature quantity conversion process of the fourth embodiment is executed by the acoustic feature quantity conversion device 1D when normalizing the acoustic feature quantity. Step S10 in FIG. 14 is generally the same as in FIG. 2, except that there are multiple sources for obtaining the operating parameters.

In FIG. 14, in step S24 subsequent to step S10, the sound source separation unit 12 extracts the recognition target from the input acoustic signal containing noise, such as noise, reverberation, and sound from a sound source other than the recognition target task device. Only the acoustic signals emitted from the task device are extracted and separated, and the post-separation acoustic signals are calculated. At this time, the sound source separation unit 12 may refer to the operating parameters to separate the sound sources, for example, as shown in the second embodiment.

Subsequently, in step S25, the acoustic feature quantity extraction unit 13 calculates a digital acoustic feature quantity from the analog post-separation acoustic signal. At this time, the calculated acoustic feature amount may be, for example, a spectrogram, a mel feature amount, or a mel-cepstrum feature amount. Further, at this time, the acoustic feature amount extraction unit 13 may refer to the operating parameters to separate the sound sources, as shown in the second embodiment, for example. The acoustic feature quantity extracted by the acoustic feature quantity extracting unit 13, the operating parameter A acquired by the operating parameter acquiring unit 110, and the operating parameter B acquired by the operating parameter acquiring unit 111 are linked, and the sound source data is stored. Stored in unit 24 .

Subsequently, in step S26, the sound source model learning unit 113 acquires the acoustic feature amount extracted by the acoustic feature amount extraction unit 13 and the operating parameter A acquired by the operating parameter acquisition unit 110, which are stored in the sound source data storage unit 24. , and the operating parameter B acquired by the operating parameter acquiring unit 111, learning of the sound source model is performed.

Specifically, the sound source model learning unit 113 utilizes the fact that there is a correlation between the acoustic signal emitted when the device is in operation and the plurality of operating parameters, and uses the acoustic feature quantity and the plurality of operating parameters of the device to obtain a plurality of A sound source model is generated to normalize the acoustic features with the operating parameters. At this time, the correlation between the acoustic signal to be used and the operating parameter is, for example, a correlation obtained by statistical calculation or machine learning.

Here, for example, if the input acoustic signal is s, the input operating parameter A is xA, the input operating parameter B is xB, and the normalized acoustic signal is s', then the input operating parameter A and the input operating parameter B are each given a specific Acoustic feature amount normalization models FαA and αB for normalizing the input acoustic signal s into the normalized acoustic signal s′ when the operating parameters are αA and αB are represented by the following equation (4).

At this time, the specific operating parameters to be standardized are, for example, those specified by the operator, those automatically generated from the distribution of the operating parameters of the learning data, and the like. In addition, although two operating parameters are referred to in the acoustic feature amount normalized model, it is not necessary to refer to two operating parameters as long as at least two or more operating parameters are referred to. The acoustic feature amount normalized model generated in this manner is stored in the sound source model storage unit 25 .

<Acoustic model learning process of the fourth embodiment>
FIG. 15 is a diagram exemplifying the acoustic model learning processing flow of the fourth embodiment. Steps S10 and S24 to S25 in FIG. 15 are the same as in FIG.

In step S27 following step S25, the acoustic model learning device 2D normalizes the acoustic feature quantity using the acoustic feature quantity conversion device 1D. The acoustic feature quantity normalization unit 112 of the acoustic feature quantity conversion device 1D stores the operating parameter A acquired by the operating parameter acquiring unit 110, the operating parameter B acquired by the operating parameter acquiring unit 111, and the sound source model storage unit 25. Using the stored acoustic feature amount normalization model, the acoustic feature amount is normalized. The acoustic feature amount normalization unit 112 normalizes, for example, the acoustic feature amount at an arbitrary operating speed, pressure, and outside temperature into an acoustic feature amount at a specified operating speed, pressure, and outside temperature. The normalized acoustic feature amount is stored in the normalized acoustic feature amount storage unit 21 .

In step S15 following step S27, the acoustic model learning unit 15 learns the acoustic model using the normalized acoustic feature amount stored in the normalized acoustic feature amount storage unit 21 as learning data. Known methods such as Gaussian mixture models, support vector machines, and neural networks are used for learning. The acoustic model learned by the acoustic model learning unit 15 is stored in the acoustic model storage unit 22 .

<Sound Recognition Processing of Example 4>
FIG. 16 is a diagram exemplifying the acoustic recognition processing flow of the fourth embodiment. Steps S10 and S24 to S25 in FIG. 16 are the same as in FIGS.

In FIG. 16, in step S27 subsequent to step S25, the acoustic recognition device 3D normalizes the acoustic features using the acoustic feature conversion device 1D. Acoustic feature value conversion device 1D uses operating parameter A acquired by operating parameter acquiring unit 110, operating parameter B acquired by operating parameter acquiring unit 111, and sound source model storage unit 25 in acoustic feature value normalizing unit 112. The acoustic features are normalized using the acoustic feature quantity normalization model stored in the .

In step S16 following step S27, the acoustic recognition unit 16 performs identification using the normalized acoustic features normalized by the acoustic feature conversion device 1D and the acoustic model stored in the acoustic model storage unit 22. .

14, the acoustic model learning process of the fourth embodiment shown in FIG. 15, and the acoustic recognition process of the fourth embodiment shown in FIG. The execution order of step S25 and step S26 or step S27 can be interchanged.

At this time, for example, the separated acoustic signal output from the sound source separation unit 12 is stored in the sound source data storage unit 24 and is also input to the acoustic feature amount standardization unit 112 . The sound source model is trained on the acoustic signal, and the normalized acoustic signal normalized by the operating parameters is output from the acoustic feature quantity normalization unit 112 . The normalized acoustic signal is input to the acoustic feature amount extraction unit 13, and the normalized acoustic feature amount is output.

As described above, according to the fourth embodiment, even if sufficient learning data regarding the recognition target task cannot be prepared, the effective amount of data can be expanded by standardizing the data using a plurality of operating parameters. In addition, by performing similar data standardization at the time of identification, identification accuracy is improved. It also reduces the time and money costs required to record datasets, reduces the required storage capacity, makes machine learning models lighter, and reduces the time cost of training.

<Computer Realizing Acoustic Feature Conversion Device, Acoustic Model Learning Device, and Acoustic Recognition Device>
FIG. 17 is a diagram illustrating a computer that implements the acoustic feature value conversion device, the acoustic model learning device, and the acoustic recognition device. Acoustic feature quantity conversion devices 1 to 1D, acoustic model learning devices 2 to 2D, and acoustic recognition devices 3 to 3D of Examples 1 to 4 include, for example, a central processing unit (CPU) 5300, a main memory ( 5400 (RAM: Random Access Memory) 5400 or the like, and each program for acoustic feature conversion processing, acoustic model learning processing, and acoustic recognition processing is loaded into the device.

The computer 5000 executes each process under the control of a central processing unit 5300 cooperating with a main storage unit 5400, for example.

Computer 5000 includes central processing unit 5300, main memory 5400, input device 5600 (e.g. keyboard, mouse, touch panel, etc.), and output device 5700 (e.g. video graphics card connected to an external display monitor) through memory controller 5500. interconnected. In the computer 5000, each program for realizing the acoustic feature value conversion devices 1 to 1D, the acoustic model learning devices 2 to 2D, and the acoustic recognition devices 3 to 3D is transferred via an I/O (Input/Output) controller 5200 to an SSD. or an external storage device 5800 such as an HDD, and executed by cooperation of the central processing unit 5300 and the main storage device 5400, the acoustic feature value conversion devices 1 to 1D, the acoustic model learning devices 2 to 2D, and acoustic recognition devices 3 to 3D are realized.

Alternatively, each program for realizing the acoustic feature value conversion devices 1 to 1D, the acoustic model learning devices 2 to 2D, and the acoustic recognition devices 3 to 3D is obtained from an external computer through communication via the network interface 5100. good too. Each program for realizing the acoustic feature value conversion devices 1 to 1D, the acoustic model learning devices 2 to 2D, and the acoustic recognition devices 3 to 3D is stored in a distributable storage medium and read from the storage medium by a reading device. may be issued and retrieved.

In addition, data calculated during input and output to each processing unit of the acoustic feature value conversion devices 1 to 1D, acoustic model learning devices 2 to 2D, and acoustic recognition devices 3 to 3D, and processing are stored in the main memory, for example. Stored in device 5400 and used for other processes as needed. Also, at least part of the processing units of the acoustic feature quantity conversion devices 1 to 1D, the acoustic model learning devices 2 to 2D, and the acoustic recognition devices 3 to 3D may be configured by hardware such as integrated circuits.

Each storage unit provided in the acoustic model learning devices 2 to 2D is configured by a main storage device 5400 such as a RAM, an auxiliary storage device such as a hard disk, an optical disk, or a flash memory, or middleware such as a database. can do. Each storage unit included in the acoustic model learning devices 2 to 2D may be theoretically divided, and may be physically stored in one storage device.

The present invention is not limited to the above-described embodiments, and includes various modifications. For example, the above-described embodiments have been described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the described configurations. In addition, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Moreover, it is possible to add, delete, replace, integrate, or distribute a part of the configuration of each embodiment. Further, each configuration and each process shown in the embodiment may be appropriately distributed or integrated based on processing efficiency or implementation efficiency.

1, 1B, 1C, 1D: acoustic feature quantity conversion device, 2, 2B, 2C, 3D: acoustic model learning device, 3, 3B, 3C, 3D: acoustic recognition device, 10: acoustic signal acquisition unit, 11, 110, 111: operating parameter acquisition unit, 12, 17: sound source separation unit, 13, 18: acoustic feature amount extraction unit, 14, 112: acoustic feature amount standardization unit, 15: acoustic model learning unit, 16: acoustic recognition unit, 19 : Acoustic feature amount extension unit 113: Sound source model learning unit 21: Normalized acoustic feature amount storage unit 22: Acoustic model storage unit 23: Extended acoustic feature amount storage unit 24: Sound source data storage unit 25 : tone generator model storage unit, 5000: computer

Claims (8)

A sound source separation unit that separates an acoustic signal whose sound source is a target device from the acoustic signal acquired by the acoustic signal acquisition unit;
an acoustic feature extraction unit that extracts an acoustic feature from the acoustic signal separated by the sound source separation unit;
an acoustic feature normalization unit or an acoustic feature extension unit,
The acoustic feature amount normalization unit
The acoustic feature is obtained based on the operating parameter of the device acquired by the operating parameter acquisition unit using an acoustic feature amount normalization model based on the correlation between the acoustic signal having the device as the sound source and the operating parameter of the device. generating a normalized acoustic feature quantity, which is an acoustic feature quantity at a specific operating parameter of the device, from the acoustic feature quantity extracted by the quantity extraction unit;
The acoustic feature extension unit includes:
Based on the specific operating parameters of the device acquired by the operating parameter acquiring unit, the above 1. An acoustic feature quantity conversion apparatus, characterized by generating an expanded acoustic feature quantity, which is an acoustic feature quantity for an arbitrary operating parameter of said equipment, from the acoustic feature quantity extracted by an acoustic feature quantity extraction unit. 2. The acoustic feature quantity conversion apparatus according to claim 1, wherein the sound source separation unit separates an acoustic signal having the device as a sound source based on the operating parameter. 2. The acoustic feature quantity conversion apparatus according to claim 1, wherein the acoustic feature quantity extraction unit extracts the acoustic feature quantity based on the operating parameter. 2. The acoustic feature quantity conversion apparatus according to claim 1, wherein the operating parameter is a physical quantity indicating an operating state of the device or a control signal for controlling the operating state of the device. 2. The acoustic feature quantity conversion apparatus according to claim 1, wherein the operating parameters of the device include a plurality of types of operating parameters of the device acquired by the operating parameter acquiring unit. At least one of the sound source separation unit, the acoustic feature quantity extraction unit, the acoustic feature quantity normalization unit, and the acoustic feature quantity extension unit is configured to combine an acoustic signal with the device as a sound source and an operating parameter of the device. 2. The acoustic feature quantity conversion apparatus according to claim 1, wherein a neural network configured based on the correlation of is used. An acoustic feature conversion method executed by an acoustic feature conversion device,
A sound source separation step of separating an acoustic signal whose sound source is the target device from the acoustic signal acquired by the acoustic signal acquisition unit;
an acoustic feature quantity extraction step of extracting an acoustic feature quantity from the acoustic signal separated by the sound source separation step;
either one of the acoustic feature amount normalization process and the acoustic feature amount expansion process,
The acoustic feature amount normalization step includes:
The acoustic feature is obtained based on the operating parameter of the device acquired by the operating parameter acquisition unit using an acoustic feature amount normalization model based on the correlation between the acoustic signal having the device as the sound source and the operating parameter of the device. generating a normalized acoustic feature quantity, which is an acoustic feature quantity at a specific operating parameter of the device, from the acoustic feature quantity extracted by the quantity extraction step;
The acoustic feature amount expansion step includes:
Based on the specific operating parameters of the device acquired by the operating parameter acquiring unit, the acoustic An acoustic feature amount conversion method, comprising: generating, from the acoustic feature amount extracted by the feature amount extraction step, an expanded acoustic feature amount, which is an acoustic feature amount at an arbitrary operating parameter of the device. An acoustic feature conversion program for causing a computer to function as the acoustic feature conversion device according to any one of claims 1 to 6.

Images