JP7557874B2 - Trained network including multiple stages of feedforward neural networks, and method for generating said trained network and information processing system - Google Patents

Description

The present invention relates to a network including multiple stages of feedforward neural networks and a technology for generating a trained model of the network.

There is a method (e.g., Non-Patent Document 5) in which the weights between the input layer and hidden layer of a feed-forward neural network (FFNN) with one hidden layer are determined by random numbers regardless of the training data, and only the weights between the hidden layer and the output layer are obtained from the training data, thereby achieving learning with a lighter amount of calculation compared to the method of learning all the weights. There are various technologies that attempt to improve the inference accuracy of a network by stacking multiple FFNNs trained in this way. For example, there is a method (e.g., Non-Patent Document 1) in which residual features are learned by associating only the input to the lowest-stage feed-forward neural network that performs inference using only observed features, and a method (e.g., Non-Patent Document 2) in which residual features are learned by associating only the output from the lowest-stage feed-forward neural network. However, these methods have the problem that they cannot accurately learn the tendency of inference errors in the lower-stage feed-forward neural network, and as a result, a significant improvement in inference accuracy cannot be expected. There is also a known technique in which all other feedforward neural networks stacked on top of the lowest-level feedforward neural network are trained with exactly the same input (e.g., Non-Patent Document 3), but this technique is known to have poor learning efficiency.

There are many variations in the configuration of networks that use multiple stages of FFNNs in which some weights are determined by random numbers, and there is room for further ingenuity to achieve higher inference accuracy and generalization performance compared to conventional technology.

SEN ZHANG , ZHENG LIU, XUEJIAO HUANG, AND WENDONG XIAO, "A Modified Residual Extreme Learning Machine Algorithm and Its Application," DOI 10.1109/ACCESS.2018.2876360, IEEE Access, Vol.6, 2018. Alexander Waibel, Tashiyuki Hanazawa, Geoffrey Hinton, Kiyohito Shikano, Kevin J. Lang, "Phoneme Recognition Using Time-Delay Neural Networks," IEEE Transactions on Acoustics, Speech, and Signal Processing, Volume 37, No.3, pp.328- 339 March 1989. Yinghao Chen Xiaoliang Xie Tianle Zhang Jiaxian Bai Muzhou Hou, "QA deep residual compensation extreme learning machine and applications,"＜https://doi.org/10.1002/for.2663＞, Journal of FORCASTING, vol.39, issue 6, pp.986-999, September, 2020. Guang-Bin Huang, "Extreme Learning Machine for Regression and Multiclass Classification," IEEE Trans. SYSTEMS, MAN, AND CYBERNETICS-PART B: CYBERNETICS, VOL. 42, NO. 2, pp.513-529, APRIL 2012. Pao Y-H, Park G-H, Sobajic D., "Learning and generalization characteristics of the random vector functional-link net," Neurocomputing 1994, 6:163-180.

Therefore, one aspect of the object of the present invention is to provide a technology for achieving higher inference accuracy in a network constructed by stacking multiple FFNNs.

The trained network according to the present invention includes (A) a first network including a first feedforward neural network that receives as input an observed feature at a time specified based on a predetermined timing pattern, and (B) a second network of (L-1) stages (L is an integer of 2 or more) including a second feedforward neural network that receives as input a concatenated feature that is a concatenation of an output feature and an observed feature of a network one stage lower at a time specified based on a predetermined timing pattern, and each of the second networks includes an adder that calculates an output feature by adding the output of the second feedforward neural network to the output feature of the network one stage lower, and is a trained network that can be executed by a computer. The predetermined timing patterns include first to m-th (1 to m is an integer of L or less), and the (x mod m)-th timing pattern of the first to m-th timing patterns is applied to each of the first and second networks based on the number of stages x (1≦x≦L) from the bottom stage of the network.

The method for generating a trained network as described above includes the steps of: (A) identifying multiple candidates for the configuration of the network to be constructed based on the duration of the observed event that is the source of the observed feature and the receptive domain size, which is the difference between the oldest and newest time of the observed feature used in the entire network to be constructed; (B) generating a trained network for each of the multiple candidates; and (C) selecting the trained network to be used based on the inference accuracy of the generated trained network.

On the one hand, a network constructed by stacking multiple FFNNs can achieve higher inference accuracy.

FIG. 1 is a diagram showing an example of the configuration of a base network according to an embodiment of the present invention. FIG. 2 is a diagram showing an example of the configuration of a residual network according to an embodiment of the present invention. FIG. 3 is a diagram illustrating an example of the overall configuration of a network according to the first embodiment. FIG. 4 is a diagram showing an example of an on/off pattern of a switch according to an embodiment of the present invention. FIG. 5 is a diagram showing a schematic diagram of input and output timing of the network at each stage according to the embodiment of the present invention. FIG. 6 is a diagram showing another example of an on/off pattern in the switch according to the embodiment of the present invention. FIG. 7 is a functional block diagram of an information processing device that executes a process for generating a quasi-optimal trained network in an embodiment of the present invention. FIG. 8 is a diagram for explaining the concept of the process of generating a trained network according to the first embodiment. FIG. 9 is a diagram for explaining the concept of the process of generating a trained network according to the first embodiment. FIG. 10 is a diagram for explaining the concept of the process of generating a trained network according to the first embodiment. FIG. 11 is a diagram illustrating a process flow of a trained network generation process according to the first embodiment. FIG. 12 is a diagram showing a process flow of the first model construction evaluation process. FIG. 13 is a diagram showing a process flow of the first model construction evaluation process. FIG. 14 is a diagram showing a process flow of the second model construction evaluation process. FIG. 15 is a diagram showing a process flow of the second model construction evaluation process. FIG. 16 illustrates an example of an overall configuration of a network according to the second embodiment. FIG. 17 is a diagram illustrating another example of the overall configuration of the network according to the second embodiment. In FIG. FIG. 18 is a diagram for explaining the concept of the process of generating a trained network according to the second embodiment. FIG. 19 is a diagram illustrating a process flow of a trained network generation process according to the second embodiment. FIG. 20 is a diagram illustrating a process flow of a trained network generation process according to the second embodiment. FIG. 21 is a diagram showing the relationship between the number of stages in the network and the size of the receptive area in the embodiment. FIG. 22 is a diagram showing the relationship between the number of model parameters that can be adjusted by learning and the F-measure. FIG. 23 is a diagram showing the relationship between the F-measure and the number of model parameters that can be adjusted by learning when the input features to a residual network are only the input features of the network immediately below. FIG. 24 shows the relationship between the F-measure and the number of model parameters that can be adjusted by learning when a pattern in which all 103 consecutive features from the current time t to time (t-102) are input to an FFNN is applied to a total of 10-stage networks. FIG. 25 is a diagram showing the relationship between the F-measure for a convolutional neural network and the number of parameters that can be adjusted by learning. FIG. 26 shows the relationship between the number of stages and the F-measure when the number of nodes in the hidden layer is the same for all 12-stage networks and there are 10 types in increments of 1000 from 1000 to 10,000. FIG. 27 is a functional block diagram of an information processing device.

[First embodiment]
[Network Configuration Example According to the Present Embodiment]
In this embodiment, an observation feature, which is obtained by performing a short-time Fourier transform or a wavelet transform on a one-dimensional observation signal such as sound or vibration and is a parameter related to a frequency component distribution near a feature extraction time, is used as an input. Specific examples of the observation feature include, but are not limited to, a Mel filter bank feature, a Mel spectrogram feature, and a Mel cepstrum feature. Meanwhile, the inference result by the network is, for example, the type of an observation event that is the source of the observation signal, but is not limited to this.

In this embodiment, an entire network is constructed by stacking one or more residual networks as shown in Fig. 2 on a base network as shown in Fig. 1. As shown in Fig. 1, the base network has an FFNN 100 with one hidden layer, a memory mechanism 120 to which an observed feature f _t is input and which connects a predetermined number of delay elements Z ⁻¹ (M ₁ in the figure) in series, and a switch group 110 for selecting an observed feature to be input to the input layer of the FFNN 100 from the observed features f _t-M1 to f _t−1 and the observed feature f _t stored in the memory mechanism 120.

Only observed features are input to the base network, and inference is made from the time series of observed features, so that the memory mechanism 120 stores observed features at times in the past for the number of delay elements Z ⁻¹ . Furthermore, a switch included in a switch group 110 is connected to each path of the memory mechanism 120, and by turning on or off each switch of the switch group 110, observed features at any time (i.e., any timing) are selected and input to the FFNN 100. The calculations in the FFNN 100 will be described in detail later. The internal parameters of the FFNN 100 are learned so that the output feature y _1,t of the FFNN 100 becomes the target feature of the training data.

The residual network shown in Fig. 2 includes an FFNN 200 with one hidden layer, a concatenation unit 230 that concatenates the observed feature f _t and the previous stage output feature y _l-1,t output by the previous stage (lower stage) network to generate a concatenated feature c _t , a memory mechanism 220 to which the concatenated feature c _t is input and in which a predetermined number (M _l in the figure) of delay elements Z ^-1 are connected in series, a group of switches 210 for selecting concatenated features to be input to the input layer of the FFNN 200 from the concatenated features c _t-Ml to c _t-1 and the concatenated feature c _t stored in the memory mechanism 220, and an adder 240 that adds the output of the FFNN 200 and the previous stage output feature y _l-1,t to generate an output feature y _l,t . l represents the number of stages of this residual network when the base network is the first stage.

As mentioned above, the input to the FFNN200 of the residual network is a concatenated feature that concatenates the same observation feature as the input to the base network with the previous stage output feature (the result of inferring the target feature), which is the output of the previous stage network. Using such concatenated features as the input to the FFNN200, the FFNN200 infers the residual feature that is the target feature minus the previous stage output feature, which is the result of inferring the target by the previous stage network including the base network, so that the FFNN200 of the residual network can accurately learn the characteristics and tendencies of inference errors mainly of the previous stage network. This is different from Non-Patent Documents 1 and 2.

Furthermore, the learning of the FFNN200 of the residual network differs from the learning of the base network in the following respects. That is, the FFNN200 of the residual network does not learn to directly infer the target feature, but learns to infer the above-mentioned residual feature, which is also one of the features of this embodiment. Then, the result of the FFNN200 of the residual network inferring the residual feature is added to the output feature that is the result of the previous stage network inferring the target, so that the output feature that is the result of the residual network inferring the target feature is generated. If the result of the base network or the previous stage network including the base network inferring the target feature is completely consistent with the target feature, the residual feature learned by the FFNN200 of the residual network is a zero vector. This is realized by learning that all the weights of all connections between the hidden layer and the output layer in the FFNN200 of the residual network are all 0. Therefore, it is guaranteed that the inference accuracy of the output of this residual network is improved to the same or greater extent than the output of the base network or the previous stage network including the base network.

Furthermore, in this embodiment, both the base network and the residual network are provided with a storage mechanism in which a predetermined number of delay elements are connected in series, and a group of switches, and the feature to be input to the FFNN can be arbitrarily selected by turning on or off the switches connected to each path. The on/off pattern (i.e., the timing pattern that determines whether or not the feature is adopted) can be set to be different for each of the base network and one or more residual networks. This allows the input to the FFNN 100 of the base network and the input to the FFNN 200 of each residual network to be
This allows the input features to be varied each time the residual network is built up, improving learning efficiency compared to Non-Patent Document 3.

Because of these characteristics, it is possible to achieve higher inference accuracy than conventional techniques such as those described in Non-Patent Documents 1 to 3.

Figure 3 shows an example of a base network and two-stage residual networks stacked together. Note that in the second-stage residual network and the third-stage residual network, the previous-stage output feature, which is the output from the previous-stage network, is connected at the connection section to the observed feature at the timing when the previous-stage output feature is output, so that the higher the stage, the later the observed feature is input.

Next, Figure 4 shows a schematic example of the on/off patterns of the switch group. Figure 4 shows four stages of on/off patterns (i.e., timing patterns) that belong to a power-of-two pattern. In each on/off pattern, the rightmost box indicates the on or off state of the switch that controls whether or not to output the feature at the current time t, and the leftmost box indicates the on or off state of the switch that controls whether or not to output the feature for time (t-55). The hatched boxes indicate the on state, and the blank boxes indicate the off state.

In this example, two features are input to the FFNN for all on-off patterns. One is the feature for the current time t, and the other is the feature for the time (t-Δ) Δ= ^2l-1 hours in the past from the current time. As mentioned above, l represents the number of stages of the network when l=1 is the base network, and l≧2 represents the residual network.

When four stages of networks are stacked with this on-off pattern, the output features obtained by the fourth stage residual network inferring the target features are based on 16 observed features from the current time t to time (t-15). This is shown diagrammatically in Figure 5. In Figure 5, there is one FFNN for each stage, and the blocks shown correspond to the number of inferences from time t to time (t-15). The inputs to the base network are the observed features from two consecutive times (for example, (t-15) and (t-14)), and the output of the FFNN 100 is generated at a later time (for example, (t-14)). In the second-stage residual network, a concatenated feature obtained by concatenating the output feature (the feature obtained by inferring the target) and the observed feature of the base network is input, but concatenated features at two times with an interval of one frame (for example, (t-14) and (t-12)) are used, and the output of the FFNN 200 (the feature obtained by inferring the residual feature) is generated at a later time (for example, (t-12)). Since the output of the FFNN 200 is a residual, it is added to the output feature of the base network to become the input of the third stage (the feature obtained by inferring the target). In the third-stage residual network, a concatenated feature obtained by concatenating the output feature and the observed feature of the second-stage residual network is input, but concatenated features at two times with an interval of three frames (for example, (t-12) and (t-8)) are used, and the output of the FFNN 200 (the feature obtained by inferring the residual feature) is generated at a later time (for example, (t-8)). The output of this FFNN200 is a residual, so it is added to the output features of the second stage to become the input of the fourth stage (features that infer the target). In the fourth stage residual network, the input is a concatenated feature that concatenates the output features of the third stage residual network and the observed features, but the concatenated features at two times that are seven frames apart (for example, (t-8) and t) are used, and the output of FFNN200 (features that infer the residual features) is generated at a later time (for example, t). The output of this FFNN200 is a residual, so it is added to the output features of the third stage to become the final output.

In Figure 5, blocks arranged vertically (feature blocks and FFNN inference blocks) represent features or inferences in the FFNN at the same time.

In this way, the time difference between the most recent time and the most recent time used to infer one output feature in a network of a certain stage is called the receptive field size (RF). If the on-off pattern (i.e., timing pattern) for the lth stage is represented by a vector p _l , the time from the time corresponding to the right end position where the vector p _l is on to the time corresponding to the left end position represents the receptive field size, and the time difference Δ specified by this on-off pattern is represented by Δ(p _l ). For example, in the case of the power-of-two pattern in FIG. 4, it is as follows.
Δ ₂ (p _l )=2 ^l−1

And for power-of-two patterns, the receptive field size of the multi-stage network as a function of the vector p _l is expressed as follows:
RF ₂ (p ₁ ,..., p _L )=1+Σ _l=1 ^L Δ ₂ (p _l )
Here, L represents the number of stages of the network to be stacked.

In the case of FIG. 4, L=2 and Δ ₂ (p _l )=2 ^l−1 , so we get the following:
RF ₂ (p ₁ ,..., p ₄ )=1+Σ _l=1 ⁴ 2 ^l-1 =16
This corresponds to the values stated above.

In addition to the method of monotonically expanding the time difference by a power, such as Δ ₂ (p _l ) = 2 ^l-1 , each time the network is stacked, a method of repeating from the initial pattern again every time a certain number of stages are stacked may be adopted. For example, when repeating from the initial pattern every m stages, if the pattern is a power of 2, the time difference for the lth stage will be Δ ₂ ^m = 2 ^{(l-1) mod m} . When the pattern in Figure 4 is repeated twice to stack L = 8 stages, the receptive field size will be as follows:
RF ₂ ⁴ (p ₁ ...p ₈ )=1+Σ _l=1 ⁸ 2 ^{(l-1)mod m} =31
On the other hand, if the blocks are monotonically stacked in L=8 stages in a power-of-two pattern, the receptive field size is as follows:
RF ₂ (p ₁ ...p ₈ )=1+Σ _l=1 ⁸ 2 ^(l-1) =256
In this way, even with the same number of stages, changing the on/off pattern setting results in a large change in the receptive area size.

It is also possible to adopt a power-of-three pattern instead of a power-of-two pattern. This case is shown diagrammatically in Fig. 6. As can be seen from Fig. 6, three feature quantities are input to the FFNN in each stage. The time intervals between the three feature quantities are equal. The time difference in the lth stage in the power-of-three pattern is expressed as Δ ₃ (p _l ) = 2 × 3 ^{l - 1} .

Generally, the time difference in the l-th stage in a power pattern of base b is Δ _b (p _l ) = (b-1) x b ^l-1 , and the receptive field size when this is stacked up in L stages is expressed as follows.
RF _b (p ₁ ,..., p _L )=1+Σ _l=1 ^L Δ _b (p _l )

When repeating from the initial pattern every m stages, the time difference in the lth stage is Δ _b ^m (p _l ) = (b-1) × b ^{(l-1) mod m} , and the receptive field size of the model stacked in L stages is expressed as follows:
RF _b ^m (p ₁ ,..., p _L )=1+Σ _l=1 ^L Δ _b ^m (p _l )

Even if the pattern is not a power pattern, the time difference from the time corresponding to the rightmost position where the on-off pattern in the lth stage turns on to the time corresponding to the leftmost position can be expressed as Δ(p _l ) and the receptive area size of the model stacked up in L stages can be expressed as follows, so that the same treatment can be used.
RF (p ₁ ,..., p _L )=1+Σ _l=1 ^L Δ(p _l )
Even if the pattern is not a power pattern, the initial pattern may be repeated every m stages. In this case, the acceptance range of the model stacked in L stages is expressed as follows:
RF ^m (p ₁ ,..., p _L )=1+Σ _l=1 ^L Δ ^m (p _l )

Next, the specific calculation of the output feature for the observed feature input to the base network will be described. The observed feature at the current time t extracted from the observed signal such as an acoustic signal is input to the base network. Hereinafter, the number of dimensions of the observed feature is set to D, and the observed feature at time t is expressed as a vector f _t = [f _{1, t} , f _{2, t} , ..., f _{D, t} ] ^T ∈ R ^{D × 1} , where T represents transpose.

FIG. 1 shows a case where the observed feature is a two-dimensional vector. In FIG. 1, a delay element represented by z ^-1 outputs f _t-1 for an input of f _t . By connecting M1 of these delay elements, it becomes possible to store (M1+1) observed feature values from the current time t to the time (t-M1). A group of switches 110 connected to the delay elements decides which observed feature value at which time is to be input to the input layer of the FFNN 100. For example, in the case of FIG. 1, a vector combining f _t , f _t-2 , and f _(t-M1) is input to the FFNN 100.

The number of nodes P1 in the input layer of the FFNN 100 is the number of elements of the input observation feature _x1,t ∈R ^P1×1 , and is expressed as P1 = D × (the number of switches that are turned on). In FIG. 1, _x1,t = [f _t ^T f _t-2 ^T f _t-M1 ^T ] = [ _f1,t , _{f2, t, f1,t-2 , f2,t -2, f1,t-M1} , _f2,t-M1 ] ^T , and the number of nodes in the input layer is P1 = 6. Here, the subscript "1" in M1, P1, x1 _,t, etc. indicates that the base network is a network in which a base network and one or more residual networks are stacked as shown in FIG. 3, and that the base network is the first-stage network.

Furthermore, if the number of nodes in the hidden layer is Q1, the weight matrix of all connections from the input layer to the hidden layer is _W1∈RQ1 ^×P1 , the bias vector in the hidden layer is _b1∈RQ1 ^×1 , and the activation function that performs calculations for each element of the vector is h(.), the output vector _z1, ^t∈RQ1×1 of the hidden layer is expressed as follows.
z _1,t =h(W ₁・x _1,t +b ₁ )
The activation function may be, but is not limited to, ReLU (Rectified Linear Unit). In the example of FIG. 1, the number of nodes in the hidden layer is Q1=9.

Next, if the number of nodes in the output layer is R and the weight matrix of all connections from the hidden layer to the output layer is β ₁ εR ^Q1×R , the output vector y _1,t εR ^R×1 of the output layer is obtained by the following equation.
y _1,t ^T =z _1,t ^T・β ₁
This y1 _,t becomes the output feature of the base network. The number of nodes R in the output layer is set to the same value in the base network and all residual networks. In the example of Figure 1, R = 3.

Next, the calculation of the output feature quantity for the input feature quantity to the residual network will be explained. The base network is the first-stage network, and the residual network immediately above it is called the second-stage residual network. The l-th stage residual network counting from the base network is called the l-th stage residual network (where l≧2). The l-th stage residual network receives a vector c _t =[f t T y _{l-1,t T} ] ^T ∈R ^(D+R)×1 , which is a concatenation of the vector f _t ∈R ^D×1 of the observed feature quantity at time _t ^and the vector y _l-1 , ^t ∈R ^R×1 of the output feature quantity of the network immediately below. Note that the subscript l of c _t is omitted for simplicity.

Each element of the vector of the connected feature is expressed as c _t = [c _{1, t} . . . c _{D + R, t} ]. In the example of FIG. 2, the number of dimensions of the observed feature vector is D=2, and the number of dimensions of the vector of the output feature directly below is R=3. As in the base network, the connected feature is input to M1 connected delay elements. The number of delay elements may be set to any number independent of the number of other networks. The switch group 210 connected to the delay elements determines at what time the connected feature is input to the input layer of the FFNN 200. The on/off pattern (i.e., timing pattern) of the switch group 210 may also be set to any number independent of the on/off patterns of other networks.

In the case of Fig. 2, a vector combining _ct , ct _-2 , and ct _-Ml is input to the FFNN200. The number of nodes in the input layer Pl is the number of elements of the input feature _xl,t ∈R ^Pl×1 , and is expressed as Pl = (D + R) × (number of switches turned on). In Fig. 2, _xl,t = [ _ct ^Tct _-2 ^Tct _-Ml ^T ] ^T = [c1 _,t ,... _c5,t , _c1,t-2 ,... _c5,t-2 , _c1,t-Ml ,... _c5,t-Ml ] ^T , and the number of nodes in the input layer is Pl = 15.

If the number of nodes in the hidden layer of the l-th stage residual network is Ql, the weight matrix of all connections from the input layer to the hidden layer is _Wl ∈RQl ^×Pl , the bias vector in the hidden layer is _bl ^∈RQl×1 , and the activation function that performs calculations for each element of the vector is h(·), the output vector _zl,t ∈RQl ^×1 of the hidden layer is calculated using the following formula.
z _l,t =h(W _l・x _l,t + b _l )
In the example of Fig. 2, the number of nodes in the hidden layer is Ql = 9. Next, since the number of nodes in the output layer is R, if the weight matrix of all connections from the hidden layer to the output layer is _βl ∈R ^{Ql × R} , the output vector _pl,t ∈R ^{R × 1} of the output layer can be calculated by the following formula.
p _l,t ^T =z _l,t ^T・β _l

The output feature vector y _l,t of the l-th stage residual network is obtained by y _l,t = y _l-1,t + p _l,t . In other words, it is calculated by adding the output vector p _l,t of the output layer and the output vector y _l-1,t of the network immediately below.

Next, a calculation procedure for finding parameters {W _l , b _l , β _l |l=1, . . . L} of a network constructed by stacking (L-1) stages of residual networks on a base network will be described.

The parameters between the input layer and the hidden layer {W _l , b _l |l = 1, ... L} are set as follows: Here, the number of nodes in the input layer is Pl, the number of nodes in the hidden layer is Ql, and the weight matrix of all connections from the input layer to the hidden layer is expressed as W _l = [w _{l, 1} , ... w _{l, Ql} ] ^T ∈ R ^{Ql × Pl} .

余りがある場合（Ｑｌ mod Ｐｌ≧１）は、以下のような余った行ベクトルを同様に正規直交化する。

バイアスベクトルｂ_ｌ∈Ｒ^Ｑｌ×１は、各要素に正規分布Ｎ（０，１）に従う乱数を代入し、ｂ_ｌ ^Ｔｂ_ｌ＝１（１≦ｌ≦Ｌ）となるようにノルムの正規化を行う。以上のように、入力層－隠れ層間のパラメータ｛Ｗ_ｌ，ｂ_ｌ｜ｌ＝１，．．．Ｌ｝は学習データとは全く無関係に設定される。 First, random numbers following normal distribution N(0,1) are substituted into each element of matrix _Wl . If Ql≦Pl, orthonormalization is performed using the Gram-Schmidt orthogonal method or the like so that wl _, ^jTwl _,k =δj _,k (1≦j,k≦Ql). Here, δj _,k represent Kronecker deltas, which are 1 when j=k and 0 otherwise. If Ql>Pl, the row vector _wl,j of matrix _Wl is divided into blocks { _wl,j |(m-1)Pl+1≦j≦mPl} for each Pl, and orthonormalization is performed for each block in the same manner as above. Here, m is a value as follows:

If there is a remainder (Ql mod Pl ≧1), the following remaining row vector is similarly orthonormalized.

For the bias vector b _l ∈R ^{Q l × 1} , random numbers following a normal distribution N (0, 1) are substituted into each element, and norm normalization is performed so that b _l ^T b _l = 1 (1 ≦ l ≦ L). As described above, the parameters between the input layer and hidden layer {W _l , b _l | l = 1, ... L} are set completely independent of the training data.

The parameters between the hidden layer and the output layer {β _l |l=1,...L} are calculated as follows, for example, using the method in Non-Patent Document 4. In the case of the base network, the output vector y 1 _,t ∈R ^R×1 of the output layer for the input vector z _1, t ∈R ^Q1×1 of the hidden layer is calculated as y _1,t ^T =z _1,t ^T ·β ₁ .

In the case of a residual network, the output vector p _l,t ∈R ^{R×1 of the output layer for the input vector z l,t ∈R Ql ×1} _of the hidden layer is calculated by p _l,t ^T =z _l,t ^T ·β _l . The only difference between the base network and the residual network is the variable name of the output vector, but the method of calculating the parameter β _l is exactly the same, so the following explanation will be given using the residual network as an example. Also, the subscript l will be omitted for simplicity.

ここで定数Ｃは正則化項と残差項のバランスを決めるパラメータである。この最適解は、以下の式で得られる。

Let _rt ∈ ^{R R×1} be the target vector, which is a vector of target features for the output vector p _t . Let the residual vector e _t ∈ ^{R R×1} be defined as e _t ^T = _rt ^T - _{p t} ^T = _rt ^T - z _t ^T · β. Let there be N input vectors z _t and target vectors _rt for learning in the range of 1 ≦ t ≦ N. If R = [r ₁ , ... r _N ] ^T , Z = [z ₁ , ... z _N ] ^T , and E = [e ₁ , ... e _N ] ^T , the above formula is expressed as E = R - Z · β. Under the condition that this relationship is satisfied, the parameter β that minimizes the evaluation function expressed as the sum of the regularization term and the residual term to prevent overlearning is obtained as the optimal solution.

Here, the constant C is a parameter that determines the balance between the regularization term and the residual term. The optimal solution is given by the following equation:

Deep learning optimizes an extremely large number of parameters, including all model parameters and hyperparameters that control the behavior of machine learning algorithms, through repeated learning and verification, which requires expensive computing resources such as a GPU (Graphics Processing Unit) and huge amounts of learning time and power consumption. In contrast, the network according to this embodiment sets the parameters between the input layer and hidden layer based on random numbers independent of the learning data, making it possible to easily find the optimal parameters between the hidden layer and output layer as the solution to a linear equation, enabling high-speed learning. This makes online learning and real-time learning possible on edge devices with low computing power.

[Suboptimal network configuration]
When the network according to the present embodiment is used for detecting or recognizing an observed event, that is, a sound event, it is desirable to match the receptive field size of the sound event to the average duration of the sound event to be detected or recognized. This is because, if the receptive field size is significantly longer than the average duration of the sound event, the observed features of background noise and the like are also learned, which deteriorates the accuracy of the detection or recognition. On the other hand, if the receptive field size is larger than the average duration of the sound event, the observed features of the sound event cannot be fully utilized, which deteriorates the accuracy of the detection or recognition.

The receptive area size of the network according to this embodiment can be calculated by simple analysis. Therefore, in this embodiment, the on/off pattern of the switches and the number of stacked network stages can be determined based on the average duration of the observed events to be detected or recognized, rather than arbitrarily repeating trial and error. In particular, it becomes possible to automatically select a quasi-optimal network configuration, including not only power patterns of base b, but also network configurations with general on/off patterns that are not restricted by the power law. The configuration of the information processing system for this purpose will be described below.

Figure 7 shows an example of the functional configuration of an information processing device for generating a trained network having a suboptimal network configuration. The information processing device 300 has a candidate extraction unit 310, an RF table storage unit 320, a training data storage unit 330, a training processing unit 340, a model construction unit 350, and a recognition accuracy calculation unit 360.

The learning data storage unit 330 stores pairs of observed features and target features at each time. For example, the duration of an observed event is measured by measuring the time when the target feature takes a value that satisfies a predetermined condition, and the average duration, standard deviation, and the like are calculated. Data representing a distribution, such as a histogram of duration, may be generated, and a duration to be used in place of the average duration and a time width to be used in place of the standard deviation may be determined from the data representing the distribution. The average duration and standard deviation may be calculated by the information processing device 300, or separately calculated results may be input.

The RF table storage unit 320 stores an RF table that holds the relationship between the network configuration and the receptive area size. For example, when the on/off pattern of the switch is repeated from the initial pattern for every m stages in a power-of-two pattern, the receptive area size (RF size) corresponding to the number of stacked stages L of the network has a relationship as shown in Fig. 8. In Fig. 8, the vertical axis represents the receptive area size, and the horizontal axis represents the number of stacked stages L of the network. As described above, the receptive area size is expressed as follows:
RF ₂ ^m (p ₁ ...p _L )=1+Σ _l=1 ^L Δ ₂ ^m (p _l )

L=1 is a network consisting of only the base network, with m=1. In Fig. 8, this network corresponds to point 801. If we make L=2 by stacking one level of a residual network that employs the same on-off pattern as the switches in the base network, we arrive at point L=2 on the curve extending to the right from point 801. On this curve, Δ ₂ ¹ (p _l ) = 2 ^{(l-1) mod 1} is the receptive field size.

L=2 is a network consisting of a base network and a residual network. If m=2, this network corresponds to point 802 in FIG. 8. If the on/off pattern of the switches is returned to the initial pattern and one stage of the residual network is added, the point becomes L=3 on the curve extending to the right from point 802. On this curve, Δ ₂ ² (p _l ) = 2 ^{(l-1) mod 2} is the receptive field size. Note that on this curve, the on/off pattern of the switches is repeated every two stages.

In this way, points corresponding to neural networks where L = m are lined up on line 800, and a curve extends from each point on line 800, corresponding to a network configuration in which L is increased by setting L at that point to m and returning the on/off pattern of the switches to the initial pattern and stacking them up. On this curve, the residual network is stacked up by returning the on/off pattern of the switches to the initial pattern every m stages.

For example, if the average duration of an observed event is 100 and a time range of one standard deviation is taken above and below, the result is a gray band in Figure 8. A network configuration that corresponds to a point that falls within this band is estimated to be a favorable network configuration for this observed event.

When the on/off pattern of the switches is repeated from the initial pattern for every m stages in a power-of-three pattern, the reception area size (RF size) corresponding to the number of stacked stages L of the network has the relationship shown in Fig. 9. As described above, the reception area size is expressed as follows:
RF ₃ ^m (p ₁ ...p _L )=1+Σ _l=1 ^L Δ ₃ ^m (p _l )

Even with a power-of-three pattern, as in Figure 8, a straight line of L = m extends as L increases, and from each point on the line, a curve extends that corresponds to a network configuration in which L is increased by returning the on/off pattern of the switches to the initial pattern and stacking them up. On this curve, the residual network is stacked up by returning the on/off pattern of the switches to the initial pattern every m stages. In Figure 9, for example, if the average duration of a certain observed event is 100 and a time width of one standard deviation is taken above and below, a grey band is formed. The network configuration that corresponds to the points that fall into this band is estimated to be the preferred network configuration for this observed event.

When the on/off pattern of the switches is repeated from the initial pattern for every m stages in a power-of-four pattern, the reception area size (RF size) corresponding to the number of stacked stages L of the network has the relationship shown in Fig. 10. As described above, the reception area size is expressed as follows:
RF ₄ ^m (p ₁ ... p _L ) = 1 + Σ _{l = 1} ^L Δ ₄ ^m (p _l )

Even with a power-of-four pattern, as in Figure 8, the straight line of L = m extends as L increases, and from each point on the line, a curve extends that corresponds to a network configuration in which L is increased by returning the on/off pattern of the switches to the initial pattern and stacking them up. On this curve, the residual network is stacked up by returning the on/off pattern of the switches to the initial pattern every m stages. In Figure 10, for example, if the average duration of a certain observed event is 100 and a time width of one standard deviation is taken above and below, a grey band is formed. The network configuration that corresponds to the points that fall into this band is estimated to be the preferred network configuration for this observed event.

In this way, the receptive area size is determined by the combination of L, m, and b, so the correspondence between receptive area size and network configuration is stored in a table.

Even if a switch on/off pattern other than a power pattern of the base b is used, the correspondence between size and network configuration can be stored in a table in advance.

The candidate extraction unit 310 extracts candidate network configurations for the observed event from the data stored in the RF table storage unit 320 based on the average duration of the observed event, etc. The model construction unit 350 constructs trained networks for each of the extracted candidate network configurations and selects the optimal trained network from among the candidates. During this process, the model construction unit 350 works in conjunction with the learning processing unit 340 and the recognition accuracy calculation unit 360. The learning processing unit 340 executes a network learning process using the learning data stored in the learning data storage unit 330. The recognition accuracy calculation unit 360 executes a process of calculating the recognition accuracy of the constructed network (i.e., the trained network) using, for example, the data stored in the learning data storage unit 330.

Next, the process of generating a trained network having a suboptimal network configuration will be described with reference to Figs. 11 to 15. First, the candidate extraction unit 310 calculates statistics regarding the duration of observed events (e.g., average duration, standard deviation, etc.) using data stored in the training data storage unit 330 (Fig. 11: step S1). This step is omitted if the average duration, standard deviation, etc. of observed events are input.

Next, the candidate extraction unit 310 extracts candidate models (i.e., network configurations) from the RF table stored in the RF table storage unit 320 based on the average duration of the observed events (step S3). As described above, for example, a time width of plus or minus one standard deviation is set for the average duration, and network configurations having a receptive field size that falls within the range of receptive field sizes corresponding to that time width are extracted. In other words, network configurations that fall within the bands described in Figures 8 to 10 are extracted.

Next, the model construction unit 350 identifies one unprocessed candidate model (step S5). Then, the model construction unit 350 determines whether the identified candidate model is a model with a power constraint (i.e., a power pattern model) (step S7). If it is a model with a power constraint, the model construction unit 350 executes a first model construction evaluation process (step S9). Then, the process proceeds to step S13. On the other hand, if it is not a model with a power constraint, the model construction unit 350 executes a second model construction evaluation process (step S11). Then, the process proceeds to step S13.

Then, the model construction unit 350 determines whether or not there are any unprocessed candidate models (step S13). If there are any unprocessed candidate models, the process returns to step S5. On the other hand, if there are no unprocessed candidate models, the model construction unit 350 selects a trained network to be adopted from the trained networks constructed from each candidate model based on its recognition accuracy (step S15). For example, the trained network with the highest recognition accuracy is selected.

By performing this process, it becomes possible to generate a suboptimal trained neural network based on the duration of the observed event.

Next, the first model construction evaluation process will be described with reference to Figures 12 and 13. First, the model construction unit 350 identifies b, m, and L of the candidate model (step S21). Then, the model construction unit 350 constructs a base network based on b (step S23). If b = 2, the switch on/off pattern as shown in Figure 4 will be adopted, and since this is a base network, this initial pattern will be adopted. If b = 3, the switch on/off pattern as shown in Figure 6 will be adopted, and since this is a base network, this initial pattern will be adopted. After that, the basic configuration of the base network as shown in Figure 1 will be adopted. The number of nodes and other information will also be set as a basic configuration, and will be adopted as is.

Furthermore, the model construction unit 350 causes the learning processing unit 340 to learn the base network N times using the learning data stored in the learning data storage unit 330, and causes the recognition accuracy calculation unit 360 to calculate the recognition accuracy of each of the trained base networks using the data stored in the learning data storage unit 330 (step S25). Since there is a part in the learning where parameters are set using random numbers, N times of learning are performed and the recognition accuracy is calculated for each. Then, the model construction unit 350 selects the one with the highest recognition accuracy from the N trained base networks (step S27). The process then transitions to the process of FIG. 13 via terminal A.

Moving on to the explanation of the process in FIG. 13, the model construction unit 350 sets l=2 (step S29). Then, the model construction unit 350 constructs the l-th stage residual network based on b and m (step S31). The (l mod m)-th pattern of the on-off pattern of the switches in the base b is adopted. After that, the basic configuration of the residual network as shown in FIG. 2 is adopted. The number of nodes and other details are also set to the basic configuration and adopted as is.

Then, the model construction unit 350 causes the learning processing unit 340 to perform N training of the l-th stage residual network using the learning data stored in the learning data storage unit 330 (step S33). N in this step may be a value different from N in step S25. The model construction unit 350 then builds N l-stage trained networks by stacking each l-th stage trained network on the (l-1)-stage trained network, and causes the recognition accuracy calculation unit 360 to calculate the recognition accuracy of each network (step S35). The model construction unit 350 then selects the network with the highest recognition accuracy from the N l-stage trained networks (step S37).

Then, the model construction unit 350 sets l = l + 1 (step S39). Then, the model construction unit 350 judges whether l ≦ L or not (step S41). If l ≦ L, the process returns to step S31. On the other hand, if l > L, the process returns to the process that called the call.

By performing this process, we can obtain an L-stage trained network that has been properly trained for the specific network configuration that was selected as a candidate in the power pattern.

Next, the second model construction evaluation process will be described with reference to Figures 14 and 15. When executing the second model construction evaluation process, the switch on/off pattern is not based on a power pattern, so it is assumed that the switch on/off pattern is defined in advance for each stage.

First, the model construction unit 350 constructs a base network based on the on/off pattern of the first stage switches in the candidate model (step S51). The on/off pattern of the first stage switches is applied to the basic configuration of the base network as shown in FIG. 1.

Furthermore, the model construction unit 350 causes the learning processing unit 340 to perform N training of the base network using the training data stored in the training data storage unit 330, and causes the recognition accuracy calculation unit 360 to calculate the recognition accuracy of each of the trained base networks using the data stored in the training data storage unit 330 (step S53). Since there is a part in the training where parameters are set using random numbers, N training is performed and the recognition accuracy is calculated for each. Then, the model construction unit 350 selects the one with the highest recognition accuracy from the N trained base networks (step S55). The process then transitions to the process of FIG. 15 via terminal B.

Moving on to the explanation of the process in FIG. 15, the model construction unit 350 sets l=2 (step S57). Then, the model construction unit 350 constructs the l-th stage residual network based on the on/off pattern of the l-th stage switch (step S59). The on/off pattern of the l-th stage switch is applied to the basic configuration of the residual network as shown in FIG. 2. Note that m may be set for pattern repetition, as in the case of the power pattern. In this case, the (l mod m)-th on/off pattern is adopted.

Then, the model construction unit 350 causes the learning processing unit 340 to perform N training of the l-th stage residual network using the learning data stored in the learning data storage unit 330 (step S63). N in this step may also be a value different from N in step S53. The model construction unit 350 then builds N l-stage trained networks by stacking each l-th stage trained network on the (l-1)-stage trained network, and causes the recognition accuracy calculation unit 360 to calculate the recognition accuracy of each network (step S63). The model construction unit 350 then selects the network with the highest recognition accuracy from the N l-stage trained networks (step S65).

Then, the model construction unit 350 sets l = l + 1 (step S67). Then, the model construction unit 350 judges whether l ≦ L or not (step S69). If l ≦ L, the process returns to step S59. On the other hand, if l > L, the process returns to the process that called the call.

By performing this process, it becomes possible to obtain an L-stage trained network that has been properly trained for a specific network configuration that is a candidate for a network configuration that is not a power pattern.

By performing this type of processing, it becomes possible to automatically generate a trained network that is suboptimal in detecting or recognizing observed events.

[Embodiment 2]
[Network Configuration Example According to the Present Embodiment]
In the first embodiment, the receptive field size of the network is adjusted to the average duration of the observed events. However, if the distribution of the duration of the observed events to be detected or recognized has a single peak centered on the average duration like a normal distribution, it is sufficient. However, if the distribution has multiple peaks, it is preferable to use multiple networks with different receptive field sizes and integrate the outputs of each network.

In this embodiment, for example, a fully connected linear network that generates a final output feature by weighting and integrating all output features of the base network and the residual network in a network that realizes each of a plurality of receptive field sizes corresponding to a plurality of durations is provided in the output stage. In general, a FFNN with one hidden layer that determines weights using random numbers and inputs the results of passing through a nonlinear activation function to the fully connected linear network may be provided in the output stage. Furthermore, a network in which multiple stages of such FFNNs are stacked may be provided in the output stage.

An example of such a network is shown in FIG. 16. In the example of FIG. 16, there is a network 1600 with a receptive range size RF1 corresponding to a first duration that is, for example, a power of 2 pattern and a peak of the distribution, a network 1610 with a receptive range size RF2 corresponding to a second duration that is, for example, a power of 3 pattern and a peak of the distribution, and an integrated network 1620, which is a fully connected linear network that receives as input the output features of each stage of the network 1600 and the output features of each stage of the network 1610. By using the output features of each stage in this way, it is possible to deal with cases where the distribution of duration has a spread. However, in order to reduce the amount of calculation, an integrated network 1620 that receives as input the output features of the network 1600 and the output features of the network 1610 may be added.

In addition, even if there are multiple peaks in the distribution of duration, if the selected networks have an inclusion relationship in terms of network structure, a network configuration such as that shown in FIG. 17 may be adopted. For example, as shown in FIG. 17, a first network 1700 with a receptive field size RF1 corresponding to the first duration, which is the peak of the distribution, is obtained, and a second network with a receptive field size RF2 corresponding to the second duration, which is the peak of the distribution, is obtained. If this second network has the same structure as the first network 1700 up to the second stage, an integrated network 1720, which is a fully connected linear network that inputs the output features of each stage of the first network 1700, is connected. In this case, an integrated network 1720 that inputs the output features of the second stage and the output features of the final stage may be adopted.

A more specific example will be described with reference to FIG. 18. Consider an observed event having two durations, duration 1 corresponding to peak 1 of the distribution and duration 2 corresponding to peak 2 of the distribution. A suboptimal network for each duration is selected by executing the process described in the first embodiment. For example, the network selected for duration 1 is selection model A1, and the network selected for duration 2 is selection model A2. The selection models A1 and A2 are both models in the 6th and 19th stages, respectively, on the same Δ ₂ ⁵ (p _l ) curve. In this case, the detection or recognition results of one acoustic event will be obtained separately from the two models in the 6th and 19th stages. In such a case, integration is performed by adopting the one with the larger receptive range size and adding an integration network 1720.

On the other hand, suppose that the network selected for duration 1 is selection model B1, and the network selected for duration 2 is selection model B2. In this case, selection model B1 is the 16th model on the Δ ₂ ⁵ (p _l ) curve, and selection model B2 is the 13th model on the Δ ₂ ³ (p _l ) curve. In this case, when there is no inclusion relationship, the two networks are adopted and an integrated network is further added to integrate them, as shown in FIG.

Generally, when stacking L-stage networks, the number of dimensions of the output feature vector y _l,t ∈R ^R×1 (l=1, ..L) of the l-th stage network and the output feature vector y _F,t ∈R ^R×1 of the fully connected linear network is the number of nodes in the output layer, R. The connected input feature vector of the fully connected linear network is expressed as y _C,t ^T = [y _1,t ^T , y _2,t ^T , .. y _F,t ^T ] ∈R ^1×RL .

The training data combined input feature vector _yC,t , the fully connected linear network output feature vector _yF,t , and the target feature vector _rt are assumed to be in the range of 1≦t≦N, with N pieces of vectors. Here, _YC = [ _yC,1 ,... _yC,N ] ^T ∈R ^N×RL , _YF = [ _yF,1 ,... _yF,N ] ^T ∈R ^N×R , and the fully connected linear network parameters γ∈R ^RL×R are expressed as matrices as follows:
_YF = _YC ·γ

Then, the target matrix R is represented as R = [ _r1 ,... _rN ] ^T ∈ ^{R N x R} , and the error between the target feature vector and the output vector of the fully connected linear network is represented as a matrix E = [ _e1 ,... _eN ] ^T ∈ R ^{N x R} , as follows.
E=RY _F =RY _C・γ

ここで定数Ｃは正則化項と残差項のバランスを決めるパラメータである。この最適解は、以下のように表される。

With the condition that this relationship is satisfied, the parameter γ that minimizes the evaluation function expressed as the sum of the regularization term to prevent overlearning and the residual term is obtained as the optimal solution.

Here, the constant C is a parameter that determines the balance between the regularization term and the residual term. The optimal solution is expressed as follows:

[Suboptimal network configuration]
The information processing device that performs the processing according to this embodiment is similar to that shown in Fig. 7. Then, the processing shown in Fig. 19 and Fig. 20 is executed. First, the candidate extraction unit 310 calculates distribution data of the duration of observed events (e.g., the duration of a peak, the standard deviation, etc.) using the data stored in the learning data storage unit 330 (Fig. 19: step S101). This step is omitted when the duration of a peak, the standard deviation, etc. in the distribution of the duration of observed events are input.

Next, the candidate extraction unit 310 determines whether there is one peak in the distribution of durations (step S103). If there are multiple peaks, the process proceeds to the process in FIG. 20 via terminal C. On the other hand, if there is a single peak, the model construction unit 350 executes the semi-optimization process described in the first embodiment (step S105). More specifically, steps S3 to S15 in FIG. 11 are executed.

Then, the model construction unit 350 judges whether the standard deviation in the distribution of duration is equal to or greater than a threshold (step S107). If the distribution of duration is a distribution with no spread, such as one peak with a standard deviation less than the threshold, the process is substantially the same as the first embodiment, and ends. On the other hand, if the distribution of duration is a distribution with a spread, such as one peak with a standard deviation greater than or equal to the threshold, the model construction unit 350 connects an integrated network that inputs the output features of each stage to the trained network generated in step S105, and trains the integrated network using the training processing unit 340 to generate a final trained network (step S109). In this way, even with such a spread distribution, a trained network with high recognition accuracy can be generated. Note that in step S109, the integrated network with the highest recognition accuracy may be adopted after training N integrated networks.

Moving on to the explanation of the process in FIG. 20, if there are multiple peaks in the distribution of durations, the model construction unit 350 identifies one duration related to an unprocessed peak (step S111). Then, the model construction unit 350 executes the semi-optimization process described in the first embodiment (step S105). More specifically, steps S3 to S15 in FIG. 11 are executed.

Then, the model construction unit 350 determines whether or not there is an unprocessed peak (step S115), and if there is an unprocessed peak, the process returns to step S111. On the other hand, if there is no unprocessed peak, the model construction unit 350 determines whether or not there is a structural inclusion relationship in the multiple trained networks that have been generated (step S117). This is to determine whether or not to adopt a structure such as that shown in FIG. 17. If there is no structural inclusion relationship in the multiple trained networks that have been generated, all trained networks are adopted, and the process proceeds to step S121.

On the other hand, if there is at least one structural inclusion relationship among the multiple trained networks generated, the model construction unit 350 adopts the longer of the trained networks that have a structural inclusion relationship (step S119). Trained networks that do not have a structural inclusion relationship are adopted as is. The model construction unit 350 then concatenates an integrated network that uses the output features of each stage of the adopted trained network as input, and trains the integrated network using the training processing unit 340 to generate a final trained network (step S121). The process then ends via terminal D. Note that, even in step S121, N integrated networks may be trained and the integrated network with the highest recognition accuracy may be adopted.

In addition, it is possible to set in advance whether to use the output feature values of each stage of the suboptimal trained network as the input of the integrated network, or the final output feature values of the suboptimal trained network as the input of the integrated network. It is also possible to set in advance whether to use the output feature values of the stage corresponding to the trained network that was not adopted, which is structurally inclusive, and the final output feature values of the adopted trained network as the input of the integrated network, or the output feature values of each stage of the adopted trained network as the input of the integrated network.

By performing this process, it becomes possible to generate an appropriate trained network that corresponds to the distribution of the duration of observed events.

[Example]
As an experiment for recognizing work activities based on work sounds (i.e., air-conducted sounds), an experiment for recognizing life activities based on nine types of life sounds was conducted. The life activities were: making a phone call, cooking, washing dishes, eating, cleaning, dealing with visitors, watching television, and working. The life sounds were digital signals sampled at 16 kHz and quantized at 16 bits. After passing through a high-frequency emphasis filter, FFT (Fast Fourier Transformation) was calculated with a window width of 30 ms and a window shift width of 15 ms to calculate a 40-dimensional Mel filter bank feature time series. Life sounds from 4 hours and 30 minutes (30 minutes for each life activity) were used for learning, and life sounds from 2 hours and 15 minutes (15 minutes for each life activity) were used for testing.

Learning of each network was repeated until 11 residual networks were stacked on top of the base network. The on/off pattern of the switches connected to the memory mechanism was a power of 2 pattern as shown in FIG. 4. The time difference between the time of the rightmost on switch and the time of the leftmost on switch was a monotonically increasing pattern of Δ(p _l ) = 2 ^{l - 1} for l = 1, . . . , 12, respectively. The receptive field size when the residual networks were stacked up to 10 stages was 1024, which is about 15 seconds when converted to time. The receptive field size when the residual networks were stacked up to 11 stages was 2048 (about 30 seconds), and when the residual networks were stacked up to 12 stages, it was 4096 (about 1 minute). The results of converting the receptive field size at each stage into time are shown in FIG. 21. The horizontal axis is the number of stages L of the network stack, and the vertical axis is seconds.

The number of nodes in the hidden layer was the same for all 12-stage networks, and in this experiment, 10 types were used, ranging from 1000 to 10,000, in increments of 1000. Learning and testing were performed on all 12-stage networks for each number of nodes. FIG. 22 shows the recognition results calculated using the F-value. In FIG. 22, the horizontal axis represents the number of network parameters that can be adjusted by learning, and the vertical axis represents the F-value. In this embodiment, the total connection weight matrix between the input layer and the hidden layer {W _l |l=1,...,12} is set from random numbers regardless of the learning data, so it is not included in the number of adjustable network parameters. Only the total connection weight matrix between the hidden layer and the output layer {β _l |l=1,...,12} is adjusted by learning.

One of the features of the network of this embodiment is that the input features to the residual network are a combination of the input features (observation features such as acoustic features) of the immediately lower base network or residual network and the output features (features inferring the target). To confirm this effect, an experiment was conducted in which the on-off pattern was the same as in the above experiment, but the input features to the residual network were only the input features (observation features such as acoustic features) of the immediately lower network. The results are shown in Figure 23. For ease of comparison, the results of the network according to this embodiment (Figure 22) are also shown in Figure 23. As such, the F-measure is higher in this embodiment even with the same level of adjustable network parameters, and the effect is evident.

The network according to this embodiment is equipped with a storage mechanism in which a finite number of delay elements are connected in series, and by turning on and off the switches connected to each path, the features to be input to the FFNN can be arbitrarily selected within the stored range. The on/off patterns can be set differently for the base network and each residual network. To verify this effect, the input features to the residual network are a combination of the input features (observation features such as acoustic features) and output features (features inferring the target) of the base network or residual network immediately below, which is the same as in the experiment with the network according to this embodiment, but an experiment was conducted in which the on/off pattern was the same for all networks.

The on-off pattern in this experiment was such that all 103 consecutive features from the current time t to time (t-102) were input to the FFNN, and this pattern was applied to a total of 10 stages of the network. In this case, the receptive field size was 1021. The results are shown in Figure 24. For ease of comparison, the results of the network according to this embodiment (Figure 22) are also shown in Figure 24. As can be seen, the F-value of the network according to this embodiment is higher, and the effect is evident.

In all of the networks used in the above three experiments, the FFNN of the residual network does not learn to directly infer target features, but rather learns to infer residual features obtained by subtracting from the target features the output features obtained by inferring the target features by the network one level lower. The output features obtained by the residual network inferring the target are obtained by adding the features obtained by inferring the residual features by the FFNN of the residual network to the output features obtained by inferring the target by the network one level lower. The above experimental results confirm that the recognition accuracy of the network of this embodiment, which has all three features, is the best.

Figure 25 shows the results of a comparative experiment with a CNN (Convolutional Neural Network). To meet the condition that the receptive field size of the network according to this embodiment is 1024, a Mel filter bank feature time series of (40 feature dimensions) x (receptive field size 1024) was regarded as an image, and a CNN was trained. In the example of Figure 25, when the number of adjustable parameters is small, CNN may be better, but when the number of adjustable parameters increases, CNN overlearns and the recognition accuracy deteriorates. In contrast, the F-measure of the network according to this embodiment increases monotonically even when the number of stacked network stages increases and the number of parameters increases, and the effect of this embodiment on the recognition accuracy and generalization performance is recognized.

In addition, we conducted a recognition experiment of 11 types of manual tasks based on body-conducted sound. The manual tasks used in the experiment were drawing a circle with a ballpoint pen, opening and closing a lid on a drink cup, typing on a keyboard, clicking the mouse, moving the mouse, rotating the mouse wheel, gripping paper, cutting paper with scissors, stapling, gripping plastic, and resting. The body-conducted sound was A/D converted with 8 kHz sampling and 16-bit quantization, passed through a high-frequency emphasis filter, and then FFT was calculated with a window width of 30 ms and a window movement width of 15 ms to calculate a 40-dimensional Mel filter bank feature time series. 22 minutes of body-conducted sound (2 minutes for each manual task) was used for model training, and 11 minutes of body-conducted sound (1 minute for each manual task) was used for testing. Learning of each network was repeated until 11 levels of residual networks were stacked on top of the base network. The on-off pattern of the switch connected to the memory mechanism was the power-of-two pattern shown in Figure 4. The results of converting the receptive area size to time for each number of stacking layers are the same as those shown in Figure 21 above.

The number of nodes in the hidden layer was the same for all 12-stage networks, and in this experiment, there were 10 types, ranging from 1,000 to 10,000, in increments of 1,000. Then, learning and testing were performed for each network. Figure 26 shows the recognition results calculated using the F-value. In Figure 26, the horizontal axis represents the number of network stacking stages L, and the vertical axis represents the F-value. For each number of stacking stages, 10 bar graphs show the recognition results for 10 types of numbers of nodes in the hidden layer, ranging from 1,000 to 10,000. As this shows, as the number of network stages increases, the F-value increases, but it saturates at around 8 stages. On the other hand, increasing the number of nodes in the hidden layer is effective when the number of network stages is small.

Although the embodiment of the present invention has been described above, the present invention is not limited to this. For example, the functional configuration example of the information processing device shown in FIG. 7 is an example, and may not match the program module configuration. In addition, multiple function blocks may be integrated into one function block, or one function block may be divided into multiple function blocks. In addition, the processing flow is also an example, and multiple steps may be executed in parallel or the execution order may be changed as long as the processing result does not change.
Furthermore, the information processing device 300 may be implemented by one computer or may be implemented by multiple computers.

The information processing device 300 described above is a computer device, and as shown in FIG. 27, a memory 2501, a CPU (Central Processing Unit) 2503, a hard disk drive (HDD: Hard Disk Drive) 2505, a display control unit 2507 connected to a display device 2509, a drive device 2513 for a removable disk 2511, an input device 2515, and a communication control unit 2517 for connecting to a network are connected by a bus 2519. The HDD may be a storage device such as a solid state drive (SSD: Solid State Drive). An operating system (OS) and an application program for implementing the processing in the embodiment of the present invention are stored in the HDD 2505, and are read from the HDD 2505 to the memory 2501 when executed by the CPU 2503. The CPU 2503 controls the display control unit 2507, the communication control unit 2517, and the drive device 2513 according to the processing contents of the application program to perform a predetermined operation. Data during processing is primarily stored in memory 2501, but may also be stored in HDD 2505. In an embodiment of the present technology, an application program for implementing the above-described processing is distributed by being stored in a computer-readable removable disk 2511, and is installed in HDD 2505 from drive device 2513. It may also be installed in HDD 2505 via a network such as the Internet and communication control unit 2517. Such a computer device realizes the various functions described above by organic cooperation between hardware such as the above-described CPU 2503 and memory 2501 and programs such as the OS and application programs.

The data used in executing the above-mentioned processes is stored in a storage device such as the memory 2501 or the HDD 2505, regardless of whether it is data in the middle of processing or the result of processing.
The above-described embodiment can be summarized as follows.

The trained network according to the present embodiment includes (A) a first network (e.g., a base network) including a first feedforward neural network that receives as input an observed feature at a time specified based on a predetermined timing pattern, and (B) a second network (e.g., a residual network) of (L-1) stages (L is an integer of 2 or more) including a second feedforward neural network that receives as input a concatenated feature that is a concatenation of an output feature and an observed feature of a network one stage lower at a time specified based on a predetermined timing pattern, and each of the second networks (e.g., a residual network) includes an adder that calculates an output feature by adding the output of the second feedforward neural network to the output feature of the network one stage lower, and is a trained network that can be executed by a computer. The predetermined timing patterns include first to m-th (1 to m is an integer of L or less), and the (x mod m)-th timing pattern of the first to m-th timing patterns is applied to each of the first and second networks based on the number of stages x (1≦x≦L) from the bottom stage of the network.

With a network having such a structure, the second network can effectively correct biases and errors in the inference results below the second network, thereby improving inference accuracy. In addition, since inference can be performed using features of various timing patterns, learning efficiency is also improved. Note that the receptive field size, which is the difference between the oldest and newest times of the observed features used in the entire network to be constructed, varies widely depending on the variation in timing patterns, making it easier to determine a network with a more preferable configuration depending on the observed event.

Each of the first and second networks may use a memory area (e.g., a storage mechanism) for storing observed features at times specified by the timing pattern for that network. In this way, it becomes easier to utilize observed features at any time.

Furthermore, the y-th (1≦y≦m) timing pattern among the first to m-th timing patterns may include b times at equal intervals, with the difference between the latest time and the oldest time at which the observed feature should be used being (b−1)×b ^y−1 . This makes it possible to handle power patterns of base b.

The trained network may further include a third network (e.g., an integrated network) that receives as input the output features of the top-stage network and at least one of the output features of the first network and a second network other than the top-stage network. This is because including such a configuration may result in higher inference accuracy.

The trained network may also include a plurality of trained networks as described above, and may further include a third network (e.g., an integrated network) that uses as input the output features of at least the top-stage network in each trained network, and may be executable by a computer. This is because such a configuration may have higher inference accuracy than a single trained network.

In addition, in the generation method according to the present embodiment, a network includes a first network including a first feedforward neural network that receives as input an observed feature at a time specified based on a predetermined timing pattern, and a second network of (L-1) stages (L is an integer of 2 or more) that includes a second feedforward neural network that receives as input a concatenated feature that is a concatenation of the output feature and the observed feature of a network one stage lower at a time specified based on a predetermined timing pattern, and each of the second networks includes an adder that calculates an output feature by adding the output of the second feedforward neural network to the output feature of the network one stage lower, and the predetermined timing patterns include first to mth (m is an integer of 1 to L), and a network that applies the (x mod m)th timing pattern of the first to mth timing patterns to each of the first and second networks based on the number of stages x (≦L) from the bottom stage of the network is used as a target network. The generation method includes the steps of: (A) identifying multiple candidates for the configuration of the network to be constructed based on the duration of the observed event that is the source of the observed feature and the receptive domain size, which is the difference between the oldest and newest time of the observed feature used in the entire network to be constructed; (B) generating a trained network for each of the multiple candidates; and (C) selecting the trained network to be used based on the inference accuracy of the generated trained network.

A network with a receptive field size that corresponds to the duration of an observed event is able to perform more appropriate inference, so a trained network is generated using the process described above. Note that for networks with the characteristics described above, the relationship between the network structure and receptive field size is clear, making processing easy.

The step of identifying multiple candidates for the configuration of the network to be constructed may also include a step of identifying, as candidates, combinations of m and L of the network to be constructed that have a receptive range size that falls within a predetermined range corresponding to a predetermined time range including the duration. In this way, it becomes possible to appropriately select candidate networks.

In addition, the receptive field size mentioned above may be calculated by adding the difference between the latest and oldest times at which the observed features should be used in each network from the bottom to the top for the entire network being constructed, and then adding this result to 1.

Furthermore, the yth (1≦y≦m) timing pattern among the first to mth timing patterns described above may include b times at equal intervals, with the difference between the latest time and the oldest time at which the observed feature should be used being (b-1)×b ^y-1 . Also, the above-mentioned receptive field size may be calculated as 1+Σ ^L _i (b-1)×b ^{(i-1) mod m} .

Furthermore, when there are multiple durations to be considered in the observed event that is the source of the observed features, the generation method may further include (D) executing a step of identifying the multiple candidates for each of the multiple durations, a step of generating a trained network for each of the multiple candidates, and a step of selecting the trained network, and (E) a step of training a third network that integrates at least the output features of the selected trained networks.

Depending on the observed event, there may be one peak in the duration distribution or multiple peaks. When there are multiple peaks, performing this type of processing makes it possible to generate a trained network with high inference accuracy. For example, this generates a trained network like the one shown in Figure 16.

In addition, when the first trained network among the selected trained networks includes the second trained network, the third network may be a network that receives as input at least the output features of the first trained network and the output features of a stage in the first trained network that corresponds to the second trained network.

Even if there are multiple peaks in the duration distribution, in special cases, a trained network like the one above is appropriate. For example, a trained network like the one shown in Figure 17 is generated.

When a first trained network among the selected trained networks does not include a second trained network, a third network may be a network that receives as input at least the output features of the first trained network and the output features of the second trained network.

A program can be created to cause a computer to execute the generation method described above, and the program can be stored on various storage media.

In addition, the information processing device that executes the generation method described above may be realized by a single computer or may be realized by multiple computers, and these will collectively be referred to as an information processing system or simply a system.

300 Information processing device 310 Candidate extraction unit 320 RF table storage unit 330 Learning data storage unit 340 Learning processing unit 350 Model construction unit 360 Recognition accuracy calculation unit

Claims (14)

a first network including a first feedforward neural network that receives an observed feature value at a time specified based on a predetermined timing pattern;
a second network of (L-1) stages (L is an integer of 2 or more) including a second feedforward neural network that receives as input a concatenated feature that is a concatenation of an output feature and an observed feature of a network in a stage one level lower at a time specified based on the predetermined timing pattern, and including an adder that calculates an output feature by adding an output of the second feedforward neural network and the output feature of the network in the stage one level lower, the second network being executable by a computer,
The predetermined timing pattern includes first to m-th timing patterns (where m is an integer equal to or larger than 1 and equal to or smaller than L),
a (x mod m)-th timing pattern among the first to m-th timing patterns is applied to each of the first and second networks based on a number x (1≦x≦L) of stages from the bottom stage of the network. Each of the first and second networks comprises:
The trained network of claim 1 , further comprising a memory area for storing observed features at times specified by a timing pattern for the network. A y-th (1≦y≦m) timing pattern among the first to m-th timing patterns is
3. The trained network according to claim 1, wherein the difference between the newest time and the oldest time at which the observed feature should be used is (b-1) x b ^y-1 , and b times are included at equal intervals. The trained network according to any one of claims 1 to 3, further comprising a third network that receives as input the output feature of the top-stage network and at least one of the output feature of the first network and the second network other than the top-stage network. A system including a plurality of trained networks according to any one of claims 1 to 3,
A trained network executable by a computer, further comprising a third network that receives as input the output features of at least the top-stage network in each trained network. a first network including a first feedforward neural network receiving as input an observed feature at a time specified based on a predetermined timing pattern; and a second network of (L-1) stages (L is an integer of 2 or more) including a second feedforward neural network receiving as input a concatenated feature which is a concatenation of an output feature and an observed feature of a network one stage lower at a time specified based on the predetermined timing pattern, the second network including an adder that calculates an output feature by adding an output of the second feedforward neural network and an output feature of the network one stage lower, the predetermined timing pattern including first to m-th (m is an integer of 1 to L), and the network to be constructed applies an (x mod m)-th timing pattern of the first to m-th timing patterns to each of the first and second networks based on a stage number x (≦L) of the network from the bottom stage,
Identifying a plurality of candidates for the configuration of the network to be constructed based on the duration of the observed event that is the source of the observed feature and a receptive domain size that is the difference between the oldest time and the newest time of the observed feature used in the entire network to be constructed;
generating a trained network for each of the plurality of candidates;
Selecting a trained network to be used based on the inference accuracy of the generated trained network;
A computer-implemented method for generating a trained network. The step of identifying a plurality of candidates for the configuration of the network to be constructed includes:
The method of claim 6 , further comprising: identifying as candidates combinations of m and L of a network to be constructed that have the receptive range size falling within a predetermined range corresponding to a predetermined time range including the duration. The generation method according to claim 6 or 7, wherein the receptive field size is calculated by adding the difference between the latest time and the oldest time at which the observed feature should be used in each network from the bottom to the top for the entire network to be constructed and adding 1 to the result. A y-th (1≦y≦m) timing pattern among the first to m-th timing patterns is
The difference between the newest time and the oldest time for which the observed feature should be used is (b-1) x b ^y-1 , and b times are included at equal intervals.
8. The method according to claim 6, wherein the acceptance field size is calculated as 1+Σ ^L _i (b-1)×b ^{(i-1) mod m} . When there are a plurality of durations to be considered in the observed event that is the source of the observed feature,
performing, for each of the plurality of durations, the steps of identifying the plurality of candidates, generating a trained network for each of the plurality of candidates, and selecting the trained network;
The method according to claim 6 , further comprising the step of training a third network that integrates at least output features of the selected trained networks. Among the selected trained networks, if a first trained network includes a second trained network,
The generation method according to claim 10 , wherein the third network is a network that receives as input at least the output features of the first trained network and the output features of a stage in the first trained network that corresponds to the second trained network. Among the selected trained networks, a first trained network does not include a second trained network;
The generating method according to claim 10 or 11, wherein the third network is a network that receives as input at least the output features of the first trained network and the output features of the second trained network. A program for causing a computer to execute the generation method according to any one of claims 6 to 12. a first network including a first feedforward neural network receiving as input an observed feature at a time specified based on a predetermined timing pattern; and a second network of (L-1) stages (L is an integer of 2 or more) including a second feedforward neural network receiving as input a concatenated feature which is a concatenation of an output feature and an observed feature of a network one stage lower at a time specified based on the predetermined timing pattern, the second network including an adder that calculates an output feature by adding an output of the second feedforward neural network and an output feature of the network one stage lower, the predetermined timing pattern including first to m-th (m is an integer of 1 to L), and the network to be constructed applies an (x mod m)-th timing pattern of the first to m-th timing patterns to each of the first and second networks based on a stage number x (≦L) of the network from the bottom stage,
A means for identifying a plurality of candidates for the configuration of the network to be constructed based on a duration of an observation event that is the source of the observation feature and a receptive domain size that is a difference between the oldest time and the newest time of the observation feature used in the entire network to be constructed;
means for generating a trained network for each of the plurality of candidates;
A means for selecting a trained network to be used based on the inference accuracy of the trained network generated;
An information processing system having the above configuration.

Images