JP7764688B2 - Acoustic analysis method, acoustic analysis system and program - Google Patents

Description

This disclosure relates to technology for analyzing acoustic signals.

Analysis techniques have been proposed for estimating the beats of a piece of music by analyzing audio signals that represent the sounds being played in that piece of music. For example, Patent Document 1 discloses a technique for estimating the beats of a piece of music using a probabilistic model such as a hidden Markov model.

JP 2015-114361 A

Conventional techniques for estimating beats in music may erroneously estimate, for example, a backbeat as a beat, or may erroneously estimate a beat corresponding to a tempo twice the original tempo of the music. Furthermore, there is a possibility that the estimated beat may not match the user's intention, such as when a backbeat is estimated when the user is expecting a downbeat. Considering these circumstances, it is important to have a configuration that allows the user to change the positions on the time axis of multiple beats estimated from an audio signal. However, there is a problem in that the task of changing each individual beat throughout a piece of music to a desired time point is excessively burdensome. In consideration of these circumstances, one aspect of the present disclosure aims to obtain a time series of beats that matches the user's intention, while reducing the burden on the user of instructing changes to the position of each beat.

To solve the above problems, an acoustic analysis system according to one aspect of the present disclosure estimates multiple beats in a piece of music by analyzing an acoustic signal representing the sounds of the music being played, accepts instructions from a user to change the positions of some of the multiple beats, and updates the positions of the multiple beats in accordance with the instructions from the user.

An acoustic analysis system according to one aspect of the present disclosure includes an analysis processing unit that estimates multiple beats of a piece of music by analyzing an acoustic signal representing the sound of the piece of music being played, an instruction receiving unit that receives instructions from a user to change the positions of some of the multiple beats, and a beat updating unit that updates the positions of the multiple beats in response to instructions from the user.

A program according to one aspect of the present disclosure causes a computer system to function as an analysis processing unit that estimates multiple beats of a piece of music by analyzing an audio signal representing the sound of the music being played, an instruction receiving unit that receives instructions from a user to change the positions of some of the multiple beats, and a beat updating unit that updates the positions of the multiple beats in response to instructions from the user.

1 is a block diagram illustrating a configuration of an acoustic analysis system according to a first embodiment. FIG. 1 is a block diagram illustrating an example of the functional configuration of an acoustic analysis system. FIG. 10 is an explanatory diagram of an operation in which a feature extraction unit generates feature data. FIG. 1 is a block diagram illustrating the configuration of an estimation model. FIG. 1 is an explanatory diagram of machine learning for establishing an estimation model. 10 is a flowchart illustrating a specific procedure of a probability calculation process. FIG. 2 is an explanatory diagram of a state transition model. FIG. 10 is an explanatory diagram of a beat point estimation process. 10 is a flowchart illustrating a specific procedure of beat point estimation processing. FIG. 10 is a schematic diagram of an analysis screen. FIG. 10 is an explanatory diagram of an estimation model update process. 10 is a flowchart illustrating a specific procedure of an estimation model update process. 4 is a flowchart illustrating a specific procedure of a process executed by a control device. 10 is a flowchart illustrating a specific procedure of an initial analysis process. 10 is a flowchart illustrating a specific procedure of a beat point updating process. FIG. 10 is a block diagram illustrating an example of the functional configuration of an acoustic analysis system according to a second embodiment. FIG. 11 is a schematic diagram of an analysis screen in the second embodiment. 1 is an explanatory diagram of an estimated tempo curve, a maximum tempo curve, and an initial tempo curve. 10 is a flowchart illustrating a specific procedure of beat point estimation processing in the second embodiment. FIG. 11 is an explanatory diagram of a process for generating output data in the third embodiment.

A: First Embodiment FIG. 1 is a block diagram illustrating the configuration of an acoustic analysis system 100 according to a first embodiment. The acoustic analysis system 100 is a computer system that estimates multiple beats of a piece of music by analyzing an acoustic signal A that represents the performance sounds of the piece of music. The acoustic analysis system 100 includes a control device 11, a storage device 12, a display device 13, an operation device 14, and a sound emission device 15. The acoustic analysis system 100 is realized, for example, by a portable information device such as a smartphone or tablet terminal, or a portable or stationary information device such as a personal computer. The acoustic analysis system 100 can be realized as a single device, or as multiple devices configured separately from each other.

The control device 11 is composed of one or more processors that control each element of the acoustic analysis system 100. For example, the control device 11 is composed of one or more types of processors, such as a CPU (Central Processing Unit), SPU (Sound Processing Unit), DSP (Digital Signal Processor), FPGA (Field Programmable Gate Array), or ASIC (Application Specific Integrated Circuit).

The storage device 12 is one or more memories that store the programs executed by the control device 11 and various data used by the control device 11. The storage device 12 is composed of a known storage medium, such as a magnetic recording medium or semiconductor recording medium, or a combination of multiple types of storage medium. Note that the storage device 12 may also be a portable storage medium that can be attached to or detached from the acoustic analysis system 100, or a storage medium (e.g., cloud storage) that the control device 11 can write to or read from via a communication network such as the Internet.

The storage device 12 stores the audio signal A. The audio signal A is a series of samples representing the waveform of the musical performance sound. Specifically, the audio signal A represents at least one of the musical instrument sounds and the vocal sounds of the musical piece. The data format of the audio signal A is arbitrary. The audio signal A may be supplied to the acoustic analysis system 100 from a signal supply device separate from the acoustic analysis system 100. The signal supply device is, for example, a playback device that supplies the audio signal A recorded on a recording medium to the acoustic analysis system 100, or a communication device that supplies the audio signal A received from a distribution device (not shown) via a communication network to the acoustic analysis system 100.

The display device 13 displays images under the control of the control device 11. For example, various display panels such as a liquid crystal display panel or an organic EL (Electroluminescence) panel may be used as the display device 13. Note that the display device 13 may be separate from the acoustic analysis system 100 and connected to the acoustic analysis system 100 by wire or wirelessly. The operation device 14 is an input device that accepts instructions from the user. The operation device 14 may be, for example, a control operated by the user or a touch panel that detects contact by the user.

The sound emitting device 15 reproduces sound under the control of the control device 11. For example, a speaker or headphones may be used as the sound emitting device 15. Note that the sound emitting device 15 may be separate from the acoustic analysis system 100 and connected to the acoustic analysis system 100 by wire or wirelessly.

Figure 2 is a block diagram illustrating the functional configuration of the acoustic analysis system 100. The control device 11 executes a program stored in the storage device 12 to realize multiple functions for processing the acoustic signal A (analysis processing unit 20, display control unit 24, playback control unit 25, instruction receiving unit 26, and estimation model update unit 27).

The analysis processing unit 20 estimates multiple beats in a piece of music by analyzing the audio signal A. Specifically, the analysis processing unit 20 generates beat data B from the audio signal A. The beat data B is data representing each beat in the piece of music. Specifically, the beat data B is time-series data that specifies the time of each beat in the piece of music. For example, the beat data B specifies the time of each beat relative to the start point of the audio signal A. The analysis processing unit 20 of the first embodiment includes a feature extraction unit 21, a probability calculation unit 22, and an estimation processing unit 23.

[Feature extraction unit 21]
FIG. 3 is an explanatory diagram of the operation of the feature extraction unit 21. The feature extraction unit 21 generates a feature f[m] (m = 1 to M) of the audio signal A for each of M time points (hereinafter referred to as "analysis time points") t[m] on the time axis. Each analysis time point t[m] is a time point set on the time axis at a predetermined interval. The feature f[m] is an index representing the acoustic characteristics of the audio signal A. Specifically, a feature f[m] that tends to fluctuate significantly before and after a beat is used. For example, information regarding the intensity of the audio signal A, such as volume and amplitude, is used as the feature f[m]. Information regarding the frequency characteristics (timbre) of the audio signal A, such as MFCC (Mel-Frequency Cepstrum Coefficients), MSLS (Mel-Scale Log Spectrum), or CQT (Constant-Q Transform), is also used as the feature f[m]. However, the types of feature f[m] are not limited to the above examples. Furthermore, the feature f[m] may be a combination of multiple types of information related to the audio signal A.

The feature extraction unit 21 generates feature data F[m] for each analysis time point t[m]. The feature data F[m] corresponding to a given analysis time point t[m] is a time series of multiple feature quantities f[m] within a period U (hereinafter referred to as a "unit period") that includes that analysis time point t[m]. Figure 3 illustrates an example in which one unit period U includes five analysis time points t[m-2] to t[m+2], centered around the mth analysis time point t[m]. Therefore, the feature data F[m] is a time series of five feature quantities f[m-2] to f[m+2] within the unit period U. Note that the unit period U may include only one analysis time point [m]. In other words, the feature data F[m] may be composed of only one feature quantity f[m]. As can be understood from the above explanation, the feature extraction unit 21 generates feature data F[m] including the feature quantities f[m] of the acoustic signal A for each analysis time point t[m].

[Probability calculation unit 22]
2 generates output data O[m] from feature data F[m], which represents the probability P[m] that each analysis time point t[m] corresponds to a beat point in a piece of music. The generation of output data O[m] is repeated for each analysis time point t[m]. The larger the probability P[m], the higher the likelihood that analysis time point t[m] corresponds to a beat point. The probability calculation unit 22 uses an estimation model 50 to generate the output data O[m].

There is a correlation between the feature data F[m] at each analysis time t[m] of the audio signal A and the likelihood that the analysis time t[m] corresponds to a beat. The estimation model 50 is a statistical model that has learned this correlation. Specifically, the estimation model 50 is a trained model that has learned the relationship between the feature data F[m] and the output data O[m] through machine learning.

The estimation model 50 is configured, for example, by a deep neural network (DNN). The estimation model 50 is realized by combining a program that causes the control device 11 to execute a calculation to generate output data O[m] from feature data F[m], and multiple variables (specifically, weights and biases) that are applied to the calculation. The program and multiple variables that realize the estimation model 50 are stored in the storage device 12. The numerical values of each of the multiple variables that define the estimation model 50 are set in advance by machine learning.

Figure 4 is a block diagram illustrating the specific configuration of the estimation model 50. The estimation model 50 is composed of a convolutional neural network including an input layer 51, multiple intermediate layers 52 (52a, 52b), and an output layer 53. Multiple feature quantities f[m-2] to f[m+2] contained in one piece of feature data F[m] are input in parallel to the input layer 51.

The multiple intermediate layers 52 are hidden layers located between the input layer 51 and the output layer 53. The multiple intermediate layers 52 include multiple intermediate layers 52a and multiple intermediate layers 52b. The multiple intermediate layers 52a are located between the input layer 51 and the multiple intermediate layers 52b. Each intermediate layer 52a is composed of, for example, a combination of a convolutional layer and a pooling layer. Each intermediate layer 52b is, for example, a fully connected layer with ReLU as the activation function. The output layer 53 outputs output data O[m].

The estimation model 50 is divided into a first portion 50a and a second portion 50b. The first portion 50a is the input side portion of the estimation model 50. Specifically, the first portion 50a is the first half portion consisting of an input layer 51 and multiple intermediate layers 52a. The second portion 50b is the output side portion of the estimation model 50. Specifically, the second portion 50b is the second half portion consisting of multiple intermediate layers 52b and an output layer 53. The first portion 50a is the portion that generates intermediate data D[m] according to feature data F[m]. The intermediate data D[m] is data that represents the features of the feature data F[m]. Specifically, the intermediate data D[m] is data that represents the features that contribute to outputting output data O[m] that is statistically valid for the feature data F[m]. The second portion 50b is the portion that generates output data O[m] according to the intermediate data D[m].

Figure 5 is an explanatory diagram of machine learning that establishes the estimation model 50. For example, the estimation model 50 is established through machine learning by a machine learning system 200 that is separate from the acoustic analysis system 100, and the estimation model 50 is provided to the acoustic analysis system 100. For example, the estimation model 50 is sent from the machine learning system 200 to the acoustic analysis system 100.

Multiple sets of training data Z are used in the machine learning of the estimation model 50. Each of the multiple sets of training data Z is composed of a combination of training feature data Ft and training output data Ot. The feature data Ft represents features at a specific time point in the acoustic signal A prepared for training. Specifically, like the above-mentioned feature data F[m], the feature data Ft is composed of a time series of multiple features corresponding to different times on the time axis. The training output data Ot corresponding to a specific time point is data (i.e., a correct answer value) that represents the probability that that time point corresponds to a beat in the music. Multiple sets of training data Z are prepared for a large number of known songs.

The machine learning system 200 calculates an error function that represents the error between the output data O[m] output by an initial or provisional model (hereinafter referred to as the "provisional model") 59 when the feature data Ft of each training data Z is input, and the output data Ot of that training data Z. The machine learning system 200 then updates multiple variables of the provisional model 59 so as to reduce the error function. The provisional model 59 at the time when the above process has been repeated for each of the multiple training data Z is determined as the estimated model 50.

Therefore, the estimation model 50 outputs statistically valid output data O[m] for unknown feature data F[m] based on the underlying relationship between feature data Ft and output data Ot in multiple training data Z. In other words, the estimation model 50 is a trained model that has learned the relationship between training feature data Ft corresponding to each time point on the time axis and training output data Ot representing the probability that that time point corresponds to a beat. By inputting the feature data F[m] for each analysis time point t[m] into the estimation model 50 established by the above procedure, the probability calculation unit 22 generates output data O[m] representing the probability P[m] that the analysis time point t[m] corresponds to a beat.

Figure 6 is a flowchart illustrating the specific steps of the process Sa (hereinafter referred to as the "probability calculation process") executed by the probability calculation unit 22. The control device 11 executes the probability calculation process Sa by functioning as the probability calculation unit 22.

When the probability calculation process Sa begins, the probability calculation unit 22 inputs the feature data F[m] corresponding to the analysis time t[m] into the estimation model 50 (Sa1). The probability calculation unit 22 acquires the intermediate data D[m] output by the first part 50a of the estimation model 50 and stores the intermediate data D[m] in the storage device 12 (Sa2). The probability calculation unit 22 also acquires the output data O[m] output by the estimation model 50 (second part 50b) and stores the output data O[m] in the storage device 12 (Sa3).

The probability calculation unit 22 determines whether the above processing has been performed for M analysis times t[1] to t[M] in the song (Sa4). If the determination result is negative (Sa4: NO), the probability calculation unit 22 generates intermediate data D[m] and output data O[m] for the unprocessed analysis time point t[m] (Sa1 to Sa3). If processing has been performed for M analysis times t[1] to t[M] (Sa4: YES), the probability calculation unit 22 terminates the probability calculation processing Sa. As can be understood from the above explanation, as a result of the probability calculation processing Sa, M pieces of intermediate data D[1] to D[M] corresponding to different analysis times t[m] and M pieces of output data O[1] to O[M] corresponding to different analysis times t[m] are saved in the storage device 12.

[Estimation processing unit 23]
2 estimates multiple beat positions in a piece of music from M pieces of output data O[m] calculated for different analysis times t[m] by the probability calculation unit 22. Specifically, as described above, the estimation processing unit 23 generates beat position data B representing the time of each beat position in the piece of music. The probability calculation unit 22 generates the beat position data B using a state transition model 60.

Figure 7 is a block diagram illustrating the configuration of state transition model 60. State transition model 60 is a statistical model composed of multiple (N) states Q. Specifically, state transition model 60 is composed of a Hidden Semi-Markov Model (HSMM), and multiple beats are estimated using the Viterbi algorithm, an example of dynamic programming.

Figure 7 illustrates beats on a time axis. The time interval δ between two successive beats on the time axis (hereafter referred to as the "beat interval") is a variable value depending on the tempo of the music. Specifically, the faster the tempo, the shorter the beat interval δ. Multiple points (hereafter referred to as "elapsed points") Y[j] are set within the beat interval δ. Each elapsed point Y[i] (i = 1 to 4) is set on the time axis based on the beat. Specifically, elapsed point Y[0] is the point corresponding to the beat (the beginning of the beat), and elapsed points Y[1] to Y[4] are points that equally divide the beat interval δ. Elapsed point Y[3] is located after elapsed point Y[4], elapsed point Y[2] is located after elapsed point Y[3], and elapsed point Y[1] is located after elapsed point Y[2]. Elapsed point Y[0] corresponds to the end point (start or end point) of the beat interval δ. The time length from each beat point (elapsed point Y[0]) to each elapsed point Y can also be expressed as the phase based on the beat point. For example, time progresses in the order of elapsed point Y[4] → elapsed point Y[3] → elapsed point Y[2] → elapsed point Y[1], and reaches elapsed point Y[0] (a beat point) after elapsed point Y[1].

Each of the N states Q in the state transition model 60 corresponds to one of multiple tempos X[i] (i = 1, 2, 3, ...). Specifically, the N states Q correspond to different combinations of multiple tempos X[i] and multiple elapsed points Y[0] to Y[4]. That is, for each tempo X[i], there are five time series of states Q corresponding to different elapsed points Y[j]. In the following description, a state Q corresponding to a combination of tempo X[i] and elapsed point Y[j] may be referred to as "state Q[i,j]." On the other hand, when the distinction between tempo X[i] and elapsed point Y[j] is not particularly important, it will simply be referred to as "state Q." Note that the distinction between states Q according to elapsed point Y[j] may be omitted. That is, a configuration in which each of multiple states Q corresponds to a different tempo X[i] is also conceivable. In a form in which the progress point Y[j] is not distinguished, for example, a Hidden Markov Model (HMM) is used as the state transition model 60.

In the first embodiment, it is assumed that the tempo X changes only at beat points on the time axis (i.e., at elapsed point Y[0]). Under these assumptions, state Q[i,j] corresponding to each elapsed point Y[j] other than elapsed point Y[0] transitions only to state Q[i,j-1] corresponding to the immediately succeeding elapsed point Y[j-1]. For example, state Q[i,4] transitions to state Q[i,3], state Q[i,3] transitions to state Q[i,2], and state Q[i,2] transitions to state Q[i,1]. On the other hand, state Q[i,0], which corresponds to a beat, transitions occur from multiple states Q[i,1] (Q[1,1], Q[2,1], Q[3,1], ...) corresponding to different tempos X[i].

Figure 8 is an explanatory diagram of the process Sb (hereinafter referred to as the "beat point estimation process") in which the estimation processing unit 23 estimates multiple beat points in a piece of music using the state transition model 60. Also, Figure 9 is a flowchart illustrating the specific steps of the beat point estimation process Sb. The control device 11 functions as the estimation processing unit 23 to execute the beat point estimation process Sb.

When the beat point estimation process Sb begins, the estimation processing unit 23 calculates the observation likelihood Λ[m] for each of the M analysis times t[1] to t[M] (Sb1). The observation likelihood Λ[m] for each analysis time t[m] is set to a value corresponding to the probability P[m] represented by the output data O[m] for that analysis time t[m]. For example, the observation likelihood Λ[m] is set to the probability P[m] represented by the output data O[m], or a value calculated by a predetermined calculation for that probability P[m].

The estimation processing unit 23 calculates the path p[i,j] and likelihood λ[i,j] for each analysis time t[m] for each state Q[i,j] of the state transition model 60 (Sb2). The path p[i,j] is the path from another state Q to state Q[i,j], and the likelihood λ[i,j] is an index of the probability that the state Q[i,j] will be observed.

As mentioned above, only unidirectional transitions occur between the multiple states Q[i,0] to Q[i,4] corresponding to any given tempo X[i]. Therefore, as can be seen from Figure 8, the only path p[1,1] that reaches state Q[1,1] corresponding to tempo X[1] and passing point Y[1] at analysis time t[m] is path p from state Q[1,2] corresponding to the given tempo X[1] and the immediately preceding passing point Y[2]. Furthermore, the likelihood λ[1,1] of state Q[1,1] at analysis time t[m] is set to the likelihood corresponding to time t1, which is the time length d[1] corresponding to the given tempo X[1], prior to analysis time t[m]. Specifically, the likelihood λ[1,1] of state Q[1,1] is calculated by interpolating (e.g., linearly interpolating) the observation likelihood Λ[mA] at the analysis time t[mA] immediately before time t1 and the observation likelihood Λ[mB] at the analysis time t[mB] immediately after time t1.

On the other hand, the tempo X[i] may change at the elapsed point Y[0]. Therefore, as can be seen from Figure 8, for example, state Q[1,0], which corresponds to the tempo X[1] and the elapsed point Y[0], is reached by separate paths p from each of the multiple states Q[i,1] corresponding to different tempos X[i]. For example, state Q[1,0] is reached by path p1 from state Q[1,1], which corresponds to the combination of the tempo X[1] and the immediately preceding elapsed point Y[1], as well as path p2 from state Q[2,1], which corresponds to the combination of the tempo X[2] and the immediately preceding elapsed point Y[1]. The likelihood λ1 for path p1 from state Q[1,1] to state Q[1,0] is calculated by interpolating (e.g., linearly interpolating) the observation likelihood Λ[mA] at the analysis time t[mA] immediately before time t1 and the observation likelihood Λ[mB] at the analysis time t[mB] immediately after time t1, as in the previous example. Furthermore, the likelihood λ2 for path p2 from state Q[2,1] to state Q[1,0] is set to the likelihood at time t2, which is a time length d[2] corresponding to the tempo X[2] of state Q[2,1], before analysis time t[m]. Specifically, the likelihood λ2 is calculated by interpolating (e.g., linearly interpolating) the observation likelihood Λ[mC] at analysis time t[mC] immediately before time t2 and the observation likelihood Λ[mA] at analysis time t[mA] immediately after time t2. The estimation processing unit 23 selects the maximum value of the multiple likelihoods λ (λ1, λ2, ...) calculated for different tempos X[i] as the likelihood λ[1,0] of state Q[1,0] at analysis time t[m], and determines the path p corresponding to this likelihood λ[1,0] from among the multiple paths p (p1, p2, ...) reaching state Q[1,0] as the path p[1,0] to state Q[1,0]. Using the above procedure, the process of calculating the path p[i,j] and likelihood λ[i,j] for each of the N states Q is performed in the forward direction of the time axis for each analysis time t[m]. In other words, the path p[i,j] and likelihood λ[i,j] for each state Q are calculated for each of the M analysis times t[1] to t[M].

The estimation processing unit 23 generates a time series of M states Q corresponding to different analysis times t[m] (hereinafter referred to as the "state series") (Sb3). Specifically, the estimation processing unit 23 connects paths p[i,j] in order along the reverse direction of the time axis, starting from the state Q[i,j] corresponding to the maximum of the N likelihoods λ[i,j] calculated for the last analysis time t[M] of the song, and generates a state series from the M states Q located on the connected series of paths (i.e., the maximum likelihood path). In other words, the state series is generated by arranging, for each analysis time t[m], the states Q with the largest likelihood λ[i,j] among the N states Q.

The estimation processing unit 23 estimates each analysis time t[m] at which a state Q corresponding to the elapsed point Y[0] is observed among the M states Q constituting the state sequence as a beat, and generates beat data B specifying the time of each beat (Sb4). As can be understood from the above explanation, the analysis time t[m] at which the probability P[m] represented by the output data O[m] is high and the tempo transitions naturally to the ear is estimated as a beat in the music piece.

As described above, in the first embodiment, feature data F[m] for each analysis time point t[m] is input into the estimation model 50 to generate output data O[m] for each analysis time point t[m], and multiple beats are estimated from the output data O[m]. Therefore, based on the underlying relationship between the learning feature data Ft and the learning output data Ot, it is possible to generate output data O[m] that is statistically valid for unknown feature data F[m]. A specific example of the configuration of the analysis processing unit 20 is as described above.

The display control unit 24 in FIG. 2 causes the display device 13 to display an image. Specifically, the display control unit 24 causes the display device 13 to display the analysis screen 70 in FIG. 10. The analysis screen 70 is an image showing the results of the analysis of the acoustic signal A by the analysis processing unit 20.

The analysis screen 70 includes a first area 71 and a second area 72. The first area 71 displays a waveform 711 of the acoustic signal A. The second area 72 displays the results of an analysis of a portion of the acoustic signal A specified in the first area 71 (hereinafter referred to as the "specified period") 712. The second area 72 includes a waveform area 73, a probability area 74, and a beat area 75.

A common time axis is set for the waveform area 73, probability area 74, and beat point area 75. The waveform area 73 displays a waveform 731 within the specified period 712 of audio signal A and onsets 732 in audio signal A. The probability area 74 displays a time series 741 of the probability P[m] represented by the output data O[m] at each analysis time point t[m]. Note that the time series 741 of the probability P[m] represented by the output data O[m] may be displayed in the waveform area 73, superimposed on the waveform 731 of audio signal A.

The beat point area 75 displays multiple beat points in the music piece estimated by analyzing the audio signal A. Specifically, the beat point area 75 displays a time series of multiple beat images 751 corresponding to different beat points in the music piece. Of the multiple beat points in the music piece, beat images 751 corresponding to one or more beat points (hereinafter referred to as "candidate correction points") that satisfy predetermined conditions are highlighted in a display mode separate from the other beat images 751. Candidate correction points are beat points that the user is likely to request a change to.

The playback control unit 25 in FIG. 2 controls the playback of sound by the sound emitting device 15. Specifically, the playback control unit 25 causes the sound emitting device 15 to play the performance sound represented by sound signal A. In parallel with the playback of sound signal A, the playback control unit 25 plays a predetermined notification sound at a time point corresponding to each of the multiple beat points. Furthermore, the display control unit 24 highlights one beat image 751 of the multiple beat images 751 in the beat point area 75 that corresponds to the time point being played by the sound emitting device 15, in a display mode separate from the other beat images 751 in the beat point area 75. In other words, in parallel with the playback of sound signal A, each of the multiple beat images 751 is highlighted sequentially in chronological order.

However, in the process of estimating multiple beats in a piece of music from audio signal A, there is a possibility that, for example, a backbeat of the music may be erroneously estimated as a beat. Furthermore, the estimated beat may not match the user's intention, such as when a backbeat of the music is estimated when the user is expecting a downbeat to be estimated. By operating the operation device 14, the user can instruct a change in the position on the time axis of any of the multiple beats in the music. Specifically, the user moves one of the multiple beat images 751 in the beat area 75 along the time axis to instruct a change in the position of the beat corresponding to that beat image 751. For example, the user instructs a change in the position of a correction candidate point among the multiple beats.

The instruction receiving unit 26 in FIG. 2 receives an instruction from the user to change the position of some of the multiple beats in a piece of music (hereinafter referred to as a "change instruction"). In the following explanation, we will assume that the instruction receiving unit 26 receives a change instruction to move one beat on the time axis from analysis time t[m1] to analysis time t[m2] (m1, m2 = 1 to M, m1 ≠ m2). Analysis time t[m1] is the beat initially estimated by the analysis processing unit 20 (i.e., the beat before the change due to the change instruction), and analysis time t[m2] is the beat after the change due to the change instruction from the user.

The estimation model update unit 27 in Figure 2 updates the estimation model 50 in response to a change instruction from the user. Specifically, the estimation model update unit 27 updates the estimation model 50 so that the change in beat point associated with the change instruction is reflected in the estimation of multiple beat points throughout the entire piece of music.

Figure 11 is an explanatory diagram of the process Sc (hereinafter referred to as the "estimation model update process") in which the estimation model update unit 27 updates the estimation model 50. The estimation model update process Sc is a process (additional learning) in which the estimation model 50 trained by the machine learning system 200 is updated to reflect change instructions from the user.

In the estimation model update process Sc, an adaptation block 55 is added between the first part 50a and the second part 50b of the estimation model 50. The adaptation block 55 is configured, for example, with an attention block whose activation function is initialized to an identity function. Therefore, the initial adaptation block 55 supplies the intermediate data D[m] output from the first part 50a to the second part 50b without any changes.

The estimation model update unit 27 sequentially inputs feature data F[m1] for analysis time t[m1], where the pre-change beat point is located, and feature data F[m2] for analysis time t[m2], where the post-change beat point is located, to the first portion 50a (input layer 51). The first portion 50a generates intermediate data D[m1] corresponding to the feature data F[m1] and intermediate data D[m2] corresponding to the feature data F[m2]. The intermediate data D[m1] and intermediate data D[m2] are sequentially input to the adaptation block 55.

The estimation model update unit 27 also sequentially supplies each of the M pieces of intermediate data D[1] to D[M] calculated in the immediately preceding probability calculation process Sa (Sa2) to the adaptation block 55. That is, intermediate data D[m] (D[m1], D[m2]) corresponding to a portion of the analysis time point t[m] related to the change instruction among the M analysis time points t[1] to t[M] in the song, and each of the M pieces of intermediate data D[1] to D[M] spanning the entire song are input to the adaptation block 55. The adaptation block 55 calculates the similarity between the intermediate data D[m] (D[m1], D[m2]) corresponding to the analysis time point t[m] related to the change instruction and the intermediate data D[m] supplied from the estimation model update unit 27.

As mentioned above, analysis time t[m2] was estimated not to be a beat in the previous probability calculation process Sa, but was designated as a beat in the change instruction. In other words, the probability P[m2] represented by the output data O[m2] at analysis time t[m2] was set to a small value in the previous probability calculation process Sa, but should be set to a value close to 1 under the user's change instruction. Furthermore, not only for analysis time t[m2], but also for each of the M analysis times t[1] through t[M] in the song where intermediate data D[m] similar to intermediate data D[m2] at analysis time t[m2] is observed, the probability P[m] represented by the output data O[m] at that analysis time t[m] should be set to a value close to 1. Therefore, when the similarity between intermediate data D[m] and intermediate data D[m2] exceeds a predetermined threshold, the estimation model update unit 27 updates multiple variables of the estimation model 50 so that the probability P[m] of output data O[m] approaches a sufficiently large value (e.g., 1). Specifically, the estimation model update unit 27 updates the coefficients defining each of the first portion 50a, the adaptive block 55, and the second portion 50b so as to reduce the error between the probability P[m] of output data O[m] generated by the estimation model 50 from each piece of intermediate data D[m] whose similarity with intermediate data D[m2] exceeds the threshold and the value indicating a beat (i.e., 1).

On the other hand, analysis time t[m1] is a time point that was estimated to correspond to a beat in the immediately preceding probability calculation process Sa, but was designated as not corresponding to a beat in the change instruction. In other words, the probability P[m1] represented by the output data O[m1] at analysis time t[m1] was set to a large value in the immediately preceding probability calculation process Sa, but should be set to a value close to 0 under the user's change instruction. Furthermore, not only for analysis time t[m1], but also for each of the M analysis times t[1] through t[M] in the song where intermediate data D[m] similar to intermediate data D[m1] at analysis time t[m1] is observed, the probability P[m] represented by the output data O[m] at that analysis time t[m] should be set to a value close to 0. Therefore, when the similarity between intermediate data D[m] and intermediate data D[m1] exceeds a predetermined threshold, the estimation model update unit 27 updates multiple variables of the estimation model 50 so that the probability P[m] of output data O[m] approaches a sufficiently small value (e.g., 0). Specifically, the estimation model update unit 27 updates the coefficients defining each of the first portion 50a, the adaptive block 55, and the second portion 50b so as to reduce the error between the probability P[m] of output data O[m] generated by the estimation model 50 from each piece of intermediate data D[m] whose similarity with intermediate data D[m1] exceeds the threshold and the value indicating that the data does not correspond to a beat point (i.e., 0).

As can be understood from the above explanation, in the first embodiment, not only the intermediate data D[m1] and intermediate data D[m2] directly related to the change instruction are used to update the estimation model 50, but also intermediate data D[m] similar to intermediate data D[m1] or intermediate data D[m2] among the M pieces of intermediate data D[1] to D[M] throughout the entire song. Therefore, even though the beats at which the user instructs changes are only some of the beats within the song, after executing the estimation model update process Sc, the estimation model 50 can generate M pieces of output data O[1] to O[M] that reflect the change instructions throughout the entire song.

Figure 12 is a flowchart illustrating the specific steps of the estimation model update process Sc. The control device 11 executes the estimation model update process Sc by functioning as the estimation model update unit 27.

When the estimation model update process Sc starts, the estimation model update unit 27 determines whether an adaptive block 55 has already been added to the estimation model 50 (Sc1). If an adaptive block 55 has not been added to the estimation model 50 (Sc1: NO), the estimation model update unit 27 adds a new initial adaptive block 55 between the first portion 50a and the second portion 50b of the estimation model 50 (Sc2). On the other hand, if an adaptive block 55 has already been added in a previous estimation model update process Sc (Sc1: YES), the addition of the adaptive block 55 (Sc2) is not executed.

When a new adaptive block 55 is added, the estimation model 50 including the new adaptive block 55 is updated by the following process. When an adaptive block 55 has already been added, the estimation model 50 including the existing adaptive block 55 is updated by the following process. That is, when the adaptive block 55 has been added to the estimation model 50, the estimation model update unit 27 updates multiple variables of the estimation model 50 by performing additional learning (Sc3 and Sc4) that applies the beat positions before and after the change in response to a change instruction from the user. Note that when the user instructs changes to the positions of two or more beats, additional learning (Sc3 and Sc4) is performed for each beat related to the change instruction.

The estimation model update unit 27 updates multiple variables of the estimation model 50 using feature data F[m1] for analysis time t[m1], which corresponds to the beat point before the change due to the change instruction (Sc3). Specifically, while supplying feature data F[m1] to the estimation model 50, the estimation model update unit 27 sequentially supplies each of M intermediate data D[1] to D[M] to the adaptation block 55, and updates multiple variables of the estimation model 50 so that the probability P[m] of output data O[m] generated from each intermediate data D[m] similar to intermediate data D[m1] of feature data F[m1] approaches zero. Therefore, the estimation model 50 is trained to generate output data O[m] with a probability P[m] close to zero when feature data F[m] similar to feature data F[m1] for analysis time t[m1] is input.

The estimation model update unit 27 also updates multiple variables of the estimation model 50 using feature data F[m2] for analysis time t[m2], which corresponds to the beat position after the change in response to the change instruction (Sc4). Specifically, while supplying feature data F[m2] to the estimation model 50, the estimation model update unit 27 sequentially supplies each of M intermediate data D[1] to D[M] to the adaptation block 55, and updates multiple variables of the estimation model 50 so that the probability P[m] of output data O[m] generated from each intermediate data D[m] similar to intermediate data D[m2] of feature data F[m2] approaches 1. Therefore, the estimation model 50 is trained to generate output data O[m] with a probability P[m] close to 1 when feature data F[m] similar to feature data F[m2] for analysis time t[m2] is input.

In addition to updating the estimation model 50 in response to the change instruction by the estimation model update process Sc illustrated above, in the first embodiment, multiple updated beats are estimated by executing the beat estimation process Sb under constraints in response to the change instruction.

As mentioned above, of the five elapsed points Y[0] to Y[4] within the beat interval δ, elapsed point Y[0] corresponds to a beat, while the remaining four elapsed points Y[1] to Y[4] do not. Analysis time point t[m2] on the time axis corresponds to the beat after the change in response to the change instruction. Therefore, the estimation processing unit 23 forcibly sets to 0 the likelihoods λ[i,j] corresponding to elapsed points Y[j'] (j' = 1 to 4) other than elapsed point Y[0] among the N likelihoods λ[i,j] corresponding to different states Q at analysis time point t[m2]. Furthermore, the estimation processing unit 23 maintains the likelihood λ[i,0] corresponding to elapsed point Y[0] among the N likelihoods λ[i,j] at analysis time point t[m2] at the value calculated using the method described above. Therefore, in generating a state sequence (Sb3), a maximum likelihood path is estimated that always passes through state Q of passing point Y[0] at analysis time t[m2]. In other words, analysis time t[m2] is estimated to correspond to a beat point. As can be understood from the above explanation, the beat point estimation process Sb is executed under the constraint that state Q of passing point Y[0] is observed at analysis time t[m2] of the beat point after the change in response to a change instruction from the user.

On the other hand, analysis time t[m1] on the time axis does not correspond to a beat point after the change due to the change instruction. Therefore, the estimation processing unit 23 forcibly sets the likelihood λ[i,0] corresponding to the progress point Y[0] among the N likelihoods λ[i,j] corresponding to different states Q at analysis time t[m1] to 0. Furthermore, the estimation processing unit 23 maintains the likelihood λ[i,j'] corresponding to the progress point Y[j'] other than the progress point Y[0] among the N likelihoods λ[i,j] at analysis time t[m1] at a significant value calculated using the method described above. Therefore, in generating the state sequence (Sb3), the maximum likelihood path that does not pass through state Q at the progress point Y[0] at analysis time t[m1] is estimated. In other words, analysis time t[m1] is estimated not to correspond to a beat point. As can be understood from the above explanation, the beat point estimation process Sb is executed under the constraint that the state Q of the elapsed point Y[0] is not observed at the analysis time t[m1] before the change due to the change instruction from the user.

As described above, the likelihood λ[i,0] of the passing point Y[0] at analysis time t[m1] is set to 0, and the likelihood λ[i,j'] of the passing points Y[j'] other than Y[0] at analysis time t[m2] is set to 0, thereby changing the most likely path throughout the entire piece. In other words, even though the beats that the user requests to be changed are only a portion of the beats in the piece, the change request is reflected in multiple beats throughout the piece.

Figure 13 is a flowchart illustrating the specific steps of the process executed by the control device 11. For example, the process of Figure 13 is started in response to an instruction from the user via the operation device 14. When the process starts, the control device 11 executes a process (hereinafter referred to as "initial analysis process") to estimate multiple beats in the music by analyzing the audio signal A (S1).

Figure 14 is a flowchart illustrating the specific steps of the initial analysis process. When the initial analysis process begins, the control device 11 (feature extraction unit 21) generates feature data F[m] for each of M analysis points t[1] to t[M] on the time axis (S11). As described above, the feature data F[m] is a time series of multiple feature amounts f[m] within a unit period U that includes the analysis point t[m].

The control device 11 (probability calculation unit 22) executes the probability calculation process Sa illustrated in FIG. 6 to generate M pieces of output data O[m] corresponding to different analysis times t[m] (S12). Furthermore, the control device 11 (estimation processing unit 23) executes the beat point estimation process Sb illustrated in FIG. 9 to estimate multiple beat points within the piece of music (S13).

The control device 11 (display control unit 24) identifies one or more correction candidate points from among the multiple beat points estimated by the beat point estimation process Sb (S14). Specifically, a beat point where the beat interval δ between the previous or next beat point deviates from the average value in the music piece, or a beat point where the duration of the beat interval δ is significantly different from the δ between the previous and next beat intervals, is identified as a correction candidate point. Furthermore, a beat point where the probability P[m] of the multiple beat points is below a predetermined value may be identified as a correction candidate point. The control device 11 (display control unit 24) causes the display device 13 to display the analysis screen 70 shown in FIG. 10 (S15).

After executing the initial analysis process exemplified above, the control device 11 (instruction receiving unit 26) waits until it receives an instruction from the user to change some of the beats in the music, as illustrated in FIG. 13 (S2: NO). If an instruction to change is received (S2: YES), the control device 11 (estimation model update unit 27 and analysis processing unit 20) executes a beat point update process (S3) to update the positions of the multiple beats estimated in the initial analysis process in accordance with the change instruction from the user.

Figure 15 is a flowchart illustrating the specific steps of the beat point update process. The control device 11 (estimation model update unit 27) executes the estimation model update process Sc illustrated in Figure 12 to update multiple variables of the estimation model 50 in response to change instructions from the user (S31).

The control device 11 (probability calculation unit 22) generates M pieces of output data O[1] to O[M] by executing the probability calculation process Sa of FIG. 6 using the estimation model 50 updated by the estimation model update process Sc (S32). The control device 11 (analysis processing unit 20) also generates beat point data B by executing the beat point estimation process Sb of FIG. 9 using the M pieces of output data O[1] to Q[M] (S33). In other words, multiple beat points within the music are estimated. The beat point estimation process Sb within the beat point update process is executed under the aforementioned constraints in accordance with the change instructions.

As can be understood from the above explanation, multiple updated beat positions are estimated by the estimation model update process Sc, which updates the estimation model 50, the probability calculation process Sa, which uses the updated estimation model 50, and the beat position estimation process Sb, which uses the output data O[m] generated by the probability calculation process Sa. In other words, the estimation model update unit 27, the probability calculation unit 22, and the analysis processing unit 20 implement an element (beat position update unit) that updates the positions of multiple estimated beat positions.

Similar to step S14 described above, the control device 11 (display control unit 24) identifies one or more correction candidate points from among the multiple beat points estimated by the beat point estimation process Sb (S34). The control device 11 (display control unit 24) causes the display device 13 to display the analysis screen 70 of FIG. 10, which includes beat images 751 representing each updated beat point (S35).

When the beat point update process illustrated above is executed, the control device 11 determines whether the user has instructed the end of the process (S4), as illustrated in FIG. 13. If the user has not instructed the end of the process (S4: NO), the control device 11 waits for a change instruction from the user (S2). The control device 11 executes the beat point update process in response to another change instruction from the user (S3). In the estimation model update process Sc (S31) of the second or subsequent beat point update process, the result of the determination of the presence or absence of an adaptive block 55 (Sc1) is positive, so a new adaptive block 55 is not added. In other words, the estimation model 50 to which the adaptive block 55 was added in the first beat point update process is cumulatively updated with each subsequent execution of the estimation model update process Sc. On the other hand, if the end of the process is instructed (S4: YES), the control device 11 terminates the process of FIG. 13.

As explained above, in the first embodiment, in response to a user instruction to change some of the multiple beats estimated by analyzing audio signal A, the positions of multiple beats in the song, including those other than those beats, are updated. In other words, an instruction to change a portion of the song is reflected in the entire song. Therefore, compared to a configuration in which the user must instruct a change in the position of each beat in the song, it is possible to obtain a time series of beats that matches the user's intentions while reducing the burden on the user of instructing a change in the position of each beat.

When an adaptation block 55 is added between the first part 50a and the second part 50b in the estimation model 50, the estimation model 50 is updated through additional learning that applies the beat positions before and after the change in response to a change instruction from the user. Therefore, the estimation model 50 can be specialized to a state where it can estimate beat positions that are in line with the user's intentions or preferences.

Furthermore, multiple beat points are estimated using a state transition model 60 composed of multiple states Q corresponding to any of multiple tempos X[i]. Therefore, multiple beat points can be estimated so that the tempo X[i] transitions naturally. In the first embodiment, in particular, the multiple states Q of the state transition model 60 correspond to different combinations of each of the multiple tempos X[i] and each of the multiple elapsed points Y[j] within the beat interval δ, and the beat point estimation process Sb is executed under the constraint that the state Q corresponding to the elapsed point Y[0] is observed at the analysis time t[m] of the beat point after the change due to a change instruction from the user. Therefore, multiple beat points can be estimated, including the time after the change due to a change instruction from the user as the beat point.

B: Second Embodiment A second embodiment will now be described. Note that, in the following examples, elements that have the same functions as those in the first embodiment will be denoted by the same reference numerals as those used in the description of the first embodiment, and detailed descriptions of each element will be omitted as appropriate.

Figure 16 is a block diagram illustrating the functional configuration of an acoustic analysis system 100 in the second embodiment. The control device 11 in the second embodiment functions as a curve setting unit 28 in addition to the same elements as in the first embodiment (analysis processing unit 20, display control unit 24, playback control unit 25, instruction receiving unit 26, and estimation model update unit 27).

In the second embodiment, the analysis processing unit 20 estimates the tempo T[m] of a piece of music in addition to estimating multiple beats within the piece of music. That is, the analysis processing unit 20 analyzes the audio signal A to estimate a time series of M tempos T[1] to T[M] corresponding to different analysis points t[m] on the time axis.

Figure 17 is a schematic diagram of the analysis screen 70 in the second embodiment. In addition to the same elements as in the first embodiment, the analysis screen 70 in the second embodiment includes an estimated tempo curve CT, a maximum tempo curve CH, and a minimum tempo curve CL. Specifically, the waveform area 73 of the analysis screen 70 displays the waveform 731 of audio signal A, the estimated tempo curve CT, the maximum tempo curve CH, and the minimum tempo curve CL on a common time axis. Note that in Figure 17, the display of the onset points 732 in audio signal A has been omitted for convenience.

Figure 18 is a schematic diagram focusing on the estimated tempo curve CT, maximum tempo curve CH, and minimum tempo curve CL. The estimated tempo curve CT is a curve representing the time series of the tempo T[m] estimated by the analysis processing unit 20. The maximum tempo curve CH is a curve representing the time change in the maximum value H[m] of the tempo T[m] estimated by the analysis processing unit 20 (hereinafter referred to as the "maximum tempo"). In other words, the maximum tempo curve CH represents the time series of M maximum tempos H[1] to H[M] corresponding to different analysis times t[m] on the time axis. The minimum tempo curve CL is a curve representing the time change in the minimum value L[m] of the tempo T[m] estimated by the analysis processing unit 20 (hereinafter referred to as the "minimum tempo"). In other words, the minimum tempo curve CL represents the time series of M minimum tempos L[1] to L[M] corresponding to different analysis times t[m] on the time axis.

As can be understood from the above explanation, the analysis processing unit 20 estimates the tempo T[m] of a piece of music for each analysis time point t[m] within the range R[m] between the maximum tempo H[m] and the minimum tempo L[m] (hereinafter referred to as the "limited range"). Therefore, the estimated tempo curve CT is located between the maximum tempo curve CH and the minimum tempo curve CL. The position and width of the limited range R[m] change over time.

The curve setting unit 28 in FIG. 16 sets the maximum tempo curve CH and the minimum tempo curve CL. For example, the user can specify a maximum tempo curve CH and a minimum tempo curve CL of a desired shape by operating the operation device 14. The curve setting unit 28 sets the maximum tempo curve CH and the minimum tempo curve CL in response to a user's instruction on the analysis screen 70 (waveform area 73). For example, the curve setting unit 28 sets a continuous curve that passes through multiple points specified by the user in time series within the waveform area 73 as the maximum tempo curve CH or the minimum tempo curve. Furthermore, the user can specify in the waveform area 73 to change the previously set maximum tempo curve CH and minimum tempo curve CL by operating the operation device 14. The curve setting unit 28 changes the maximum tempo curve CH and the minimum tempo curve CL in response to a user's instruction on the analysis image (waveform area 73). As can be understood from the above explanation, according to the second embodiment, the user can easily change the maximum tempo curve CH and minimum tempo curve CL while checking the analysis screen 70.

In the second embodiment, the waveform 731 of audio signal A and the maximum tempo curve CH and minimum tempo curve CL are displayed on a common time axis, making it easy for the user to visually grasp the relationship between the change over time in the maximum tempo H[m] or minimum tempo L[m] and the waveform 731 of audio signal A. Furthermore, because the estimated tempo curve CT is displayed along with the maximum tempo curve CH and minimum tempo curve CL, the user can visually grasp the change over time in the tempo T[m] of the song estimated between the maximum tempo curve CH and minimum tempo curve CL.

Figure 19 is a flowchart illustrating the specific steps of the beat point estimation process Sb in the second embodiment. After setting the observation likelihood Λ[m] for each analysis time t[m] in the same manner as in the first embodiment (Sb1), the estimation processing unit 23 calculates the path p[i,j] and likelihood λ[i,j] for each state Q[i,j] of the state transition model 60 for each analysis time t[m] (Sb2). In the second embodiment, the estimation processing unit 23 sets to zero the likelihood λ[i,j] corresponding to each tempo X[i] that exceeds the maximum tempo H[m] among the multiple tempos X[i] and the likelihood λ[i,j] corresponding to each tempo X[i] that falls below the minimum tempo L[m] for each analysis time t[m]. In other words, among the N states Q of the state transition model 60, states Q corresponding to tempos X[i] outside the limit range R[m] are set to an invalid state. Furthermore, for each analysis time point t[m], the estimation processing unit 23 sets the likelihood λ[i,j] corresponding to each tempo X[i] inside the limit range R[m] to a significant value, as in the first embodiment. That is, of the N states Q of the state transition model 60, the state Q corresponding to the tempo X[i] inside the limit range R[m] is set to a valid state.

The estimation processing unit 23 generates a state sequence using a method similar to that of the first embodiment (Sb3). That is, a sequence in which the states Q with the largest likelihood λ[i,j] among the N states Q are arranged for each analysis time t[m] is generated as the state sequence. As described above, the likelihood λ[i,j] of the state Q[i,j] corresponding to the tempo X[i] outside the restricted range R[m] at the analysis time t[m] is set to 0. Therefore, the state Q corresponding to the tempo X[i] outside the restricted range R[m] is not selected as an element of the state sequence. As can be understood from the above explanation, the invalid state of each state Q means that the state Q is not selected.

The estimation processing unit 23 generates beat data B (Sb4) as in the first embodiment, and identifies the tempo T[m] at each analysis time point t[m] from the state series (Sb5). That is, the tempo X[i] of the state Q corresponding to the analysis time point t[m] in the state series is set as the tempo T[m]. As mentioned above, a state Q corresponding to a tempo X[i] outside the limit range R[m] is not selected as an element of the state series, so the tempo T[m] is limited to a value within the limit range R[m].

As explained above, in the second embodiment, the maximum tempo curve CH and the minimum tempo curve CL are set in response to instructions from the user. The tempo T[m] of the music is then estimated within the limited range R[m] between the maximum tempo H[m] represented by the maximum tempo curve CH and the minimum tempo L[m] represented by the minimum tempo curve CL. This reduces the likelihood of estimating a tempo that deviates excessively from the tempo intended by the user (for example, a tempo that is twice or half the value intended by the user). In other words, the tempo T[m] of the music represented by the audio signal A can be estimated with high accuracy.

In the second embodiment, a state transition model 60 consisting of multiple states Q corresponding to any of multiple tempos X[i] is used to estimate multiple beats. Therefore, a tempo T[m] that transitions naturally over time is estimated. Furthermore, a tempo T[m] restricted to within the restricted range R[m] can be estimated by a simple process of invalidating the state Q corresponding to a tempo X[i] outside the restricted range R[m].

C: Third Embodiment In the first embodiment, the output data O[m] representing the probability P[m] calculated by the probability calculation unit 22 using the estimation model 50 is applied to the beat position estimation process Sb by the estimation processing unit 23. In the third embodiment, the probability P[m] calculated by the estimation model 50 (hereinafter referred to as "probability P1[m]") is adjusted in response to an operation by the user via the operation device 14, and the output data O[m] representing the adjusted probability P2[m] is applied to the beat position estimation process Sb.

Figure 20 is an explanatory diagram of the process by which the probability calculation unit 22 of the third embodiment generates output data O[m]. While listening to the musical piece played by the sound emitting device 15 by the playback control unit 25, the user operates the operation device 14 at each time point that the user recognizes as a beat. For example, while the musical piece is being played, the user taps on the touch panel of the operation device 14 at each time point that the user recognizes as a beat. Figure 20 illustrates the time point τ at which the user operates (hereinafter referred to as the "operation time point") on the time axis.

The probability calculation unit 22 sets a unit distribution W for each operation time τ. The unit distribution W is a distribution of weighted values w[m] on the time axis. For example, a probability distribution such as a normal distribution with a predetermined variance is used as the unit distribution W. In each unit distribution W, the weighted value w[m] is maximum at the operation time τ, and decreases as the distance from the operation time τ increases.

The probability calculation unit 22 calculates the adjusted probability P2[m] by multiplying the probability P1[m] generated by the estimation model 50 for the analysis time t[m] by the weighted value w[m] at that analysis time t[m]. Therefore, even if the probability P1[m] generated by the estimation model 50 is small for an analysis time t[m], if the analysis time t[m] is close to the operation time τ, the adjusted probability P2[m] is set to a large value. The probability calculation unit 22 supplies output data O[m] representing the adjusted probability P2[m] to the estimation processing unit 23. The procedure of the beat point estimation process Sb, in which the estimation processing unit 23 estimates multiple beat points using the output data O[m], is the same as in the first embodiment.

The third embodiment also achieves the same effects as the first embodiment. Furthermore, in the third embodiment, the weighted value w[m] of the unit distribution W set at the time τ of the user's operation is multiplied by the probability P1[m], which has the advantage of being able to estimate a beat point that fully reflects the user's intentions or preferences. The configuration of the second embodiment can also be applied to the third embodiment.

D: Modifications Specific modifications that can be added to the above-mentioned embodiments are exemplified below. Two or more embodiments arbitrarily selected from the following examples may be combined as appropriate within the scope of not mutually contradicting each other.

(1) The configuration of the estimation model 50 is not limited to the example shown in Figure 4. For example, the estimation model 50 may include a recurrent neural network. Furthermore, additional elements such as long short-term memory (LSTM) may be incorporated into the estimation model 50. The estimation model 50 may also be configured by combining multiple types of deep neural networks.

(2) The specific steps of the process for estimating multiple beats in a piece of music by analyzing the audio signal A are not limited to the examples given in the above-mentioned embodiments. For example, the analysis processing unit 20 may estimate the analysis time t[m] at which the probability P[m] represented by the output data O[m] is maximized as a beat. In other words, the use of the state transition model 60 is omitted. Alternatively, the analysis processing unit 20 may estimate the time at which a feature f[m], such as the volume of the audio signal A, significantly increases as a beat. In other words, the use of the estimation model 50 is omitted.

(3) The configuration of the first embodiment, which updates multiple beat points estimated by the initial analysis process, may be omitted in the second embodiment. In other words, the configuration of the first embodiment, which updates multiple beat points throughout the entire musical piece in response to an instruction to change some of the multiple estimated beat points, and the configuration of the second embodiment, which estimates the tempo T[m] of the musical piece within a limited range R[m] in response to an instruction from the user, may be established independently of each other.

(4) The acoustic analysis system 100 may be realized by a server device that communicates with an information device such as a smartphone or tablet terminal. For example, the acoustic analysis system 100 generates beat data B by analyzing an acoustic signal A received from the information device and transmits the beat data B to the information device. Similarly, the reception of a change instruction from the user (S2) and the beat update process (S3) are also performed by the acoustic analysis system 100 that communicates with the information device.

(5) As mentioned above, the functions of the acoustic analysis system 100 exemplified above are realized through cooperation between one or more processors constituting the control device 11 and a program stored in the storage device 12. The program according to the present disclosure can be provided in a form stored on a computer-readable recording medium and installed on a computer. The recording medium is, for example, a non-transitory recording medium, and a good example is an optical recording medium (optical disk) such as a CD-ROM, but also includes any known type of recording medium, such as a semiconductor recording medium or a magnetic recording medium. Note that a non-transitory recording medium includes any recording medium excluding a transient, propagating signal, and does not exclude volatile recording media. Furthermore, in a configuration in which a distribution device distributes a program via a communication network, the storage device 12 that stores the program in the distribution device corresponds to the non-transitory recording medium described above.

E: Supplementary Note From the above-described exemplary embodiments, the following configurations can be understood, for example.

An acoustic analysis method according to one aspect (aspect 1) of the present disclosure estimates multiple beats of a piece of music by analyzing an audio signal representing the music being played, receives instructions from a user to change the positions of some of the multiple beats, and updates the positions of the multiple beats in accordance with the user's instructions. In the above aspect, in accordance with an instruction to change the positions of some of the multiple beats estimated by analyzing the audio signal, the positions of the multiple beats, including those other than the some of the multiple beats, are updated. Therefore, compared to a configuration in which the user must change the positions of all of the multiple beats, it is possible to obtain a time series of beats that matches the user's intentions, while reducing the burden on the user of instructing changes to the positions of each beat.

In a specific example (Aspect 2) of Aspect 1, the beat point estimation includes a feature extraction process that generates feature data including features of the audio signal for each of multiple analysis time points on the time axis; a probability calculation process that generates output data indicating the probability that the analysis time point corresponds to a beat point by inputting the feature data generated for each analysis time point by the feature extraction process into an estimation model that has learned the relationship between the training feature data corresponding to the time point on the time axis and training output data indicating the probability that the time point corresponds to a beat point; and a beat point estimation process that estimates the multiple beat points from the output data generated by the probability calculation process. According to the above aspect, it is possible to generate output data that is statistically valid for unknown feature data based on the underlying relationship between the training feature data and the training output data.

In a specific example of Aspect 2 (Aspect 3), the multiple beat positions are updated by adding an adaptive block between the first input portion and the second output portion of the estimation model, performing additional training that applies the beat positions before or after the change in response to the user's instruction, and then updating the estimation model. The updated multiple beat positions are then estimated by the probability calculation process using the updated estimation model and the beat estimation process using the output data generated by the probability calculation process. According to the above aspect, the estimation model is updated by additional training that applies the beat positions before or after the change in response to the user's instruction. This allows the estimation model to be specialized to estimate beat positions that are in line with the user's intentions or preferences.

The adaptive block is a block that generates the similarity between first intermediate data generated by the first portion from feature data corresponding to the beat position before or after the change in response to a user instruction, and second intermediate data corresponding to the feature data at each of multiple analysis points in the music. The entire estimation model, including the adaptive block, is updated so that the output data at the analysis point corresponding to the second intermediate data that is similar to the first intermediate data for the beat position before the change in response to a user instruction approaches a numerical value indicating that the output data does not correspond to a beat, and so that the output data at the analysis point corresponding to the second intermediate data that is similar to the first intermediate data for the beat position after the change approaches a numerical value indicating that the output data corresponds to a beat.

In a specific example (Aspect 4) of Aspect 2 or Aspect 3, the beat estimation process estimates the multiple beats using a state transition model composed of multiple states corresponding to any of multiple tempos. According to the above aspect, multiple beats are estimated using a state transition model composed of multiple states corresponding to any of multiple tempos. Therefore, multiple beats are estimated so that the tempo transitions naturally over time.

In a specific example (Aspect 5) of Aspect 4, the multiple states of the state transition model correspond to different combinations of each of the multiple tempos and each of multiple elapsed points within a beat interval, and in the beat point estimation process, a time point at which a state of the multiple elapsed points corresponding to an end point of the beat interval is observed is estimated as a beat point, and in updating the positions of the multiple beat points, the beat point estimation process is performed under the constraint that a state corresponding to an end point of the beat interval is observed at the time point of the beat point after the change based on the user's instruction, thereby estimating the multiple updated beat points. According to the above aspect, multiple beat points can be estimated, including the beat point after the change based on the user's instruction.

An acoustic analysis system according to one aspect (aspect 6) of the present disclosure includes an analysis processing unit that estimates multiple beats of a piece of music by analyzing an acoustic signal representing the sound of the piece of music being played, an instruction receiving unit that receives instructions from a user to change the positions of some of the multiple beats, and a beat updating unit that updates the positions of the multiple beats in response to instructions from the user.

A program according to one aspect (aspect 7) of the present disclosure causes a computer system to function as an analysis processing unit that estimates multiple beats of a piece of music by analyzing an audio signal representing the music being played, an instruction receiving unit that receives instructions from a user to change the positions of some of the multiple beats, and a beat updating unit that updates the positions of the multiple beats in response to instructions from the user.

In this specification, "tempo" refers to any numerical value that represents the speed of a performance, and is not limited to tempo in the narrow sense of the number of beats per minute (BPM).

100...acoustic analysis system, 11...control device, 12...storage device, 13...display device, 14...operation device, 15...sound emission device, 20...analysis processing unit, 21...feature extraction unit, 22...probability calculation unit, 23...estimation processing unit, 24...display control unit, 25...playback control unit, 26...instruction reception unit, 27...estimation model update unit, 28...curve setting unit, 50...estimation model, 50a...first part, 50b...second part, 51...input layer, 52 (52a, 52b)...intermediate layer, 53...output layer, 55...adaptation block, 59...tentative model, 60...state transition model.

Claims (9)

estimating positions of a plurality of beats in the piece of music by analyzing the audio signal, which includes inputting feature data including feature amounts of the audio signal representing the performance sound of the piece of music into an estimation model that has learned the relationship between feature amounts of the training audio signal and beats in the training audio signal;
receiving an instruction from a user to change the positions of some of the beats;
updating the positions of the plurality of beats in response to an instruction from the user .
In updating the positions of the plurality of beat points,
The estimation model is updated in response to an instruction from the user, and the positions of the plurality of beats are updated by performing an analysis of the acoustic signal, including a process of inputting the feature data into the updated estimation model.
Acoustic analysis method. estimating positions of a plurality of beats in the piece of music by analyzing the audio signal, which includes inputting feature data including feature amounts of the audio signal representing the performance sound of the piece of music into an estimation model that has learned the relationship between feature amounts of the training audio signal and beats in the training audio signal;
receiving an instruction from a user to change the positions of some of the beats;
updating the positions of the plurality of beats in response to an instruction from the user .
the estimation model is a model that has learned a relationship between training feature data including features corresponding to time points on a time axis in the training acoustic signal and training output data that indicates a probability that the time points correspond to beat points;
The estimation of the beat point is
a feature extraction process for generating the feature data for each of a plurality of analysis time points on a time axis;
a probability calculation process for generating output data representing the probability that each analysis time point corresponds to a beat by inputting the feature data generated for each analysis time point by the feature extraction process into the estimation model;
a beat point estimation process for estimating the plurality of beat points from output data generated by the probability calculation process.
Acoustic analysis method. In updating the positions of the plurality of beat points,
updating the estimation model by performing additional learning that applies beat positions before or after the change in accordance with the instruction from the user, in a state in which an adaptation block is added between a first portion on the input side and a second portion on the output side of the estimation model;
The acoustic analysis method according to claim 2, wherein the updated beat positions are estimated by the probability calculation process using the updated estimation model and the beat position estimation process using output data generated by the probability calculation process. 4. The acoustic analysis method according to claim 2 , wherein the beat estimation process estimates the plurality of beats using a state transition model configured with a plurality of states corresponding to any of a plurality of tempos. the plurality of states of the state transition model correspond to different combinations of each of the plurality of tempos and each of a plurality of passing points within a beat interval;
In the beat point estimation process, a time point at which a state corresponding to an end point of the beat interval is observed among the plurality of elapsed points is estimated as a beat point;
In updating the positions of the plurality of beat points,
The acoustic analysis method according to claim 4, wherein the updated beat points are estimated by executing the beat point estimation process under a constraint that a state corresponding to an end point of the beat interval is observed at the time of the beat point after the change in response to the user's instruction. an analysis processing unit that estimates positions of a plurality of beats in the music piece by analyzing the audio signal, the analysis processing including a process of inputting feature data including feature amounts of the audio signal representing the performance sound of the music piece into an estimation model that has learned the relationship between feature amounts of the training audio signal and beats in the training audio signal;
an instruction receiving unit that receives an instruction from a user to change the positions of some of the plurality of beats;
a beat point updating unit that updates the positions of the plurality of beat points in response to an instruction from the user ,
The beat point updating unit
The estimation model is updated in response to an instruction from the user, and the positions of the plurality of beats are updated by performing an analysis of the acoustic signal, including a process of inputting the feature data into the updated estimation model.
Acoustic analysis system. an analysis processing unit that estimates positions of a plurality of beats in the music piece by analyzing the audio signal, the analysis processing including a process of inputting feature data including feature amounts of the audio signal representing the performance sound of the music piece into an estimation model that has learned the relationship between feature amounts of the training audio signal and beats in the training audio signal;
an instruction receiving unit that receives an instruction from a user to change the positions of some of the plurality of beats;
a beat point updating unit that updates the positions of the plurality of beat points in response to an instruction from the user ,
the estimation model is a model that has learned a relationship between training feature data including features corresponding to time points on a time axis in the training acoustic signal and training output data that indicates a probability that the time points correspond to beat points;
The analysis processing unit
a feature extraction process for generating the feature data for each of a plurality of analysis time points on a time axis;
a probability calculation process for generating output data representing the probability that each analysis time point corresponds to a beat by inputting the feature data generated for each analysis time point by the feature extraction process into the estimation model;
and executing a beat point estimation process for estimating the plurality of beat points from the output data generated by the probability calculation process.
Acoustic analysis system. an analysis processing unit that estimates positions of a plurality of beats in the music piece by analyzing the audio signal, the analysis processing including a process of inputting feature data including feature amounts of the audio signal representing the performance sound of the music piece into an estimation model that has learned the relationship between feature amounts of the training audio signal and beats in the training audio signal;
an instruction receiving unit that receives an instruction from a user to change the positions of some of the beats; and
a program that causes a computer system to function as a beat point updating unit that updates positions of the plurality of beat points in response to an instruction from the user,
The beat point updating unit
The estimation model is updated in response to an instruction from the user, and the positions of the plurality of beats are updated by performing an analysis of the acoustic signal, including a process of inputting the feature data into the updated estimation model.
program. an analysis processing unit that estimates positions of a plurality of beats in the music piece by analyzing the audio signal, the analysis processing including a process of inputting feature data including feature amounts of the audio signal representing the performance sound of the music piece into an estimation model that has learned the relationship between feature amounts of the training audio signal and beats in the training audio signal;
an instruction receiving unit that receives an instruction from a user to change the positions of some of the beats; and
a program that causes a computer system to function as a beat point updating unit that updates positions of the plurality of beat points in response to an instruction from the user,
the estimation model is a model that has learned a relationship between training feature data including features corresponding to time points on a time axis in the training acoustic signal and training output data that indicates a probability that the time points correspond to beat points;
The analysis processing unit
a feature extraction process for generating the feature data for each of a plurality of analysis time points on a time axis;
a probability calculation process for generating output data representing the probability that each analysis time point corresponds to a beat by inputting the feature data generated for each analysis time point by the feature extraction process into the estimation model;
and executing a beat point estimation process for estimating the plurality of beat points from the output data generated by the probability calculation process.
program.