JP7772056B2 - Chord estimation device, training device, chord estimation method, and training method - Google Patents

Description

The present invention relates to a chord estimation device and method for estimating chords for playing a musical instrument, and a training device and method for building a chord estimation device.

There are musical scores with chords added. Performers can enjoy playing musical instruments by playing the chords on instruments such as piano and guitar. When creating musical scores with chords, the creator adds chords based on the melody and accompaniment indicated by the notes. The task of adding chords requires musical knowledge and sense. Patent Document 1 listed below discloses a chord progression estimation and detection device that estimates chords from performance information or audio signals.

Patent No. 6151121

In Patent Document 1, chords are estimated for each specific section. For example, one chord is estimated for each measure. If chord estimation could be performed with greater flexibility from given notes, it is expected that more appropriate support could be provided for the creation of sheet music with chords.

The object of the present invention is to perform highly flexible chord estimation based on a sequence of notes.

A chord estimation device according to one aspect of the present invention includes a receiving unit that receives time series data including a note sequence consisting of a plurality of notes, and an estimation unit that uses a trained model to estimate chord sequence information indicating a chord sequence corresponding to the note sequence based on the time series data.

A training device according to another aspect of the present invention comprises a first acquisition unit that acquires input time series data including a reference note sequence consisting of a plurality of notes, a second acquisition unit that acquires output chord sequence information indicating a chord sequence corresponding to the reference note sequence, and a construction unit that constructs a trained model that has acquired the input/output relationship between the input time series data and the output chord sequence information.

A chord estimation method according to yet another aspect of the present invention is executed by a computer, accepts time series data including a sequence of notes consisting of a plurality of notes, and uses a trained model to estimate chord sequence information indicating a chord sequence corresponding to the sequence of notes based on the time series data.

A training method according to yet another aspect of the present invention is executed by a computer, acquires input time series data including a reference note sequence consisting of a plurality of notes, acquires output chord sequence information indicating a chord sequence corresponding to the reference note sequence, and constructs a trained model that has acquired the input/output relationship between the input time series data and the output chord sequence information.

According to the present invention, highly flexible chord estimation can be performed based on a sequence of notes.

FIG. 1 is a block diagram showing the configuration of a processing system including a chord estimation device and a training device according to an embodiment of the present invention. FIG. 2 is a diagram showing an example of input time series data included in the training data. FIG. 3 is a diagram showing an example of output code string information included in the training data. FIG. 4 is a block diagram showing the configuration of the training device and the chord estimation device. FIG. 5 shows an example of an arranged musical score displayed on the display unit. FIG. 6 is a flowchart showing an example of the training process. FIG. 7 is a flowchart showing an example of the chord estimation process. FIG. 8 shows a modified example of the output code string information included in the training data.

(1) Configuration of the Processing System A chord estimation device, training device, chord estimation method, and training method according to embodiments of the present invention will now be described in detail with reference to the drawings. Fig. 1 is a block diagram showing the configuration of a processing system including a chord estimation device and training device according to one embodiment of the present invention. As shown in Fig. 1, the processing system 100 includes a RAM (random access memory) 110, a ROM (read only memory) 120, a CPU (central processing unit) 130, a storage unit 140, an operation unit 150, and a display unit 160.

The processing system 100 is realized by a computer such as a personal computer, tablet terminal, or smartphone. Alternatively, the processing system 100 may be realized by the cooperative operation of multiple computers connected by a communication path such as Ethernet, or may be realized by an electronic musical instrument with a performance function such as an electronic piano.

RAM 110, ROM 120, CPU 130, memory unit 140, operation unit 150 and display unit 160 are connected to bus 170. RAM 110, ROM 120 and CPU 130 constitute the training device 10 and chord estimation device 20. In this embodiment, the training device 10 and chord estimation device 20 are constituted by a common processing system 100, but may also be constituted by separate processing systems.

RAM 110 is made of, for example, volatile memory and is used as a working area for CPU 130. ROM 120 is made of, for example, non-volatile memory and stores a training program and a chord estimation program. CPU 130 performs training processing by executing the training program stored in ROM 120 on RAM 110. CPU 130 also performs chord estimation processing by executing the chord estimation program stored in ROM 120 on RAM 110. Details of the training processing and chord estimation processing will be described later.

The training program or chord estimation program may be stored in the memory unit 140 instead of the ROM 120. Alternatively, the training program or chord estimation program may be provided in a form stored on a computer-readable storage medium and installed in the ROM 120 or the memory unit 140. Alternatively, if the processing system 100 is connected to a network such as the Internet, the training program or chord estimation program may be distributed from a server (including a cloud server) on the network and installed in the ROM 120 or the memory unit 140.

The storage unit 140 includes a storage medium such as a hard disk, optical disk, magnetic disk, or memory card, and stores the trained model M and multiple pieces of training data D. The trained model M or each piece of training data D may not be stored in the storage unit 140, but may be stored in a computer-readable storage medium. Alternatively, if the processing system 100 is connected to a network, the trained model M or each piece of training data D may be stored on a server on the network.

(2) Training Data The trained model M is a machine learning model trained to present chord sequences that a user of the chord estimation device 20 (hereinafter referred to as a performer) can refer to when playing a piece of music. The trained model M is constructed using multiple sets of training data D. A user of the training device 10 can generate the training data D by operating the operation unit 150. The training data D is data created based on the musical knowledge or musical sense of a reference performer. The reference performer has a relatively high level of skill in playing a piece of music. The reference performer may be the performer's instructor or teacher in playing a piece of music.

Training data D represents a set of input time series data and output chord sequence information. The input time series data represents a reference note sequence consisting of multiple notes. For example, the input time series data is data that comprises a melody or accompaniment sound made up of multiple notes. The input time series data may be image data representing an image of musical score. The output chord sequence information is data in which chords corresponding to the reference note sequence are arranged in time series. The chord sequence corresponding to the reference note sequence is assigned by a reference performer.

Figures 2 and 3 show examples of each training data D. The example in Figure 2 shows input time series data including a reference note sequence consisting of multiple notes. The example in Figure 3 shows output chord sequence information indicating the chord sequence corresponding to the reference note sequence.

In this embodiment, the input time series data has a metrical structure and additional information in addition to a reference note sequence. Input time series data A shown in Figure 2 is an excerpt of the first two bars of a song. Input time series data A is divided into bars by "bars" and into beats by "beats." In this way, input time series data A has a metrical structure based on "bar" and "beat" information. Elements A1 to A37 indicate the reference note sequence of the first bar. In other words, elements A1 to A37 are divided into bars by the "bar" before element A1 and the "bar" after element A37. Furthermore, elements A8, A18, and A26 are divided into beats by the "beats" after them.

Element A0 is additional information. Examples of additional information include key information, genre information, and difficulty level information. In the example of Figure 2, key information is added using the Key element. Key information specifies the key of the music represented by the reference note sequence. The number following Key specifies the key. By specifying key information as additional information, a chord sequence corresponding to the reference note sequence and the key is machine-learned. Genre information specifies the genre of the music represented by the reference note sequence. Examples of genre information include rock, pop, and jazz. By specifying genre information as additional information, a chord sequence corresponding to the reference note sequence and the genre is machine-learned. Difficulty information indicates the difficulty level of the musical score represented by the reference note sequence. By specifying difficulty information as additional information, a chord sequence corresponding to the reference note sequence and the difficulty level of the musical score is machine-learned. For example, for a low-difficulty score, machine learning is performed by interpolating notes from a small number of notes. On the other hand, for a high-difficulty score, machine learning is performed by selecting notes that constitute chords from an excessive number of notes.

Of the elements of the input time series data A, elements other than element A0, "bar", and "beat" correspond to the reference note sequence. Elements A1 to A37 indicate the reference note sequence for the first bar. In this example, element A0 is placed at the beginning of the input time series data A, i.e., before the reference note sequence (elements A1 to A37), but it may be placed at any position in the input time series data A.

As illustrated by elements A1 through A37, in the reference note sequence, "L" represents the left hand, "R" represents the right hand, and the number following "L" or "R" represents the scale. Furthermore, "on" and "off" represent key depression and key release, respectively. Furthermore, "wait" represents waiting, and the number following "wait" represents the length of time. Therefore, elements A1 through A5 indicate that the right hand simultaneously presses keys 77 and 74, while the left hand simultaneously presses keys 53 and 46, and then maintains these positions for 11 time units. After maintaining these positions for 11 time units, elements A6 through A8 indicate that the left hand simultaneously releases keys 53 and 46, and then maintains these positions for one time unit. After maintaining these positions for one time unit, elements A9 through A11 indicate that the left hand again presses keys 53 and 46, and then waits for five time units.

The output chord sequence information B shown in Figure 3 indicates a chord sequence corresponding to the reference note sequence contained in the input time series data A. The chord sequence corresponding to elements A1 to A37 of the input time series data A is represented by elements B1 to B3 and elements B4 to B6. In other words, elements B1 to B6 indicate the chord sequence corresponding to the first measure of the input time series data A. In the output chord sequence information B, measures are also separated by "bar" and beats are separated by "beat." The range separated by the "bar" before element B1 and the "bar" after element B6 corresponds to the first measure.

In output chord sequence information B, one chord is represented by three elements. Elements B1 to B3 specify the chord on the first beat of the first measure. Elements B4 to B6 specify the chord on the fourth beat of the first measure. Elements B7 to B9 specify the chord on the first beat of the fourth measure. Of the three elements indicating a chord, the first element (B1, B4, B7) indicates basic chord information. Basic chord information (chord) indicates a number from 1 to 24 specifying the type of major chord or minor chord for each of the 12 notes (C, C#, D, D#, ... A, A#, B). Of the three elements indicating a chord, the second element (B2, B5, B8) indicates chord type information. Chord type information (type) indicates a number specifying the type of tension chord. Of the three elements indicating a chord, the third element (B3, B6, B9) indicates chord root information. The chord root information (root) indicates a numerical value that specifies the root note of the on-chord.

(3) Training Device and Chord Estimation Device Fig. 4 is a block diagram showing the configurations of the training device 10 and the chord estimation device 20. As shown in Fig. 4, the training device 10 includes, as functional units, a first acquisition unit 11, a second acquisition unit 12, and a construction unit 13. The functional units of the training device 10 are realized by the CPU 130 in Fig. 1 executing a training program. At least a part of the functional units of the training device 10 may be realized by hardware such as electronic circuits.

The first acquisition unit 11 acquires input time series data A from each training data D stored in the memory unit 140, etc. The second acquisition unit 12 acquires output code sequence information B from each training data D. The construction unit 13 performs machine learning for each training data D, using the input time series data A acquired by the first acquisition unit 11 as an input element and the output code sequence information B acquired by the second acquisition unit 12 as an output element. By repeating machine learning for multiple training data D, the construction unit 13 constructs a trained model M that indicates the input/output relationship between the input time series data A and the output code sequence information B.

In this example, the construction unit 13 constructs the trained model M by training the Transformer, but the embodiment is not limited to this. The construction unit 13 may also construct the trained model M by training another type of machine learning model that handles time series. The trained model M constructed by the construction unit 13 is stored, for example, in the storage unit 140. The trained model M constructed by the construction unit 13 may also be stored on a server on a network, etc.

The chord estimation device 20 includes, as functional units, a reception unit 21, an estimation unit 22, and a generation unit 23. The functional units of the chord estimation device 20 are realized by the CPU 130 in Figure 1 executing a chord estimation program. At least some of the functional units of the chord estimation device 20 may be realized by hardware such as electronic circuits.

In this embodiment, the receiving unit 21 receives time series data including a sequence of notes consisting of multiple notes. The performer can provide image data showing an image of a musical score to the receiving unit 21 as time series data. Alternatively, the performer can generate time series data by operating the operation unit 150 and provide it to the receiving unit 21. In this example, the time series data has the same structure as the input time series data A in Figure 2. In other words, the time series data has a metrical structure and additional information in addition to a sequence of notes.

The estimation unit 22 estimates chord sequence information using a trained model M stored in the memory unit 140 or the like. The chord sequence information indicates a chord sequence corresponding to the note sequence received by the reception unit 21, and is estimated based on the note sequence and additional information. Because the time series data has a structure similar to that of the input time series data A, the chord sequence information has a structure similar to that of the output chord sequence information B. The generation unit 23 generates musical score information based on the note sequence of the time series data received by the reception unit 21 and the chord sequence information estimated by the estimation unit 22. For example, the musical score information is information on an arranged piano score, and is data in which chord information is added to a staff. Alternatively, the musical score information is MIDI data to which chord sequence information has been added.

The display unit 160 displays a chord-added musical score based on the musical score information generated by the generation unit 23. Figure 5 shows an example of a chord-added musical score displayed on the display unit 160. As shown in Figure 5, the chord-added musical score displays the chord sequence information estimated by the estimation unit 22 in correspondence with each note of the note sequence accepted by the acceptance unit 21.

(4) Training Process and Chord Estimation Process Figure 6 is a flowchart showing an example of training process by the training device 10 of Figure 4. The training process of Figure 6 is performed by the CPU 130 of Figure 1 executing a training program. First, the first acquisition unit 11 acquires input time-series data A from each training data D (step S1). Furthermore, the second acquisition unit 12 acquires output chord string information B from each training data D (step S2). Steps S1 and S2 may be executed first, or may be executed simultaneously.

Next, the construction unit 13 performs machine learning for each training data D, using the input time-series data A acquired in step S1 as the input element and the output code string information B acquired in step S2 as the output element (step S3). Subsequently, the construction unit 13 determines whether sufficient machine learning has been performed (step S4). If the machine learning is insufficient, the construction unit 13 returns to step S3. Steps S3 and S4 are repeated while changing the parameters until sufficient machine learning has been performed. The number of times the machine learning is repeated varies depending on the quality conditions that the trained model M to be constructed must satisfy.

When sufficient machine learning has been performed, the construction unit 13 saves the input/output relationship between the input time series data A and the output code sequence information B acquired through the machine learning in step S3 as a trained model M (step S5). This completes the training process.

Figure 7 is a flowchart showing an example of chord estimation processing by the chord estimation device 20 of Figure 4. The chord estimation processing of Figure 7 is performed by the CPU 130 of Figure 1 executing a chord estimation program. First, the reception unit 21 receives time-series data (step S11). Next, the estimation unit 22 estimates chord sequence information from the time-series data received in step S11 using the trained model M saved in step S5 of the training process (step S12). At this time, chord sequence information including one or more chord sequences is estimated from the sequence of notes included in the time-series data, allowing for highly flexible chord estimation. Furthermore, the timing of chord changes over time is also estimated, allowing for more appropriate chord estimation. In other words, although the time-series data does not include information that marks the boundaries of chord changes, the estimation unit 22 performs chord estimation that includes the timing of chord changes.

The generation unit 23 then generates musical score information based on the note sequence of the time-series data received in step S11 and the chord sequence information estimated in step S12 (step S13). Based on the generated musical score information, a musical score with chords may be displayed on the display unit 160. This completes the chord estimation process.

(5) Effects of the Embodiment As described above, the chord estimation device 20 according to the present embodiment includes a receiving unit 21 that receives time-series data including a note sequence consisting of a plurality of notes, and an estimation unit 22 that uses a trained model M to estimate chord sequence information indicating a chord sequence corresponding to the note sequence. With this configuration, the trained model M is used to estimate appropriate chord sequence information from the temporal flow of a plurality of notes in the time-series data. This makes it possible to present a musical score with chords based on time-series data including a note sequence. Since one or more chord sequences are estimated from a note sequence, chord estimation can be performed with a high degree of flexibility.

The trained model M may be a machine learning model that has learned the input/output relationship between input time-series data A including a reference note sequence consisting of multiple notes, and output chord sequence information B indicating the chord sequence corresponding to each note in the reference note sequence. In this case, the chord sequence information can be easily estimated from the time-series data.

The estimation unit 22 may also estimate the timing of chord changes in the chord sequence. This allows for more appropriate chord estimation corresponding to the note sequence.

The input time series data A may include genre information specifying the genre of music represented by the reference note sequence. The time series data may also include genre information specifying the genre of music represented by the note sequence. The estimation unit 22 may then estimate chord sequence information based on the time series data including genre information. This allows chord estimation appropriate for the music genre.

The input time series data A may include key information specifying the key of the music expressed by the reference note sequence. The time series data may also include key information specifying the key of the music expressed by the note sequence. The estimation unit 22 may then estimate chord sequence information based on the time series data including key information. This allows chord estimation appropriate to the key of the music.

The input time series data A may include difficulty level information that specifies the difficulty level of the musical score represented by the reference note sequence. The time series data may also include difficulty level information that specifies the difficulty level of the musical score represented by the note sequence. The estimation unit 22 may then estimate chord sequence information based on the time series data that includes the difficulty level information. This allows appropriate chord estimation to be performed according to the difficulty level of the musical score represented by the note sequence.

The chord estimation device 20 may further include a generation unit 23 that generates musical score information indicating a chord-attached musical score to which chord sequence information is attached to correspond to each note in the musical note sequence.

The training device 10 according to this embodiment comprises a first acquisition unit 11 that acquires input time series data A including a reference note sequence consisting of a plurality of notes, a second acquisition unit 12 that acquires output chord sequence information B indicating a chord sequence corresponding to the reference note sequence, and a construction unit 13 that constructs a trained model M that has learned the input-output relationship between the input time series data A and the output chord sequence information B. This configuration makes it easy to construct a trained model M that has learned the input-output relationship between the input time series data A and the output chord sequence information B.

(6) Other Embodiments In the above embodiment, the input time series data A includes additional information, and the time series data includes additional information, but the embodiment is not limited to this. The input time series data A only needs to include a reference sequence of notes, and does not need to include additional information. Similarly, the time series data only needs to include a sequence of notes, and does not need to include additional information.

In the above embodiment, the input time series data A has "bar" and "beat" information as a metrical structure, but the embodiment is not limited to this. The input time series data A does not have to have a metrical structure. Figure 8 is a diagram showing an example of output chord sequence information B prepared in response to input time series data A that does not have a metrical structure. As shown in Figure 8, the output chord sequence information B does not have a metrical structure consisting of "bar" and "beat" information.

In the above embodiment, the input time series data A includes tone information, genre information, and difficulty level information as additional information. The construction unit 13 may construct different trained models M depending on the type of additional information, or may construct a single trained model M. Alternatively, the input time series data A may include multiple pieces of information selected from tone information, genre information, and difficulty level information as additional information.

In addition, in the above embodiment, the chord estimation device 20 includes the generation unit 23, but the embodiment is not limited to this. A performer can create a score with chords by transcribing the chord sequence information estimated by the estimation unit 22 into a desired musical score. Therefore, the chord estimation device 20 does not need to include the generation unit 23.

In the above embodiment, the training data D is trained to estimate chord sequence information when playing on a piano, but the embodiment is not limited to this. The training data D may also be trained to estimate chord sequence information when playing on other instruments such as a guitar or drums.

In the above embodiment, the user of the chord estimation device 20 is a performer, but the user of the chord estimation device 20 may also be, for example, a staff member of a music score production company. Furthermore, machine learning by the training device 10 may be performed in advance by a staff member of the music score production company.

Claims (14)

a receiving unit that receives time-series data including a musical note sequence consisting of a plurality of musical notes;
an estimation unit that estimates chord sequence information indicating a chord sequence corresponding to the musical note sequence, the chord sequence information including chord change timings, based on the time-series data, using a trained model;
The chord estimation device, wherein the estimation unit also estimates the timing of chord changes in the chord sequence. The chord estimation device of claim 1, wherein the trained model is a model that has learned the input/output relationship between input time-series data including a reference note sequence consisting of a plurality of notes and output chord sequence information indicating a chord sequence corresponding to the reference note sequence. the input time-series data includes genre information that specifies the genre of music expressed by the reference sequence of notes;
the time-series data includes genre information that specifies the genre of music expressed by the sequence of notes;
The chord estimation device according to claim 2 , wherein the estimation unit estimates the chord string information based on the time-series data including genre information. the input time-series data includes key information that specifies the key of the music expressed by the reference sequence of notes;
the time-series data includes key information that specifies the key of the music expressed by the sequence of notes;
The chord estimation device according to claim 2 , wherein the estimation unit estimates the chord string information based on the time-series data including key information. the input time-series data includes difficulty level information that specifies the difficulty level of the musical score indicated by the reference note sequence,
the time-series data includes difficulty level information that specifies the difficulty level of the musical score represented by the sequence of notes,
The chord estimation device according to claim 2 , wherein the estimation unit estimates the chord string information based on the time-series data including difficulty level information. The chord estimation device according to any one of claims 1 to 5, further comprising a generation unit that generates musical score information indicating a chord-attached musical score to which the chord sequence information is attached so as to correspond to each note of the musical note sequence. a first acquisition unit that acquires input time-series data including a reference note sequence consisting of a plurality of notes;
a second acquisition unit that acquires output chord sequence information, which is information indicating a chord sequence corresponding to the reference note sequence and in which chord change timings are recorded;
a construction unit that constructs a trained model that has learned the input/output relationship between the input time series data and the output code sequence information, including the timing of code changes in the code sequence. Accepts time series data containing a sequence of multiple notes,
using the trained model, based on the time-series data, to estimate chord sequence information indicating a chord sequence corresponding to the musical note sequence, the chord sequence information including chord change timings;
The computer-implemented chord estimation method also estimates the timing of chord changes in the chord sequence. The computer-executed chord estimation method of claim 8, wherein the trained model is a model that has learned the input/output relationship between input time-series data including a reference note sequence consisting of multiple notes and output chord sequence information indicating a chord sequence corresponding to the reference note sequence. the input time-series data includes genre information that specifies the genre of music expressed by the reference sequence of notes;
the time-series data includes genre information that specifies the genre of music expressed by the sequence of notes;
10. The computer-implemented chord estimation method of claim 9 , wherein said estimating comprises estimating said chord string information based on said time-series data including genre information. the input time-series data includes key information that specifies the key of the music expressed by the reference sequence of notes;
the time-series data includes key information that specifies the key of the music expressed by the sequence of notes;
10. The computer-implemented chord estimation method of claim 9, wherein said estimating comprises estimating said chord string information based on said time-series data including key information. the input time-series data includes difficulty level information that specifies the difficulty level of the musical score indicated by the reference note sequence,
the time-series data includes difficulty level information that specifies the difficulty level of the musical score represented by the sequence of notes,
10. The computer-implemented chord estimation method of claim 9, wherein said estimating comprises estimating said chord string information based on said time-series data including difficulty level information. The computer-implemented chord estimation method described in any one of claims 8 to 12 further generates musical score information indicating a chord-attached musical score to which the chord sequence information is attached so as to correspond to each note of the musical note sequence. Obtain input time series data including a reference note sequence consisting of a plurality of notes;
obtain output chord sequence information indicating a chord sequence corresponding to the reference note sequence, the output chord sequence information including recorded chord change timings;
A computer-implemented training method for constructing a trained model that has learned the input-output relationship between the input time-series data and the output code sequence information, including the timing of code changes in the code sequence.