JP7679871B2 - Fingering practice device, fingering practice method, and training method - Google Patents

Description

The present invention relates to a fingering presentation device, a training device, a fingering presentation method and a training method that present fingering when playing a musical instrument.

There are known devices for assisting practice of playing musical instruments. For example, in an information processing device described in Patent Literature 1, a performance technical level of a player is calculated, and musical pieces that the player can play are presented based on the calculated performance technical level. However, if the player is an inexperienced player, it is not easy to appropriately determine fingering (hereinafter referred to as fingering) for playing each note on an instrument. In response to this, Patent Literature 2 describes a fingering determination method for determining fingering for each note of a note sequence based on a probabilistic model.
JP 2013-083845 A JP 2007-241034 A

According to Patent Document 2, a performer can recognize fingerings for playing an instrument based on a probabilistic model. However, in reality, there are an infinite number of fingering combinations, and there is not just one optimal fingering for playing a piece of music. Therefore, it is desirable to be presented with more appropriate fingerings.

The object of the present invention is to provide a fingering presentation device, a training device, a fingering presentation method and a training method capable of presenting appropriate fingering when playing a musical instrument.

A fingering presentation device according to a first aspect of the present invention includes a receiving unit that receives time series data including a note sequence consisting of a plurality of notes, and an estimation unit that uses a trained model to estimate fingering information indicating fingers to be used when playing at least some of the notes included in the note sequence on an instrument, wherein the time series data further includes a player identifier indicating a player playing the note sequence, and the estimation unit estimates the fingering information based on the player identifier , and the trained model is a machine learning model that has mastered an input/output relationship between input time series data including a reference note sequence consisting of a plurality of notes, and output fingering information indicating fingers to be used when playing at least some of the notes included in the reference note sequence on an instrument, wherein the input time series data further includes a reference player identifier indicating a reference player playing the reference note sequence, and the output fingering information indicates the fingers of the reference player to be used when playing the notes on the instrument .
A fingering presentation device according to a second aspect of the present invention includes a receiving unit that receives time-series data including a note sequence consisting of a plurality of notes, and an estimation unit that uses a trained model to estimate fingering information indicating fingers to be used when playing at least some of the notes included in the note sequence on a musical instrument, wherein the time-series data further includes basic fingering information indicating fingers to be used when playing a first proportion of the notes included in the note sequence on a musical instrument, and the estimation unit estimates fingering information indicating fingers to be used when playing a second proportion of the notes included in the note sequence on a musical instrument, the second proportion being greater than the first proportion, based on the basic fingering information, and the trained model is a machine learning model that has learned an input-output relationship between input time-series data including a reference note sequence consisting of a plurality of notes, and output fingering information indicating fingers to be used when playing at least some of the notes included in the reference note sequence on a musical instrument, and the input time-series data further includes basic fingering information indicating fingers to be used when playing the first proportion of the notes included in the reference note sequence on a musical instrument, and the output fingering information further includes the basic fingering information included in the input time-series data .
A fingering presentation device according to a third aspect of the present invention includes a receiving unit that receives time-series data including a note sequence consisting of a plurality of notes, and an estimation unit that uses a trained model to estimate finger information indicating the fingers to be used when playing at least some of the notes included in the note sequence on an instrument, and note information indicating the notes to which fingering is to be assigned from the note sequence, wherein the estimation unit estimates intermediate finger information indicating the fingers to be used when playing each note included in the note sequence on an instrument, and the note information , and estimates finger information indicating the fingers to be used when playing notes included in the note sequence on an instrument other than the notes indicated by the note information by deleting from the intermediate finger information the finger information corresponding to the notes to which fingering is to be assigned in the note information.

A training device according to a fourth aspect of the present invention includes a first acquisition unit that acquires input time series data including a reference note sequence consisting of a plurality of notes, a second acquisition unit that acquires output fingering information indicating fingers to be used when playing at least some of the notes included in the reference note sequence on an instrument, or output note information indicating notes to which fingering is to be assigned from the reference note sequence, and a construction unit that constructs a trained model that has acquired the input/output relationship between the input time series data and the output fingering information or the output note information, wherein the input time series data further includes a reference player identifier indicating the reference player playing the reference note sequence, and the output fingering information indicates the fingers of the reference player to be used when playing on the instrument.

A fingering guidance presentation method according to a fifth aspect of the present invention receives time-series data including a note sequence consisting of a plurality of notes, and estimates fingering information indicating fingers to be used when playing at least some of the notes included in the note sequence on a musical instrument using a trained model, the time-series data further including a player identifier indicating a player playing the note sequence, and estimating the fingering information includes estimating the fingering information based on the player identifier, the trained model is a machine learning model that has acquired an input-output relationship between input time-series data including a reference note sequence consisting of a plurality of notes and output fingering information indicating fingers to be used when playing at least some of the notes included in the reference note sequence on a musical instrument, the input time-series data further including a reference player identifier indicating a reference player playing the reference note sequence, and the output fingering information indicating the fingers of the reference player to be used when playing the notes on the musical instrument, and is executed by a computer.
A fingering guidance presentation method according to a sixth aspect of the present invention receives time-series data including a note sequence consisting of a plurality of notes, and estimates fingering information indicating fingers to be used when playing at least some of the notes included in the note sequence on a musical instrument using a trained model, the time-series data further including basic fingering information indicating fingers to be used when playing a first proportion of the notes included in the note sequence on a musical instrument, and estimating the fingering information includes estimating, based on the basic fingering information, fingering information indicating fingers to be used when playing a second proportion of the notes included in the note sequence on a musical instrument, the second proportion being greater than the first proportion, and the trained model is a machine learning model that has acquired an input-output relationship between input time-series data including a reference note sequence consisting of a plurality of notes and output fingering information indicating fingers to be used when playing at least some of the notes included in the reference note sequence on a musical instrument, the input time-series data further including basic fingering information indicating fingers to be used when playing the first proportion of the notes included in the reference note sequence on a musical instrument, and the output fingering information further including the basic fingering information included in the input time-series data, and is executed by a computer.
A fingering presentation method according to a seventh aspect of the present invention accepts time-series data including a note sequence consisting of a plurality of notes, and uses a trained model to estimate finger information indicating fingers to be used when playing at least some of the notes included in the note sequence on an instrument, and note information indicating notes to which fingering is to be assigned from the note sequence, where estimating the finger information includes estimating intermediate finger information indicating fingers to be used when playing each note included in the note sequence on an instrument, and the note information, and deleting from the intermediate finger information finger information corresponding to the note to which fingering in the note information is to be assigned, thereby estimating finger information indicating fingers to be used when playing notes included in the note sequence on an instrument other than the notes indicated by the note information, and is executed by a computer.

A training method according to an eighth aspect of the present invention includes the steps of: acquiring input time series data including a reference note sequence consisting of a plurality of notes; acquiring output fingering information indicating fingers to be used when playing at least some of the notes included in the reference note sequence on an instrument, or output note information indicating notes to which fingering is to be assigned from the reference note sequence; constructing a trained model that has acquired an input/output relationship between the input time series data and the output fingering information or output note information, wherein the input time series data further includes a reference player identifier indicating a reference player playing the reference note sequence, and the output fingering information indicates the fingers of the reference player to be used when playing on the instrument, and is executed by a computer.

The present invention makes it possible to present appropriate fingering when playing a musical instrument.

FIG. 1 is a block diagram showing the configuration of a processing system including a fingering technique presentation device and a training device according to a first embodiment of the present invention. FIG. 2 is a diagram showing an example of each training data. FIG. 3 is a block diagram showing the configuration of the training device and the fingering technique presentation device. FIG. 4 shows an example of the auxiliary musical score displayed on the display unit. FIG. 5 is a flow chart showing an example of a training process performed by the training device of FIG. FIG. 6 is a flowchart showing an example of a fingering guide process performed by the fingering guide device of FIG. FIG. 7 is a diagram showing another example of the input time series data. FIG. 8 is a diagram showing an example of input time-series data in the modified example. FIG. 9 is a diagram showing an example of output finger information in the modified example. FIG. 10 is a diagram showing an example of input time-series data in the second embodiment. FIG. 11 is a diagram showing an example of output finger information in the third embodiment. FIG. 12 is a flowchart showing an example of a fingering guide suggestion process in the modified example. FIG. 13 is a diagram showing an example of finger information estimated in step S24 of the fingering pattern suggestion process.

[1] First embodiment (1) Configuration of the processing system Hereinafter, a fingering practice device, a training device, a fingering practice method, and a training method according to an embodiment of the present invention will be described in detail with reference to the drawings. Fig. 1 is a block diagram showing the configuration of a processing system including a fingering practice device and a training device according to a first embodiment of the present invention. As shown in Fig. 1, a processing system 100 includes a RAM (random access memory) 110, a ROM (read only memory) 120, a CPU (central processing unit) 130, a storage unit 140, an operation unit 150, and a display unit 160.

The processing system 100 is realized by a computer such as a personal computer, a tablet terminal, or a smartphone. Alternatively, the processing system 100 may be realized by the cooperative operation of multiple computers connected by a communication path such as Ethernet, or may be realized by an electronic musical instrument with a performance function such as an electronic piano.

The RAM 110, the ROM 120, the CPU 130, the memory unit 140, the operation unit 150 and the display unit 160 are connected to a bus 170. The training device 10 and the fingering presentation device 20 are configured by the RAM 110, the ROM 120 and the CPU 130. In this embodiment, the training device 10 and the fingering presentation device 20 are configured by a common processing system 100, but may be configured by separate processing systems.

The RAM 110 is, for example, a volatile memory, and is used as a working area for the CPU 130. The ROM 120 is, for example, a non-volatile memory, and stores a training program and a fingering presentation program. The CPU 130 performs training processing by executing the training program stored in the ROM 120 on the RAM 110. The CPU 130 also performs fingering presentation processing by executing the fingering presentation program stored in the ROM 120 on the RAM 110. Details of the training processing and the fingering presentation processing will be described later.

The training program or fingering presentation program may be stored in the storage unit 140 instead of the ROM 120. Alternatively, the training program or fingering presentation program may be provided in a form stored in a computer-readable storage medium and installed in the ROM 120 or the storage unit 140. Alternatively, when the processing system 100 is connected to a network such as the Internet, the training program or fingering presentation program distributed from a server (including a cloud server) on the network may be installed in the ROM 120 or the storage unit 140.

The storage unit 140 includes a storage medium such as a hard disk, an optical disk, a magnetic disk, or a memory card, and stores the trained model M and a plurality of training data D. The trained model M or each training data D may be stored in a computer-readable storage medium rather than in the storage unit 140. Alternatively, when the processing system 100 is connected to a network, the trained model M or each training data D may be stored in a server on the network.

(2) Training Data The trained model M is a machine learning model trained to present fingering when a user of the fingering presentation device 20 (hereinafter referred to as a performer) plays a musical piece on an instrument, and is constructed using multiple training data D. A user of the training device 10 can generate the training data D by operating the operation unit 150. The training data D is data created based on the performance knowledge or performance style of a reference performer. The reference performer has a relatively high level of skill in performing a musical piece. The reference performer may be an instructor or teacher of the performer in performing the musical piece.

Training data D indicates a set of input time series data and output fingering information. The input time series data indicates a reference note sequence consisting of a plurality of notes. The input time series data may be image data showing an image of a musical score. The output fingering information indicates the fingers of the reference performer to be used when playing each note of the reference note sequence on an instrument, and may be used to present fingering when playing the reference note sequence. The output fingering information may be a unique number assigned to each finger. In this example, the numbers "1" to "5" are assigned to the thumb, index finger, middle finger, ring finger, and little finger, respectively.

Here, the optimal fingering for playing a piece of music varies depending on the physical characteristics of the performer or the performance style of the performer. Therefore, in this embodiment, the input time series data further includes a reference performer identifier indicating the classification (category) of the reference performer who performs the reference sequence of notes. The reference performer identifier is determined to be different for at least one of the physical characteristics of the reference performer and the performance style of the reference performer. The physical characteristics of the reference performer include, for example, the hand size (finger length), age, sex, or whether the reference performer is an adult or a child.

Figure 2 is a diagram showing an example of each training data D. The example in Figure 2 shows a portion of the input time series data and output fingering information when a reference performer plays the piano. As shown in Figure 2, the input time series data A includes elements A0 to A16. Element A0 corresponds to a reference performer identifier and is represented by a character string that differs for at least one of the physical characteristics of the reference performer and the style of performance by the reference performer. Elements A1 to A16 correspond to the reference note sequence. In this example, element A0 is placed at the beginning of the input time series data A, i.e., before the reference note sequence (elements A1 to A16), but it may be placed at any position in the input time series data A.

In elements A1, A3, A5, ..., A15, "L" means the left hand, the numbers mean the numbers assigned to the keys, and "on" and "off" mean pressing and releasing a key, respectively. In elements A2, A4, A6, ..., A16, "wait" means waiting, and the numbers mean the length of time. Thus, elements A1 to A4 mean pressing key number "66" and holding it for 13 units of time, and then releasing key number "66" and holding it for 2 units of time.

The output finger information B includes elements B0 to B16, which correspond to elements A0 to A16 of the input time series data A, respectively. Element B0 indicates the reference performer identifier and is represented by the same character string as element A0. In elements B1, B3, B5, ..., B15, "L" means the left hand, the numbers mean the numbers assigned to the fingers, and "down" and "up" mean pushing up and pushing down, respectively. In elements B2, B4, B6, ..., B16, "wait" means waiting, and the numbers mean the length of time. Thus, elements B1 to B4 mean that the middle finger of the left hand is pushed down and waited for 13 units of time, and then the middle finger of the left hand is pushed up and maintained for 2 units of time.

The training data D in FIG. 2 is generated to indicate fingering for the left hand, but the embodiment is not limited to this. The training data D may be generated to indicate fingering for the right hand, or to indicate fingering for each of the left and right hands. In the elements of the input time series data A and the output fingering information B to indicate fingering for the right hand, the letter "R", for example, may be used instead of "L".

(3) Training Device and Fingering Presentation Device Fig. 3 is a block diagram showing the configurations of the training device 10 and the fingering presentation device 20. As shown in Fig. 3, the training device 10 includes, as functional units, a first acquisition unit 11, a second acquisition unit 12, and a construction unit 13. The functional units of the training device 10 are realized by the CPU 130 in Fig. 1 executing a training program. At least a part of the functional units of the training device 10 may be realized by hardware such as an electronic circuit.

The first acquisition unit 11 acquires input time series data A from each training data D stored in the memory unit 140 or the like. The second acquisition unit 12 acquires output finger information B from each training data D. The construction unit 13 performs machine learning for each training data D, using the input time series data A acquired by the first acquisition unit 11 as an input element and the output finger information B acquired by the second acquisition unit 12 as an output element. By repeating machine learning for multiple training data D, the construction unit 13 constructs a trained model M that indicates the input/output relationship between the input time series data A and the output finger information B.

In this example, the construction unit 13 constructs the trained model M by training the Transformer, but the embodiment is not limited to this. The construction unit 13 may construct the trained model M by training another type of machine learning model that handles time series. The trained model M constructed by the construction unit 13 is stored in, for example, the storage unit 140. The trained model M constructed by the construction unit 13 may be stored in a server on a network, etc.

The fingering presentation device 20 includes, as functional units, a reception unit 21, an estimation unit 22, and a generation unit 23. The functional units of the fingering presentation device 20 are realized by the CPU 130 in Fig. 1 executing the fingering presentation program. At least a part of the functional units of the fingering presentation device 20 may be realized by hardware such as an electronic circuit.

In this embodiment, the receiving unit 21 receives time series data including a string of notes consisting of a plurality of notes. The performer can provide image data showing an image of a musical score to the receiving unit 21 as time series data. Alternatively, the performer can generate time series data by operating the operation unit 150 and provide the time series data to the receiving unit 21.

In this example, the time series data has a structure similar to that of the input time series data A in FIG. 2, and further includes a performer identifier indicating a classification (category) of a performer who performs the sequence of notes. The performer identifier is determined to be different for at least one of the physical characteristics of the performer and the performance style of the performer. The physical characteristics of the performer include, for example, the size of the performer's hands, age, sex, or whether the performer is an adult or a child.

The estimation unit 22 estimates finger information using a trained model M stored in the memory unit 140 or the like. The finger information indicates the player's fingers used when playing each note in the note sequence accepted by the acceptance unit 21, and is estimated based on the note sequence and a player identifier. The finger information may be a unique number assigned to each finger. The generation unit 23 generates musical score information based on the note sequence of the time-series data accepted by the acceptance unit 21 and the finger information estimated by the estimation unit 22.

The display unit 160 displays the auxiliary musical score based on the musical score information generated by the generation unit 23. Fig. 4 shows an example of the auxiliary musical score displayed on the display unit 160. As shown in Fig. 4, the auxiliary musical score shows finger information estimated by the estimation unit 22 corresponding to each note of the sequence of notes accepted by the acceptance unit 21. In the example of Fig. 4, the finger numbers of one hand are shown as finger information.

When distinguishing between the numbers of the fingers of the left and right hands, a predetermined letter such as "L" may be added near the number of the fingers of the left hand, and another predetermined letter such as "R" may be added near the number of the fingers of the right hand. Alternatively, a predetermined color such as red may be added to the numbers of the fingers of the left hand or the corresponding notes, and another predetermined color such as blue may be added to the numbers of the fingers of the right hand or the corresponding notes.

(4) Training Processing and Fingering Presentation Processing Fig. 5 is a flow chart showing an example of training processing by the training device 10 of Fig. 3. The training processing of Fig. 5 is performed by the CPU 130 of Fig. 1 executing a training program. First, the first acquisition unit 11 acquires input time-series data A from each training data D (step S1). Furthermore, the second acquisition unit 12 acquires output finger information B from each training data D (step S2). Either step S1 or S2 may be executed first, or they may be executed simultaneously.

Next, the construction unit 13 performs machine learning for each training data D using the input time series data A acquired in step S1 as an input element and the output finger information B acquired in step S2 as an output element (step S3). Next, the construction unit 13 determines whether sufficient machine learning has been performed (step S4). If the machine learning is insufficient, the construction unit 13 returns to step S3. Steps S3 and S4 are repeated while changing the parameters until sufficient machine learning has been performed. The number of times the machine learning is repeated varies depending on the quality conditions that the trained model M to be constructed must satisfy.

When sufficient machine learning has been performed, the construction unit 13 saves the input/output relationship between the input time-series data A and the output finger information B acquired by the machine learning in step S3 as a trained model M (step S5). This ends the training process.

Figure 6 is a flowchart showing an example of fingering presentation processing by the fingering presentation device 20 of Figure 3. The fingering presentation processing of Figure 6 is performed by the CPU 130 of Figure 1 executing a fingering presentation program. First, the reception unit 21 receives time series data (step S11). Next, the estimation unit 22 estimates finger information from the time series data received in step S11 using the trained model M saved in step S5 of the training processing (step S12).

Then, the generation unit 23 generates score information based on the note sequence of the time-series data received in step S11 and the fingering information estimated in step S12 (step S13). Based on the generated score information, an auxiliary score may be displayed on the display unit 160. This completes the fingering presentation process.

(5) Effect of the embodiment As described above, the fingering presentation device 20 according to the embodiment includes a receiving unit 21 that receives time-series data including a note sequence consisting of a plurality of notes, and an estimating unit 22 that estimates fingering information indicating the finger to be used when playing each note of the note sequence on an instrument, using a trained model M. According to this configuration, appropriate fingering information is estimated from the temporal flow of the plurality of notes in the time-series data, using the trained model M. This makes it possible to present appropriate fingering when playing an instrument.

The trained model M may be a machine learning model that has learned the input/output relationship between input time-series data A including a reference note sequence consisting of a plurality of notes, and output fingering information B indicating the finger to be used when playing each note of the reference note sequence on an instrument. In this case, the fingering information can be easily estimated from the time-series data.

The time-series data may further include a performer identifier indicating a performer who plays the sequence of notes, and the estimation unit 22 may estimate finger information based on the performer identifier. In this case, appropriate finger information can be estimated according to the performer.

The performer identifier may be determined to correspond to the physical characteristics of the performer. In this case, appropriate finger information can be estimated according to the physical characteristics of the performer.

The performer identifier may be determined to correspond to the performance style of the performer. In this case, appropriate fingering information can be estimated according to the performance style of the performer.

The fingering presentation device 20 may further include a generating unit 23 that generates score information indicating an auxiliary score to which fingering information is added so as to correspond to each note of the note sequence. In this case, the performer can easily recognize the finger corresponding to each note of the note sequence by visually checking the auxiliary score.

The training device 10 according to this embodiment includes a first acquisition unit 11 that acquires input time series data A including a reference note sequence consisting of a plurality of notes, a second acquisition unit 12 that acquires output finger information B indicating the finger to be used when playing each note of the reference note sequence on an instrument, and a construction unit 13 that constructs a trained model M that has acquired the input/output relationship between the input time series data A and the output finger information B. With this configuration, a trained model M that has acquired the input/output relationship between the input time series data A and the output finger information B can be easily constructed.

(6) Other Examples of Training Data In this embodiment, the input time series data A includes a reference performer identifier, and the time series data includes a performer identifier, but the embodiment is not limited to this. The input time series data A only needs to include a reference note sequence, and does not need to include a reference performer identifier. Similarly, the time series data only needs to include a note sequence, and does not need to include a performer identifier.

In addition, in this embodiment, the input time series data A and the output finger information B are described in a so-called action-based manner, which indicates key presses or key releases in the MIDI (Musical Instrument Digital Interface) standard, but the embodiment is not limited to this. The input time series data A and the output finger information B may be described in other formats. For example, the input time series data A and the output finger information B may be described in a so-called note-based manner, which indicates the start position of a note or the length of a note in the MIDI standard. The same applies to the time series data and finger information.

Figure 7 shows another example of input time series data A. The top part of Figure 7 shows input time series data A (Ax) described on an action basis. The middle part of Figure 7 shows input time series data A (Ay) described on a note basis. The input time series data Ax and the input time series data Ay contain the same reference note sequence (the reference note sequence in the musical score shown in the bottom part of Figure 7). "Bar" and "beat" in the input time series data Ax and Ay are elements that indicate the metrical structure of the reference note sequence.

As shown in Figure 7, by describing the input time series data A on a note basis, the length of the input time series data A is shortened. This makes it possible to easily process longer input time series data A. Note that the output fingering information B corresponding to the input time series data A can be described by inserting an element indicating the finger number immediately after the element indicating the pitch number in the input time series data A ("note_○○").

Alternatively, the input time series data A and the output fingering information B may be described in a format that represents musical scores. Details of the input time series data A and the output fingering information B described in a format that represents musical scores will be described in the following modified example.

(7) Modifications Fig. 8 is a diagram showing an example of input time series data A in a modification. The upper part of Fig. 8 shows input time series data A (Az) described in a format for expressing musical scores. The lower part of Fig. 8 shows the musical score represented by the input time series data Az . As shown in the upper part of Fig. 8, the input time series data Az includes a number of elements A0 to A24. Some of the elements have attributes. The attributes of an element are described at the end of the element (after the underscore).

The element A0 indicates the proportion of notes to which fingering is to be applied among the notes included in the reference sequence of notes. The element A0 is placed at the beginning of the input time-series data Az, but may be placed at any position in the input time-series data Az. The proportion is specified by the "fingerrate" attribute of the element A0. In this example, the attribute "5" means a proportion of 100%. The proportion may have a range, for example, 20-40% or 40-60%, or may be divided into multiple ranges.

Element A1 indicates a part. Element A1 is placed immediately after element A0, but may be placed at any position in the input time series data Az. In element A1, "R" and "L" indicate the right hand and left hand parts, respectively. In this example, the element corresponding to the right hand is placed after "R". Then "L" is placed, and after "L" is placed the element corresponding to the left hand. "R" and the element corresponding to the right hand may be placed after the element corresponding to the left hand. If there is no distinction between parts, the input time series data Az does not include element A1.

Elements A2, A15, and A24 represent bar lines in the musical score. Therefore, in the example of Figure 8, the range delimited by "bar" in element A2 and "bar" in element A15 corresponds to the first bar. The range delimited by "bar" in element A15 and "bar" in element A24 corresponds to the second bar.

Element A3 indicates the clef of the musical score. The type of clef is specified by the "clef" attribute of element A3. In the example of Figure 8, since the attribute is "treble", element A3 specifies a treble clef as the clef. If the attribute were "bass", element A3 would specify a bass clef as the clef.

Element A4 indicates the time signature of the musical score. The type of time signature is specified by the "time" attribute in element A4. In the example in Figure 8, since the attribute is "4/4", element A4 specifies "4/4" as the time signature.

Notes in the reference note sequence are indicated by a pair of pitch and value. The pitch is specified by the "note" attribute in elements A5, A9, A11, A13, A16, A18, and A20. The value is specified by the "len" attribute in elements A6, A10, A12, A14, A17, A19, and A21. In this example, "len_1" corresponds to one beat.

The direction of the stem of the notes in the musical score is specified by other attributes of "len" in elements A6, A10, A12, A14, A17, A19, and A21. If the other attribute is "down", the stem extends downward from the note head. If the other attribute is "up", the stem extends upward from the note head. When multiple notes such as eighth notes or sixteenth notes are connected by a beam, the start, intermediate, and end positions of the beam are specified by further attributes of "len" in elements A10, A12, and A14, "start", "continue", and "stop", respectively.

Rests in the reference sequence are specified by the "rest" attribute in elements A7 and A22. The note value of the rest is described by the "len" attribute in elements A8 and A23.

In the example of Figure 8, elements A5 and A6 indicate note N1, and elements A7 and A8 indicate rest R1. Elements A9 and A10 indicate note N2, elements A11 and A12 indicate note N3, and elements A13 and A14 indicate note N4. Elements A16 and A17 indicate note N5, and elements A18 and A19 indicate note N6. Elements A20 and A21 indicate note N7, and elements A22 and A23 indicate rest R2.

Figure 9 is a diagram showing an example of output finger information B in a modified example. The upper part of Figure 9 shows output finger information B (Bz) described in a format for representing musical scores. The output finger information Bz corresponds to the input time series data Az in Figure 8. The lower part of Figure 9 shows the musical score represented by the output finger information Bz.

As shown in the upper part of Figure 9, the output finger information Bz includes a number of elements B0 to B24. The output finger information Bz also includes elements B5f, B9f, B11f, B13f, B16f, B18f, and B20f, which are respectively placed immediately after elements B5, B9, B11, B13, B16, B18, and B20. The elements B0 to B24 are respectively similar to the elements A0 to A24 of the input time series data Az in Figure 8. Therefore, the first acquisition unit 11 in Figure 3 can acquire the input time series data Az by deleting the elements B5f, B9f, B11f, B13f, B16f, B18f, and B20f from the output finger information Bz.

Elements B5f, B9f, B11f, B13f, B16f, B18f, and B20f indicate the finger numbers to be used when playing the notes corresponding to the immediately preceding elements B5, B9, B11, B13, B16, B18, and B20 on an instrument, respectively. The finger numbers are specified by the "finger" attribute in elements B5f, B9f, B11f, B13f, B16f, B18f, and B20f. Therefore, elements B5f, B9f, B11f, B13f, B16f, B18f, and B20f write the finger numbers "1", "1", "2", "1", "3", "3", and "2" to be used when playing notes N1 to N7, respectively, on the musical score, as shown in the lower part of Figure 9.

(8) Effects of the Modification In the modification of the first embodiment, the percentage of notes to which fingering is to be assigned among the notes included in the reference sequence of notes can be arbitrarily specified by the attribute of element A0. When the percentage is 100%, the estimation unit 22 estimates fingering information for all notes included in the sequence of notes. In this case, it is possible to present appropriate fingering when a beginner-level player plays an instrument.

When the ratio is 100%, the generation unit 23 may generate a video file showing the finger movement by animation or the like based on the finger information estimated by the estimation unit 22. This makes it possible to visualize the finger movement. The generation of such a video file may be performed before or after step S13 in the fingering presentation process of FIG. 6, may be performed in parallel with step S13, or may be performed instead of step S13.

On the other hand, when the ratio is less than 100%, the estimation unit 22 estimates some of the notes in the sequence of notes to which fingering is to be assigned and fingering information for the some of the notes. In this case, it is possible to present appropriate fingering when a player at a beginner level or intermediate level higher than the introductory level plays the instrument. In this configuration, the output fingering information Bz does not include some of the elements B5f, B9f, B11f, B13f, B16f, B18f, and B20f.

When the ratio is less than 100%, the estimation unit 22 may estimate note information indicating the note to which fingering is to be assigned from the note sequence without estimating finger information. Details will be described in the third embodiment described later.

[2] Second embodiment (1) Processing system The processing system 100 in the second embodiment will be described with respect to differences from the processing system 100 in the first embodiment. In the training device 10 in Fig. 3, a first acquisition unit 11 and a second acquisition unit 12 acquire input time-series data A and output finger information B of training data D, respectively.

Figure 10 is a diagram showing an example of input time series data A in the second embodiment. The upper part of Figure 10 shows input time series data Az described in a format for representing musical scores. The lower part of Figure 10 shows the musical score represented by the input time series data Az.

As shown in the upper part of Figure 10, the input time series data Az includes multiple elements A0 to A24. The elements A0 to A24 in Figure 10 are respectively similar to the elements A0 to A24 in the modified example of the first embodiment (Figure 8). The input time series data Az also includes additional elements that are placed immediately after parts of the elements A5, A9, A11, A13, A16, A18, and A20 that correspond to the notes. In the example of Figure 10, the input time series data Az further includes elements A5f, A11f, A16f, and A20f that are placed immediately after the elements A5, A11, A16, and A20, respectively.

Elements A5f, A11f, A16f, and A20f are finger information (hereinafter referred to as basic finger information) that indicate the finger numbers to be used when playing the notes corresponding to the immediately preceding elements A5, A11, A16, and A20 on an instrument. The finger numbers are specified by the "finger" attribute in elements A5f, A11f, A16f, and A20f. Therefore, elements A5f, A11f, A16f, and A20f write the finger numbers "1," "2," "3," and "2" to be used when playing notes N1, N3, N5, and N7, respectively, on the musical score, as shown in the lower part of Figure 10.

The output finger information Bz in this embodiment is the same as the output finger information Bz in the modified example of the first embodiment (FIG. 9). Therefore, the first acquisition unit 11 can acquire the input time series data Az by randomly deleting a portion of the elements B5f, B9f, B11f, B13f, B16f, B18f, and B20f from the output finger information Bz. The proportion of the elements B5f, B9f, B11f, B13f, B16f, B18f, and B20f to be deleted can be specified by the user of the training device 10 by operating the operation unit 150 in FIG. 1.

In this example, the input time series data Az is obtained by deleting the elements B9f, B13f, and B18f from the output finger information Bz. The elements B5f, B11f, B16f, and B20f that are not deleted remain as the elements A5f, A11f, A16f, and A20f, which are the basic finger information.

The construction unit 13 in FIG. 3 performs machine learning using the above-mentioned input time series data Az as an input element and the output finger information Bz as an output element. By repeating machine learning for multiple training data D, a trained model M showing the input/output relationship between the input time series data Az and the output finger information Bz is constructed.

In the fingering presentation device 20, the receiving unit 21 receives time series data. The time series data further includes basic fingering information indicating the fingers to be used when playing some of the notes included in the note sequence on an instrument. The estimation unit 22 estimates fingering information indicating the fingers to be used when playing the notes included in the note sequence on an instrument based on the constructed trained model M and the basic fingering information. The generation unit 23 generates musical score information based on the note sequence and fingering information of the time series data.

(2) Effects of the embodiment According to the embodiment, even if fingering information (basic fingering information) for only some of the notes included in the note sequence of the time-series data is known and fingering information for the remaining notes is not given, the fingering information for the remaining notes is supplemented. This makes it possible to present appropriate fingering for beginner-level players when playing an instrument. The generating unit 23 may generate a video file showing finger movements by animation or the like based on the fingering information estimated by the estimating unit 22. In this case, the finger movements can be visualized.

(3) Modification In this embodiment, the estimation unit 22 estimates fingering information for all notes included in the note sequence of the time-series data, but the embodiment is not limited to this. When fingering information is given for a first proportion of notes included in the note sequence, the estimation unit 22 may estimate fingering information for a second proportion of notes included in the note sequence that is greater than the first proportion. In this case, it is possible to present appropriate fingering when a beginner or intermediate level player plays a musical instrument.

In a modified example, the output finger information B of the training data D may not include some of the elements B5f, B9f, B11f, B13f, B16f, B18f, and B20f. For example, if the input time series data Az includes elements A5f, A11f, A16f, and A20f, the output finger information B includes elements B5f, B11f, B16f, and B20f. On the other hand, the output finger information B may not include some of the elements B9f, B13f, and B18f.

[3] Third embodiment (1) Processing system The processing system 100 in the third embodiment will be described with respect to differences from the processing system 100 in the first embodiment. In this embodiment, the training data D indicates a set of input time series data A and output note information. In the training device 10 in Fig. 3, the first acquisition unit 11 and the second acquisition unit 12 acquire the input time series data A and the output note information of the training data D, respectively. The acquisition of the output note information is executed in place of step S2 in the sound learning process in Fig. 5.

The input time series data Az in this embodiment is the same as the input time series data Az in the modified example of the first embodiment (FIG. 8). The first acquisition unit 11 can acquire the input time series data Az by deleting the elements C9f, C11f, and C16f from the output note information Cz in FIG. 11 described later.

Figure 11 is a diagram showing an example of output note information C in the third embodiment. The upper part of Figure 11 shows output note information C (Cz) described in a format for representing musical scores. The lower part of Figure 11 shows the musical score represented by the output note information Cz.

As shown in the upper part of Figure 11, the output note information Cz includes multiple elements C0 to C24. The elements C0 to C24 in Figure 11 are similar to the elements B0 to B24 of the output finger information Bz in the modified example of the first embodiment (Figure 9). In addition, the output note information Cz includes additional elements that are positioned immediately after some of the elements C5, C9, C11, C13, C16, C18, and C20 that correspond to the notes.

In this example, the attribute of "fingerrate" in element C0 is "2", which means a percentage of 40%. Therefore, the output note information Cz further includes elements C9f, C11f, and C16f, which are respectively placed immediately after elements C9, C11, and C16, which are approximately 40% of the elements C5, C9, C11, C13, C16, C18, and C20.

The elements C9f, C11f, and C16f indicate the notes corresponding to the immediately preceding elements C9, C11, and C16, respectively, as the notes to which fingering is to be assigned from the reference note sequence. As shown in the lower part of Figure 11, the elements C9f, C11f, and C16f allow the notes N2, N3, and N5 corresponding to the elements C9, C11, and C16 to be written in the musical score in an identifiable manner.

The construction unit 13 in FIG. 3 performs machine learning using the above-mentioned input time series data Az as an input element and the output note information Cz as an output element. By repeating machine learning for multiple training data D, a trained model M showing the input/output relationship between the input time series data Az and the output note information Cz is constructed.

In the fingering presentation device 20, the reception unit 21 receives time series data. The estimation unit 22 estimates note information indicating the notes to which fingering is to be assigned from the sequence of notes based on the trained model M constructed by the training device 10 and the time series data received by the reception unit 21. The estimation of the note information is performed in place of step S12 in the fingering presentation process of Figure 6. The generation unit 23 generates musical score information indicating an auxiliary musical score in which the notes indicated by the note information are displayed in an identifiable manner.

(2) Advantages of the embodiment According to the embodiment, it is possible to present the notes to which fingering should be assigned from the sequence of notes, thereby enabling beginner or intermediate level players to recognize the key notes when playing an instrument.

(3) Modification The estimation unit 22 may estimate fingering information indicating fingers to be used when playing some notes included in a sequence of notes on an instrument, using the first trained model M constructed in the first embodiment and the second trained model M constructed in the present embodiment. Fig. 12 is a flowchart showing an example of a fingering suggestion process in the modification.

First, the reception unit 21 receives time series data (step S21). Next, the estimation unit 22 estimates intermediate finger information from the time series data received in step S11 using the first trained model M constructed in the first embodiment (step S22). The intermediate finger information indicates the fingers to be used when playing each note included in the sequence of notes on an instrument.

In addition, the estimation unit 22 estimates note information from the time-series data accepted in step S21 by using the second trained model M constructed in this embodiment (step S23). Steps S22 and S23 may be executed either first or simultaneously.

Next, the estimation unit 22 estimates fingering information for notes included in the note sequence other than the notes indicated by the note information estimated in step S23 based on the intermediate fingering information estimated in step S22 (step S24). The generation unit 23 then generates musical score information based on the note sequence of the time-series data accepted in step S21 and the fingering information estimated in step S24 (step S25). This ends the fingering presentation process.

In this fingering presentation process, the intermediate finger information estimated in step S22 has a similar configuration to the output finger information Bz of the modified example in the first embodiment (Figure 9), for example. Also, the note information estimated in step S23 has a similar configuration to the output note information Cz in Figure 11. Figure 13 is a diagram showing an example of finger information estimated in step S24 of the fingering presentation process.

The upper part of Fig. 13 shows fingering information F (Fz) described in a method for representing musical scores. The lower part of Fig. 13 shows auxiliary musical scores represented by fingering information Fz. Fingering information Fz is estimated by deleting elements B9f, B11f, and B16f, which respectively correspond to elements C9f, C11f, and C16f indicating the notes to which fingering is to be assigned in the note information (see Fig. 11), from the intermediate fingering information (see Fig. 9).

Specifically, as shown in the upper part of Fig. 13, finger information Fz includes a number of elements F1 to F24. Elements F1 to F24 in Fig. 13 are similar to elements B1 to B24 of output finger information Bz in the modified example of the first embodiment (Fig. 9), respectively. Finger information Fz also includes additional elements arranged immediately after parts of elements F5, F9, F11, F13, F16, F18, and F20 corresponding to notes. In this example, finger information Fz further includes elements F5f, F13f, F18f, and F20f arranged immediately after elements F5, F13, F18, and F20, respectively.

Elements F5f, F13f, F18f, and F20f indicate the finger numbers to be used when playing the notes corresponding to the immediately preceding elements F5, F13, F18, and F20 on an instrument, respectively. The finger numbers are specified by the "finger" attribute in elements F5f, F13f, F18f, and F20f. Therefore, elements F5f, F13f, F18f, and F20f write the finger numbers "1", "1", "3", and "2" to be used when playing notes N1, N4, N6, and N7, respectively, in the auxiliary score, as shown in the lower part of Figure 13.

According to the modified example, fingering information for some notes is thinned out from fingering information for all notes included in the sequence of notes in the time-series data. In this case, it is possible to present appropriate fingering when a beginner or intermediate level player plays an instrument. For example, fingering information for key notes when playing an instrument is thinned out, so that a beginner or intermediate level player can develop the ability to judge appropriate fingering when practicing an instrument.

[4] Other embodiments In the above embodiment, the fingering presentation device 20 includes the generation unit 23, but the embodiment is not limited to this. A performer can create an auxiliary score by transcribing the fingering information estimated by the estimation unit 22 to a desired score. Therefore, the fingering presentation device 20 does not need to include the generation unit 23.

In the above embodiment, the training data D is trained to estimate finger information when playing a piano, but the embodiment is not limited to this. The training data D may be trained to estimate finger information when playing other instruments such as drums.

In the above embodiment, the user of the fingering presentation device 20 is a performer, but the user of the fingering presentation device 20 may be, for example, a staff member of a music score production company. Furthermore, machine learning by the training device 10 may be performed in advance by a staff member of the music score production company.

Claims (13)

A reception unit that receives time series data including a musical note sequence consisting of a plurality of musical notes;
an estimation unit that estimates fingering information indicating fingers to be used when playing at least some of the notes included in the sequence of notes on a musical instrument using a trained model,
The time-series data further includes a performer identifier indicating a performer who performs the sequence of notes,
The estimation unit estimates the fingering information based on the player identifier ,
the trained model is a machine learning model that has learned an input/output relationship between input time-series data including a reference sequence of notes, and output fingering information indicating fingers to be used when playing at least some of the notes included in the reference sequence of notes on a musical instrument;
the input time-series data further includes a reference performer identifier indicating a reference performer who performs the reference sequence of notes;
The output fingering information indicates the fingers of the reference player to be used when playing a note on the musical instrument . The estimation unit further estimates note information indicating a note to which fingering is to be assigned from the sequence of notes using the trained model;
The fingering guidance presentation device according to claim 1, wherein the trained model is a machine learning model that further acquires an input/output relationship between the input time series data and output note information indicating notes to which fingerings are to be assigned from the reference note sequence. The fingering suggestion device according to claim 1 or 2, wherein the performer identifier is determined to correspond to the physical characteristics of the performer. The fingering presentation device according to any one of claims 1 to 3, wherein the performer identifier is determined to correspond to the performance style of the performer. A reception unit that receives time series data including a musical note sequence consisting of a plurality of musical notes;
an estimation unit that estimates fingering information indicating fingers to be used when playing at least some of the notes included in the sequence of notes on a musical instrument using a trained model,
the time-series data further includes basic fingering information indicating fingers to be used when playing a first proportion of notes included in the sequence of notes on a musical instrument,
the estimation unit estimates, based on the basic fingering information, the fingering information indicating a finger to be used when playing notes having a second proportion greater than the first proportion among the notes included in the sequence of notes , using an instrument;
the trained model is a machine learning model that has learned an input/output relationship between input time-series data including a reference sequence of notes, and output fingering information indicating fingers to be used when playing at least some of the notes included in the reference sequence of notes on a musical instrument;
the input time-series data further includes the basic fingering information indicating fingers to be used when playing a first proportion of notes included in the reference sequence of notes on the musical instrument,
The output fingering information further includes the basic fingering information included in the input time-series data . A reception unit that receives time series data including a musical note sequence consisting of a plurality of musical notes;
an estimation unit that estimates fingering information indicating fingers to be used when playing at least some of the notes included in the sequence of notes on a musical instrument using a trained model, and note information indicating notes to which fingerings are to be assigned from the sequence of notes,
The fingering presentation device includes an estimation unit that estimates intermediate fingering information indicating the fingers to be used when playing each note included in the sequence of notes on an instrument, and the note information , and estimates the fingering information indicating the fingers to be used when playing notes included in the sequence of notes on an instrument other than the notes indicated by the note information, by deleting from the intermediate fingering information finger information corresponding to the notes to which fingering in the note information is to be assigned. The fingering presentation device according to any one of claims 1 to 6, further comprising a generation unit that generates musical score information indicating a first auxiliary musical score to which the fingering information is assigned so as to correspond to at least some of the notes included in the sequence of notes. 7. The fingering guidance suggestion device according to claim 2, further comprising a generating unit that generates score information representing a second auxiliary score on which the notes represented by the note information are displayed in an identifiable manner. A first acquisition unit that acquires input time series data including a reference note sequence consisting of a plurality of notes;
a second acquisition unit that acquires output fingering information indicating fingers to be used when playing at least some of the notes included in the reference sequence of notes on a musical instrument, or output note information indicating notes to which fingerings are to be assigned from the reference sequence of notes;
a construction unit that constructs a trained model that has acquired an input/output relationship between the input time-series data and the output fingering information or the output note information,
the input time-series data further includes a reference performer identifier indicating a reference performer who performs the reference sequence of notes;
The output fingering information indicates the reference player's fingers to use when playing on the musical instrument. Accepts time series data including a sequence of multiple notes,
using the trained model to estimate fingering information indicative of fingers to be used when playing at least some of the notes in the sequence of notes on a musical instrument;
The time-series data further includes a performer identifier indicating a performer who performs the sequence of notes,
estimating the finger information includes estimating the finger information based on the player identifier,
the trained model is a machine learning model that has learned an input/output relationship between input time-series data including a reference sequence of notes, and output fingering information indicating fingers to be used when playing at least some of the notes included in the reference sequence of notes on a musical instrument;
the input time-series data further includes a reference performer identifier indicating a reference performer who performs the reference sequence of notes;
the output fingering information indicates the fingers of the reference player to be used when playing a note on the musical instrument;
A computer-implemented method for providing fingering instructions. Accepts time series data including a sequence of multiple notes,
using the trained model to estimate fingering information indicative of fingers to be used when playing at least some of the notes in the sequence of notes on a musical instrument;
the time-series data further includes basic fingering information indicating fingers to be used when playing a first proportion of notes included in the sequence of notes on a musical instrument,
estimating the fingering information includes estimating, based on the basic fingering information, fingering information indicating fingers to be used when playing notes having a second proportion, which is greater than the first proportion, among the notes included in the sequence of notes, on a musical instrument; and
the trained model is a machine learning model that has learned an input/output relationship between input time-series data including a reference sequence of notes, and output fingering information indicating fingers to be used when playing at least some of the notes included in the reference sequence of notes on a musical instrument;
the input time-series data further includes the basic fingering information indicating fingers to be used when playing a first proportion of notes included in the reference sequence of notes on the musical instrument,
The output move information further includes the basic move information included in the input time-series data,
A computer-implemented method for providing fingering instructions. Accepts time series data including a sequence of multiple notes,
using the trained model, estimating fingering information indicating fingers to be used when playing at least some of the notes included in the sequence of notes on a musical instrument, and note information indicating notes to which fingerings are to be assigned from the sequence of notes;
estimating the fingering information includes estimating intermediate fingering information indicating fingers to be used when playing each note included in the sequence of notes on a musical instrument, and the note information, and deleting fingering information corresponding to a note to which fingering in the note information is to be assigned from the intermediate fingering information, thereby estimating the fingering information indicating fingers to be used when playing notes included in the sequence of notes, other than the notes indicated by the note information, on a musical instrument,
A computer-implemented method for providing fingering instructions. Obtaining input time series data including a reference sequence of a plurality of notes;
obtain output fingering information indicating fingers to be used when playing at least some of the notes included in the reference sequence of notes on a musical instrument, or output note information indicating notes to which fingering is to be assigned from the reference sequence of notes;
Constructing a trained model that has learned the input/output relationship between the input time-series data and the output fingering information or the output note information;
the input time-series data further includes a reference performer identifier indicating a reference performer who performs the reference sequence of notes;
the output fingering information indicates fingers of the reference player to be used when playing the musical instrument,
A computer-implemented training method.

Images