JP7577964B2 - Reproduction control method and reproduction control system - Google Patents

Description

This disclosure relates to technology for controlling sound playback.

Technology has been proposed for the past that allows the playback of a piece of music to follow the performance of a performer. For example, Patent Document 1 discloses a technology that estimates the performance position within a piece of music by analyzing musical tones produced by the performance of the piece of music, and controls the automatic performance of the piece of music based on the estimation result.

JP 2017-207615 A

In some cases, a performer may pause during the performance of a piece of music to reflect the performer's intention (e.g., musical expression) or a performance mark such as a fermata. The length of the pause may vary from performer to performer or from performance to performance by a performer, depending on the performer's intention or preference. Therefore, in playing a piece of music, it is difficult to appropriately control the interval between two sounds located before and after the pause in accordance with the performer's intention or preference. In consideration of the above circumstances, one of the objectives of one aspect of the present invention is to appropriately control the interval between two adjacent sounds in the playback of multiple sounds.

In order to solve the above problems, a playback control method according to one aspect of the present disclosure is a method for playing a plurality of sounds using sound data representing a time series of the sounds, in which a first sound among the plurality of sounds that is being played at the time a first instruction is issued is continued until the end point of the first sound represented by the sound data, and after playback of the first sound is stopped, playback of a second sound among the plurality of sounds that is immediately after the first sound is started in response to a second instruction from the user.

A playback control method according to another aspect of the present disclosure is a method for playing a plurality of sounds using sound data representing a time sequence of the plurality of sounds constituting a piece of music, estimating a point in time in the piece of music at which a user is playing the plurality of sounds in parallel with the playback of the plurality of sounds, and causing the playback of the plurality of sounds to follow the performance of the piece of music according to the result of the estimation, and in the playback of the plurality of sounds, playing a first sound among the plurality of sounds that corresponds to a first instruction, and after the playback of the first sound has stopped, starting the playback of a second sound among the plurality of sounds immediately after the first sound in response to a second instruction from the user.

A playback control method according to another aspect of the present disclosure is a method for playing back a plurality of sounds using sound data representing a time series of the sounds, in which a first sound among the plurality of sounds that is being played back at the time a first instruction is issued is continued until the end point of the first sound represented by the sound data.

1 is a block diagram illustrating a configuration of a playback system according to a first embodiment. FIG. 2 is a schematic diagram of music data. 2 is an explanatory diagram of the configuration and state of an operating device. FIG. FIG. 2 is a block diagram illustrating a functional configuration of the playback control system. 11 is an explanatory diagram of the relationship between the playback of a playback part by a playback device and a first instruction and a second instruction. FIG. 10 is a flowchart illustrating a specific procedure of a playback control process. 13 is an explanatory diagram relating to a state of an operating device in a second embodiment. FIG. 13 is an explanatory diagram relating to a state of an operating device in a third embodiment. FIG. FIG. 13 is a block diagram illustrating a functional configuration of a playback control system according to a fourth embodiment. FIG. 4 is an explanatory diagram of the operation of the edit processing unit. 11 is a flowchart illustrating a specific procedure of an editing process. FIG. 13 is a block diagram illustrating the configuration of a playback system according to a fifth embodiment. FIG. 13 is a block diagram illustrating a functional configuration of a playback control system according to a fifth embodiment. 11 is a flowchart illustrating a specific procedure of a reference data generation process. FIG. 13 is a schematic diagram of a preparation screen displayed in parallel with the reference data generation process. FIG. 13 is a schematic diagram of a playback screen displayed in parallel with the playback control process.

A: First embodiment FIG. 1 is a block diagram illustrating the configuration of a playback system 100 according to a first embodiment. The playback system 100 is installed in an acoustic space in which a user U is located. The user U is a performer who plays a specific part of a piece of music (hereinafter referred to as a "performance part") using a musical instrument 200 such as a string instrument. The playback system 100 is a computer system that plays the piece of music in parallel with the performance of the performance part by the user U. Specifically, the playback system 100 plays a part other than the performance part (hereinafter referred to as a "playback part") among a plurality of parts that make up a piece of music. The performance part is, for example, a part that makes up the main melody of the piece of music. The playback part is, for example, one or more parts that make up the accompaniment of the piece of music. As can be understood from the above description, the performance of the performance part by the user U and the playback of the playback part by the playback system 100 are executed in parallel to realize the performance of the piece of music. The performance part and the playback part may be a common part of the piece of music.

The playback system 100 includes a playback control system 10 and a playback device 20. The playback control system 10 and the playback device 20 are configured separately and communicate with each other via wired or wireless communication. Note that the playback control system 10 and the playback device 20 may be configured as an integrated unit.

The playback device 20 plays the playback part of the music piece under the control of the playback control system 10. Specifically, the playback device 20 is an automatic performance instrument that performs automatic performance of the playback part. For example, an automatic performance instrument (e.g., an automatic performance piano) of a different type from the instrument 200 played by the user U is used as the playback device 20. As can be understood from the above explanation, automatic performance is one aspect of "playback."

The playback device 20 of the first embodiment includes a drive mechanism 21 and a sound generation mechanism 22. The sound generation mechanism 22 is a mechanism that generates musical tones. Specifically, the sound generation mechanism 22 includes a string-striking mechanism for each key that generates sound from a string (sound source) in response to the displacement of each key on the keyboard, similar to a natural musical instrument keyboard. The drive mechanism 21 drives the sound generation mechanism 22 to perform automatic performance of a musical piece. The drive mechanism 21 drives the sound generation mechanism 22 in response to instructions from the playback control system 10, thereby realizing automatic performance of the playback part.

The playback control system 10 is a computer system that controls the playback of the playback part by the playback device 20, and includes a control device 11, a storage device 12, a sound collection device 13, and an operation device 14. The playback control system 10 is realized by, for example, a portable terminal device such as a smartphone or a tablet terminal, or a portable or stationary terminal device such as a personal computer. Note that the playback control system 10 can be realized by a single device, or by multiple devices configured separately from each other.

The control device 11 is a single or multiple processors that control each element of the playback control system 10. Specifically, the control device 11 is configured with one or more types of processors, such as a CPU (Central Processing Unit), an SPU (Sound Processing Unit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or an ASIC (Application Specific Integrated Circuit).

The storage device 12 is a single or multiple memories that store the programs executed by the control device 11 and various data used by the control device 11. The storage device 12 is composed of a known recording medium, such as a magnetic recording medium or a semiconductor recording medium, or a combination of multiple types of recording media. In addition, a portable recording medium that is detachable from the playback control system 10, or a recording medium that can be written to or read from via a communication network (e.g., cloud storage) may be used as the storage device 12.

The storage device 12 stores, for each piece of music, music data M that specifies the time sequence of the multiple notes that make up the piece of music. FIG. 2 is a schematic diagram of the music data M. The music data M includes reference data R and performance data D. The reference data R specifies the time sequence of the notes of the performance part played by the user U. Specifically, the reference data R specifies the pitch and sound duration for each of the multiple notes of the performance part. On the other hand, the performance data D specifies the time sequence of the notes of the playback part played by the playback device 20. Specifically, the performance data D specifies the pitch and sound duration for each of the multiple notes of the playback part. Each of the reference data R and the performance data D is time-series data in a Musical Instrument Digital Interface (MIDI) format in which, for example, instruction data that instructs the sounding or muting of a musical tone and time data that specifies the time point of the action instructed by the instruction data are arranged in time series. The instruction data, for example, specifies the pitch and intensity to instruct an action such as sounding or muting. The time data, for example, specifies the interval between adjacent instruction data. The period from when instruction data instructs the sounding of a particular pitch to when subsequent instruction data instructs the muting of that pitch is the sounding period for the note of that pitch. Performance data D is an example of "sound data" that represents a time series of multiple notes. The multiple notes of the playback part represented by performance data D are an example of "multiple sounds" represented by "sound data".

The sound collection device 13 in FIG. 1 collects musical sounds produced by the musical instrument 200 when played by the user U, and generates an audio signal Z representing the waveform of the musical sound. For example, a microphone is used as the sound collection device 13. For convenience, an A/D converter that converts the audio signal Z generated by the sound collection device 13 from analog to digital is omitted from the illustration. In the first embodiment, a configuration in which the sound collection device 13 is mounted on the playback control system 10 is illustrated, but a sound collection device 13 separate from the playback control system 10 may be connected to the playback control system 10 by wire or wirelessly. In addition, the playback control system 10 may receive an output signal from an electric instrument such as an electric string instrument as the audio signal Z. As can be understood from the above explanation, the sound collection device 13 may be omitted from the playback control system 10.

The operation device 14 is an input device that accepts instructions from the user U. As illustrated in FIG. 3, the operation device 14 of the first embodiment includes a movable part 141 that moves when operated by the user U. The movable part 141 is an operation pedal that the user U can operate with his/her foot. For example, a pedal-type MIDI controller is used as the operation device 14. The user U can operate the operation device 14 at a desired point in time while playing the musical instrument 200 with both hands. Note that a touch panel that detects contact by the user U may also be used as the operation device 14.

The operating device 14 transitions from one of the released state and the operating state to the other in response to the operation by the user U. The released state is a state in which the operating device 14 is not being operated by the user U. Specifically, the released state is a state in which the user U is not stepping on the movable part 141. The released state can also be expressed as a state in which the movable part 141 is in position H1. On the other hand, the operating state is a state in which the operating device 14 is being operated by the user U. Specifically, the operating state is a state in which the user U has stepped on the movable part 141. The operating state can also be expressed as a state in which the movable part 141 is in position H2, which is different from position H1. The released state is an example of a "first state", and the operating state is an example of a "second state".

Figure 4 is a block diagram illustrating the functional configuration of the playback control system 10. The control device 11 executes a program stored in the storage device 12 to realize multiple functions (performance analysis unit 31, playback control unit 32, and instruction reception unit 33) for controlling the playback of the playback part by the playback device 20.

The performance analysis unit 31 estimates a performance position X in the music piece by analyzing the audio signal Z supplied from the sound collection device 13. The performance position X is the point in time in the music piece where the user U is currently playing. The estimation of the performance position X is performed repeatedly in parallel with the performance of the performance part by the user U and the playback of the playback part by the playback device 20. In other words, the performance position X is estimated at each of multiple points in time on the time axis. The performance position X moves backward in the music piece as time passes.

Specifically, the performance analysis unit 31 calculates the performance position X by comparing the reference data R of the music data M with the audio signal Z. Any known analysis technique (score alignment technique) may be adopted for the performance analysis unit 31 to estimate the performance position X. For example, the analysis technique disclosed in JP 2016-099512 A is used to estimate the performance position X. The performance analysis unit 31 may also estimate the performance position X using a statistical estimation model such as a deep neural network or a hidden Markov model.

The playback control unit 32 causes the playback device 20 to play back each note specified by the performance data D. That is, the playback control unit 32 causes the playback device 20 to automatically play the playback part. Specifically, the playback control unit 32 moves the position Y in the music piece to be played back (hereinafter referred to as the "playback position") backward over time, and sequentially supplies instruction data in the performance data D that corresponds to the playback position Y to the playback device 20. That is, the playback control unit 32 functions as a sequencer that sequentially supplies each instruction data included in the performance data D to the playback device 20. The process in which the playback control unit 32 causes the playback device 20 to play the playback part is executed in parallel with the performance of the performance part by the user U.

The playback control unit 32 makes the playback of the playback part by the playback device 20 follow the performance of the music piece by the user U according to the result of the estimation of the performance position X by the performance analysis unit 31. That is, the automatic performance of the playback part by the playback device 20 proceeds at the same tempo as the performance of the performance part by the user U. For example, if the progression of the performance position X (i.e., the performance speed by the user U) is fast, the playback control unit 32 increases the speed of progression of the playback position Y (the playback speed by the playback device 20) and if the progression of the performance position X is slow, the playback control unit 32 decreases the speed of progression of the playback position Y. That is, the automatic performance of the playback part is executed at the same performance speed as the performance by the user U so as to synchronize with the progression of the performance position X. Therefore, the user U can play the performance part with the feeling that the playback device 20 is playing the playback part in time with his/her performance.

As described above, in the first embodiment, the playback of multiple notes in the playback part follows the performance of the instrument 200 by the user U, so that the intentions (e.g., performance expression) or preferences of the user U can be appropriately reflected in the playback of the playback part.

The instruction receiving unit 33 receives a first instruction Q1 and a second instruction Q2 from the user U. The first instruction Q1 and the second instruction Q2 are generated by the user U's operation of the operation device 14. The first instruction Q1 is an instruction to temporarily stop the playback of the playback part by the playback device 20. The second instruction Q2 is an instruction to resume the playback of the playback part stopped by the first instruction Q1.

Specifically, the instruction receiving unit 33 receives an operation by the user U to transition the operating device 14 from a released state to an operating state as the first instruction Q1. That is, the user U gives the first instruction Q1 to the playback control system 10 by stepping on the movable part 141 of the operating device 14. For example, the instruction receiving unit 33 identifies the time when the movable part 141 starts to move from position H1 (released state) to position H2 (operated state) as the time of the first instruction Q1. Note that a configuration in which the instruction receiving unit 33 identifies the time when the movable part 141 reaches a position midway between position H1 and position H2 as the time of the first instruction Q1, or a configuration in which the instruction receiving unit 33 identifies the time when the movable part 141 reaches position H2 as the time of the first instruction Q1 is also envisioned.

The instruction receiving unit 33 also receives, as the second instruction Q2, an operation by the user U to transition the operating device 14 from an operating state to a released state. That is, the user U gives the second instruction Q2 to the playback control system 10 by releasing the movable part 141 of the operating device 14 from a state in which the movable part 141 is depressed. For example, the instruction receiving unit 33 identifies the time when the movable part 141 starts to move from position H2 (operated state) toward position H1 (released state) as the time of the second instruction Q2. Note that a configuration in which the instruction receiving unit 33 identifies the time when the movable part 141 reaches a position midway between position H2 and position H1 as the time of the second instruction Q2, or a configuration in which the instruction receiving unit 33 identifies the time when the movable part 141 reaches position H1 as the time of the second instruction Q2, is also conceivable.

The user U can give the first instruction Q1 and the second instruction Q2 at any time during the performance of the performance part. Therefore, the interval between the first instruction Q1 and the second instruction Q2 is a variable length according to the user U's intention. For example, the user U gives the first instruction Q1 before the start of a rest period in the music piece, and gives the second instruction Q2 when the rest period of the user U's desired length has elapsed.

Figure 5 is an explanatory diagram of the relationship between the playback of the playback part by the playback device 20 and the first instruction Q1 and the second instruction Q2. Figure 5 shows both the sounding period of each note specified by the performance data D and the sounding period of each note actually played by the playback device 20.

Note N1 in FIG. 5 is a note that corresponds to the first instruction Q1 among the multiple notes specified by the performance data D. Specifically, note N1 is a note that is being played by the playback device 20 at the time of the first instruction Q1 among the multiple notes in the playback part. After the first instruction Q1 is generated, the playback control unit 32 causes the playback device 20 to continue playing note N1 until the end of the sounding period specified for note N1 by the performance data D. For example, the playback control unit 32 supplies instruction data to the playback device 20 to mute note N1 at the end of the sounding period of note N1. As can be understood from the above explanation, the playback of note N1 does not stop immediately at the time of the first instruction Q1, but continues until the end point specified by the performance data D even after the generation of the first instruction Q1. Note that note N1 is an example of a "first note".

Note N2 in FIG. 5 is the note immediately after note N1 among the multiple notes specified by the performance data D. After the playback of note N1 is stopped, the playback control unit 32 causes the playback device 20 to start playing note N2 in response to the second instruction Q2 from the user U. That is, regardless of the position of the start point of the sounding period specified for note N2 by the performance data D and the time length of the interval between notes N1 and N2 specified by the performance data D, playback of note N2 is started on the condition that the second instruction Q2 is generated. Specifically, when the instruction receiving unit 33 receives the second instruction Q2, the playback control unit 32 supplies the instruction data of note N2 in the performance data D to the playback device 20. Therefore, playback of note N2 is started immediately after the second instruction Q2. Note N2 is an example of a "second note".

Figure 6 is a flowchart illustrating the specific steps of the operation (hereinafter referred to as "playback control process") Sa in which the control device 11 controls the playback device 20. The playback control process Sa is started in response to an instruction from the user U.

When the playback control process Sa is started, the control device 11 determines whether the standby data W is in a valid state (Sa1). The standby data W is data (e.g., a flag) that indicates that playback of the playback part has been temporarily stopped by the first instruction Q1, and is stored in the storage device 12. Specifically, the standby data W is set to a valid state (e.g., W=1) when the first instruction Q1 is generated, and is set to an invalid state (e.g., W=0) when the second instruction Q2 is generated. The standby data W can also be said as data that indicates a state of waiting for the playback of the playback part to resume.

If the standby data W is not valid (Sa1: NO), the control device 11 (performance analysis unit 31) estimates the performance position X by analyzing the audio signal Z supplied from the sound collection device 13 (Sa2). The control device 11 (playback control unit 32) causes the playback device 20 to proceed with the playback of the playback part according to the result of the estimation of the performance position X (Sa3). In other words, the control device 11 controls the playback of the playback part by the playback device 20 to follow the performance of the performance part by the user U.

The control device 11 (instruction receiving unit 33) judges whether or not the first instruction Q1 has been received from the user U (Sa4). If the first instruction Q1 has been received (Sa4: YES), the control device 11 (playback control unit 32) causes the playback device 20 to continue playing the note N1 that is being played at the time of receiving the first instruction Q1 until the end point of the sounding period specified by the performance data D (Sa5). Specifically, the control device 11 advances the playback position Y at the same speed (tempo) as when the first instruction Q1 was generated, and when the playback position Y reaches the end point of the sounding period of the note N1, it supplies the playback device 20 with instruction data representing the muting of the note N1. After executing the above process, the control device 11 changes the standby data W from an invalid state to a valid state (W=1) (Sa6). Note that the standby data W may be updated (Sa6) before executing step Sa5.

When the standby data W is set to a valid state, the result of the judgment in step Sa1 becomes positive. When the standby data W is in a valid state (Sa1: YES), the estimation of the performance position X (Sa2), the playback control of the playback part (Sa3), and the processing related to the note N1 (Sa4-Sa6) are not executed. In other words, the playback control of the playback part linked to the performance position X is stopped in response to the first instruction Q1 from the user U. Also, if the first instruction Q1 is not accepted (Sa4: NO), the processing related to the note N1 (Sa5, Sa6) is not executed.

The control device 11 (instruction receiving unit 33) judges whether or not the second instruction Q2 has been received from the user U (Sa7). If the second instruction Q2 has been received (Sa7: YES), the control device 11 (playback control unit 32) causes the playback device 20 to play the note N2 immediately after the note N1 (Sa8). Specifically, the control device 11 updates the playback position Y to the start point of the note N2. That is, the playback of the playback part stopped by the first instruction Q1 is resumed by the second instruction Q2. The control device 11 changes the standby data W from a valid state to an invalid state (W=0) (Sa9). As described above, when the standby data W is set to an invalid state, the result of the judgment in step Sa1 becomes negative. Therefore, the second instruction Q2 is used as a trigger to resume the estimation of the performance position X (Sa2) and the playback control of the playback part (Sa3). Note that the update of the standby data W (Sa8) may be executed before the execution of step Sa8.

The control device 11 determines whether or not to end playback of the playback part by the playback device 20 (Sa10). For example, when playback of the playback part has been completed up to the end point, or when an end command is received from the user U, the control device 11 determines to end playback of the playback part. If playback of the playback part is not to be ended (Sa10: NO), the control device 11 transitions the process to step Sa1 and repeats the processes exemplified above (Sa1-Sa9). On the other hand, when the control device 11 determines that playback of the playback part is to be ended (Sa10: YES), the playback control process Sa is ended.

As described above, in the first embodiment, the note N1 corresponding to the first instruction Q1 is played, and after the playback of note N1 is stopped, the second instruction Q2 by the user U triggers the start of playback of note N2 immediately following note N1. Therefore, the interval between the playback of note N1 and the playback of note N2 (e.g., the duration of the rest period in the music piece) can be changed according to the points in time of the first instruction Q1 and the second instruction Q2.

In addition, in the first embodiment, the playback of note N1 that is being played at the time the first instruction Q1 is generated continues even after the first instruction Q1 is generated until the end point of note N1 specified by performance data D. Therefore, compared to a configuration in which the playback of note N1 is stopped at the time the first instruction Q1 is generated, it is possible to appropriately continue the playback of note N1 according to the content of performance data D.

In the first embodiment, the user U can change the interval between the notes N1 and N2 to an appropriate time length according to the user U's intention or preference by operating the operation device 14. In particular, in the first embodiment, the first instruction Q1 is generated by transitioning the operation device 14 from a released state to an operating state, and after the operating state is maintained, the second instruction Q2 is generated by transitioning the operation device 14 from the operating state to the released state at a desired time point after the generation of the first instruction Q1. That is, the first instruction Q1 and the second instruction Q2 are generated by a series of operations that transition the operation state from the released state to the operating state and then transition back to the released state. Therefore, the operation of the user U on the operation device 14 is simplified compared to a configuration in which an operation to transition the operation device 14 from the released state to the operating state is required for each of the first instruction Q1 and the second instruction Q2.

B: Second embodiment A second embodiment will be described. In each of the following exemplary embodiments, elements that have the same functions as those in the first embodiment will be denoted by the same reference numerals as those used in the description of the first embodiment, and detailed descriptions of each will be omitted as appropriate.

In the first embodiment, when the first instruction Q1 is generated, the playback position Y is advanced at the same speed as at the time of the first instruction Q1, and when the playback position Y reaches the end point of the note N1, the playback of the note N1 is stopped. In the second embodiment, the playback control unit 32 variably controls the progression speed of the playback position Y after the generation of the first instruction Q1 (i.e., the playback speed of the playback part) according to the operation speed V1 of the movable part 141. The operation speed V1 is the speed at which the movable part 141 moves from the position H1 corresponding to the released state to the position H2 corresponding to the operated state. For example, the operation speed V1 is the average value of multiple speeds calculated during the period from the position H1 to the position H2 of the movable part 141.

Figure 7 is an explanatory diagram regarding the state of the operation device 14 in the second embodiment. As illustrated in Figure 7, the instruction receiving unit 33 receives a first instruction Q1 at the time when the movable part 141 starts to move from position H1 to position H2. The playback control unit 32 controls the progress speed of the playback position Y after the first instruction Q1 is generated according to the operation speed V1 of the movable part 141.

Specifically, the playback control unit 32 increases the progression speed of the playback position Y as the operation speed V1 becomes faster. For example, as illustrated in FIG. 7, the progression speed of the playback position Y when the operation speed V1 is speed V1_H exceeds the progression speed of the playback position Y when the operation speed V1 is speed V1_L (V1_L < V1_H). Therefore, the faster the operation speed V1 is, the shorter the duration of the note N1 is. For example, the duration of the note N1 when the operation speed V1 is speed V1_H is shorter than the duration of the note N1 when the operation speed V1 is speed V1_L.

The second embodiment also achieves the same effect as the first embodiment. In the second embodiment, the duration of note N1 is controlled according to the operation speed V1, which has the advantage that the user U can adjust the duration of note N1. In addition, in the second embodiment, the operation device 14 for giving the first instruction Q1 and the second instruction Q2 is also used to adjust the duration of note N1. Therefore, there is also the advantage that the operation by user U is simplified compared to a configuration in which user U operates separate devices for each of giving the first instruction Q1 and the second instruction Q2 and adjusting the duration of note N1.

C: Third embodiment In the first embodiment, the reproduction of the note N2 is started immediately after the second instruction Q2. In the second embodiment, the time from the second instruction Q2 to the start of reproduction of the note N2 (hereinafter referred to as the "delay time") is variably controlled according to the operation speed V2. The operation speed V2 is the speed at which the movable part 141 moves from the position H2 corresponding to the operation state to the position H1 corresponding to the release state. For example, the average value of multiple speeds calculated during the period from the position H2 to the position H1 of the movable part 141 is the operation speed V2.

Figure 8 is an explanatory diagram of the state of the operation device 14 in the third embodiment. As illustrated in Figure 8, the instruction receiving unit 33 receives a second instruction Q2 at the time when the movable unit 141 starts to move from position H2 to position H1. The playback control unit 32 variably controls the delay time according to the operation speed V2.

Specifically, the playback control unit 32 shortens the delay time as the operation speed V2 becomes faster. For example, as illustrated in FIG. 8, the delay time when the operation speed V2 is speed V2_L is longer than the delay time when the operation speed V2 is speed V2_H (V2_H>V2_L). Therefore, the slower the operation speed V2 is, the later on the time axis the point at which playback of note N2 starts.

The third embodiment also achieves the same effect as the first embodiment. In the third embodiment, the time when playback of note N2 starts is controlled according to the operation speed V2, so there is an advantage that the user U can adjust the start point of the first note N2 after the playback part is resumed. In addition, in the third embodiment, the operation device 14 for giving the first instruction Q1 and the second instruction Q2 is also used to adjust the start point of note N2. Therefore, there is an advantage that the operation by the user U is simplified compared to a configuration in which the user U operates separate devices for each of giving the first instruction Q1 and the second instruction Q2 and adjusting the start point of note N2. Note that the configuration of the second embodiment may be applied to the third embodiment.

D: Fourth embodiment Fig. 9 is a block diagram illustrating the functional configuration of the playback control system 10 in the fourth embodiment. The control device 11 in the fourth embodiment functions as an edit processing unit 34 in addition to the same elements as in the first embodiment (performance analysis unit 31, playback control unit 32, and instruction receiving unit 33). The edit processing unit 34 edits the performance data D stored in the storage device 12 in response to an instruction from the user U. The operation of the elements other than the edit processing unit 34 is the same as in the first embodiment. Therefore, the fourth embodiment also achieves the same effect as the first embodiment. Note that the configuration of the second or third embodiment may be applied to the fourth embodiment.

Figure 10 is an explanatory diagram of the operation of the editing processing unit 34. Figure 10 shows notes N1 and N2 specified by performance data D of the playback part. As in each of the above-mentioned forms, the first instruction Q1 and the second instruction Q2 are generated at any time by the user U. Therefore, a time difference L occurs between the start point of note N2 specified by performance data D and the time of second instruction Q2. The editing processing unit 34 edits the performance data D so as to reduce the time difference L.

Figure 11 is a flow chart illustrating the specific steps of the process Sb (hereinafter referred to as the "edit process") in which the edit processing unit 34 edits the performance data D. For example, the edit process Sb is executed each time the playback of the playback part by the playback device 20 (the above-mentioned playback control process Sa) is repeated a predetermined number of times. Note that the edit process Sb may be started in response to an instruction from the user U.

When the editing process Sb is started, the editing processing unit 34 calculates the dispersion Δ of the time differences L in a predetermined number of past playback control processes Sa (Sb1). The dispersion Δ is a statistical quantity that represents the degree of dispersion regarding multiple time differences L. For example, the variance, standard deviation, or distribution range of multiple time differences L is used as the dispersion Δ.

The editing processing unit 34 determines whether the dispersion Δ exceeds the threshold Δth (Sb2). If the dispersion Δ exceeds the threshold Δth, it is presumed that the user U is practicing playing the music while consciously varying the waiting time until playback of note N2 resumes. Therefore, it is inappropriate to edit the performance data D according to the multiple time differences L. On the other hand, if the dispersion Δ is below the threshold Δth, it is presumed that the multiple time differences L are values that are in line with the user U's intentions or preferences (i.e., preferred values unique to user U).

Taking the above tendencies into consideration, when the dispersion degree Δ falls below the threshold value Δth (Sb2: NO), the editing processing unit 34 edits the performance data D according to the multiple time differences L (Sb3-Sb4). On the other hand, when the dispersion degree Δ exceeds the threshold value Δth (Sb2: YES), the editing processing unit 34 ends the editing process Sb without editing the performance data D (Sb3, Sb4).

When editing the performance data D, the editing processor 34 calculates the average time difference La by averaging multiple time differences L (Sb3). Then, the editing processor 34 changes the start point of the note N2 specified by the performance data D by the average time difference La (Sb4). For example, if the average time difference La is a negative number, the editing processor 34 moves the start point of the note N2 specified by the performance data D forward by the time equivalent to the average time difference La. Also, if the average time difference La is a positive number, the editing processor 34 moves the start point of the note N2 specified by the performance data D backward by the time equivalent to the average time difference La. In other words, if the user U tends to have a sufficient waiting time just before the note N2, the start point of the note N2 specified by the performance data D is changed backward, and if the waiting time tends to be short, the start point of the note N2 specified by the performance data D is changed forward.

As can be understood from the above explanation, in the fourth embodiment, the performance data D is edited according to the time difference L in the performance of the performance parts by the user U. Therefore, it is possible to reflect the tendencies unique to each user U in the performance data D.

E: Fifth embodiment Fig. 12 is a block diagram illustrating the configuration of a playback system 100 according to a fifth embodiment. The playback system 100 includes a playback control system 10 and a performance device 50. The playback control system 10 of the fifth embodiment includes a display device 15 in addition to the same elements as the playback control system 10 of the first embodiment (a control device 11, a storage device 12, a sound collection device 13, and an operation device 14). The display device 15 displays an image instructed by the control device 11. The display device 15 is, for example, a liquid crystal display panel or an organic EL display panel.

The performance device 50 is an automatic musical instrument that functions not only as a playback device that executes automatic performance of the playback part of a musical piece, but also as an instrument that can be played manually by the user U1. Specifically, the performance device 50 is equipped with a drive mechanism 21 and a sound generation mechanism 22, similar to the playback device 20 in each of the above-mentioned forms. Note that the performance device 50 of the fifth embodiment may be used as the playback device 20 of the first to fourth embodiments.

Manual performance by user U1 is performed by the user U1's physical movements, such as pressing keys on a keyboard. The sound generation mechanism 22 operates in conjunction with the performance by user U1, causing musical tones to be generated from the performance device 50. In parallel with the performance by user U1, the performance device 50 sequentially outputs instruction data d representing instructions for the performance to the playback control system 10. The instruction data d specifies, for example, pitch and intensity, and specifies actions such as generating or muting sounds. On the other hand, user U2 plays the musical instrument 200. The musical instrument 200 is a natural musical instrument, such as a stringed instrument, that generates sounds when played by user U.

Figure 13 is a block diagram illustrating the functional configuration of the playback control system 10 in the fifth embodiment. The control device 11 in the fifth embodiment executes a program stored in the storage device 12, and functions as a preparation processing unit 35 in addition to the same elements as in the first embodiment (performance analysis unit 31, playback control unit 32, and instruction reception unit 33). The preparation processing unit 35 generates music data M (performance data D and reference data R) used in the playback control process Sa. Specifically, the preparation processing unit 35 generates music data M corresponding to the performance of the performance device 50 by user U1 and the performance of the musical instrument 200 by user U2. The preparation processing unit 35 includes a first recording unit 41, a second recording unit 42, and a reference data generating unit 43.

The reference data generating unit 43 generates reference data R to be used in the playback control process Sa. Specifically, the reference data generating unit 43 generates the reference data R by executing a reference data generating process Sc (FIG. 14) before the start of the playback control process Sa. The reference data R generated by the reference data generating unit 43 is stored in the storage device 12.

During the period before the reference data generation process Sc is executed (hereinafter referred to as the "preparation period"), users U1 and U2 play a piece of music together. Specifically, user U1 plays the playback part of the piece of music using a performance device 50, and user U2 plays the performance part of the piece of music using an instrument 200. The reference data generation process Sc is a process that generates reference data R using the results of users U1 and U2 playing the piece of music during the preparation period.

The first recording unit 41 acquires performance data D that specifies multiple sounds that the user U1 instructs the performance device 50 to play during the preparation period. Specifically, the first recording unit 41 generates performance data D in which instruction data d supplied sequentially from the performance device 50 in response to the performance by the user U1 and time data specifying the interval between adjacent instruction data d are arranged in chronological order. The first recording unit 41 saves the performance data D in the storage device 12. The performance data D stored in the storage device 12 is used for the playback control process Sa, as exemplified in each of the above-mentioned embodiments. Note that the performance data D acquired by the first recording unit 41 may be edited by the editing processing unit 34 exemplified in the fourth embodiment.

The second recording unit 42 acquires the sound signal Z (hereinafter referred to as "reference signal Zr1") generated by the sound collection device 13 during the preparation period. During the preparation period, in addition to the musical tones generated by the musical instrument 200 during the performance of the user U2, the musical tones generated by the performance device 50 during the performance of the user U1 reach the sound collection device 13. Therefore, the reference signal Zr1 is a signal representing a mixed sound of the musical tones of the musical instrument 200 and the musical tones of the performance device 50. As can be understood from the above explanation, the second recording unit 42 acquires the reference signal Zr1 representing the musical tones generated by the musical instrument 200 during the performance of the musical piece using the musical instrument 200 (first performance) and the musical tones generated by the performance device 50 during the performance of the musical instrument using the performance device 50. The second recording unit 42 stores the reference signal Zr1 in the storage device 12. As can be understood from the above explanation, during the preparation period, the performance data D and the reference signal Zr1 are stored in the storage device 12.

The reference data generating unit 43 generates the reference data R by the reference data generating process Sc using the reference signal Zr1 acquired by the second recording unit 42. In the above-mentioned embodiments, the MIDI data in which the instruction data and the time data are arranged in a time series is exemplified as the reference data R. The reference data R in the fifth embodiment is data that specifies the performance period, the sounding time, and the pitch transition in the music piece. The performance period is the period during which the user U2 plays the musical instrument 200 in the preparation period. The performance period can also be said as the period during which the musical instrument 200 sounds a musical tone. For example, the start and end times of the performance period are specified by the reference data R. The sounding time is the time when the sound of each musical tone in the music piece starts (i.e., onset) during the performance by the user U2 during the preparation period. For example, the time of the sounding time is specified by the reference data R. The pitch transition is the time series of the pitch of the musical tone sounded by the musical instrument 200 during the performance by the user U1 during the preparation period.

Figure 14 is a flowchart illustrating the specific steps of the reference data generation process Sc. After the performance data D and the reference signal Zr1 are acquired during the preparation period, the reference data generation process Sc is started, for example, in response to an instruction from the user U (U1, U2) via the operation device 14.

When the reference data generation process Sc is started, the reference data generation unit 43 generates a reference signal Zr2 by emphasizing the acoustic components of the musical tones produced by the musical instrument 200 in the reference signal Zr1 (Sc1). As described above, the reference signal Zr1 contains the acoustic components of the musical tones produced by the performance device 50 and the acoustic components of the musical tones produced by the musical instrument 200. The reference data generation unit 43 generates a reference signal Zr2 by suppressing the acoustic components of the musical tones of the performance device 50 from the reference signal Zr1.

For example, the reference data generating unit 43 generates a reference signal zr2 by subtracting the amplitude spectrogram of the musical sound of the performance device 50 from the amplitude spectrogram of the reference signal Zr1. The amplitude spectrogram of the musical sound of the performance device 50 is generated, for example, by known sound source processing that generates a musical sound signal representing the musical sound specified by the performance data D, and frequency analysis such as a discrete Fourier transform of the musical sound signal. The degree to which the amplitude spectrogram of the musical sound of the performance device 50 is subtracted is adjusted according to an instruction from the user U via the operation device 14.

The process of generating the reference signal Zr2 from the reference signal Zr1 is not limited to the above examples. For example, a known sound source separation technique may be used to emphasize the acoustic components of the musical sound of the musical instrument 200 in the reference signal Zr1. Also, for example, if the musical sound produced by the performance device 50 is unlikely to reach the sound collection device 13, step Sc1 may be omitted. In a form in which step Sc1 is omitted, the following processes are performed on the reference signal Zr1 instead of the reference signal Zr2. The reference signal Zr1 and the reference signal Zr2 are examples of the "first acoustic signal."

The control device 11 displays the preparation screen 60 of FIG. 15 on the display device 15 in parallel with the reference data generation process Sc. The preparation screen 60 displays a waveform 61 of the reference signal Zr2. The user U can adjust the range of the reference signal Zr2 that is displayed on the preparation screen 60 by operating the operation device 14.

As illustrated in FIG. 14, the reference data generating unit 43 analyzes the reference signal Zr2 to identify one or more performance periods in the reference signal Zr2 (Sc2). A first HMM (Hidden Markov Model) that estimates the performance period according to the intensity of the reference signal Zr2 is used to identify the performance period. In general, a period in the reference signal Zr2 where the signal intensity exceeds a threshold value τ is identified as the performance period. Note that the method of identifying the performance period is not limited to the above example.

The preparation screen 60 displays the performance period determined from the reference signal Zr2. Specifically, as illustrated in FIG. 15, the control device 11 displays the portion 61a of the waveform 61 of the reference signal Zr2 that is within the performance period and the portion 61b that is outside the performance period in different display modes. "Display mode" refers to the characteristics of an image that can be visually distinguished by an observer. For example, the three attributes of color, hue (tone), saturation, and brightness (gradation), as well as patterns or shapes, are also included in the concept of "display mode." Note that the method of displaying the performance period is not limited to the above examples.

The preparation screen 60 also displays an operation image 62 that the user U can operate using the operation device 14. The reference data generation unit 43 controls the threshold value τ applied to identifying the performance period in response to an instruction from the user U on the operation image 62. The smaller the threshold value τ set by the user U, the more likely it is that each point in the reference signal Zr2 will be determined to be a point within the performance period.

The reference data generating unit 43 identifies multiple sound generation time points in the reference signal Zr2 by analyzing the reference signal Zr2 (Sc3). As illustrated in FIG. 15, the preparation screen 60 displays an indication image 63 representing the sound generation time points identified from the reference signal Zr2. The indication image 63 is a vertical line arranged at each sound generation time point on the time axis. However, the specific form of the indication image 63 is not limited to the above example.

The reference data generating unit 43 uses a second HMM consisting of multiple states corresponding to different pitches to identify the playing time. The time when a transition occurs between different states (i.e., the time when the pitch changes) is identified as the sounding time. In response to an instruction from the user U regarding the type of instrument 200, the reference data generating unit 43 may limit the range of state transitions in the second HMM (i.e., the range over which the pitch changes) to the range of the instrument 200. In other words, transitions to states corresponding to pitches outside the range of the instrument 200 are prohibited.

For each state of the second HMM, a first transition matrix Λ1 and a second transition matrix Λ2 are set. The first transition matrix Λ1 and the second transition matrix Λ2 for each state are matrices that define the transition probability (self-transition probability) of the state itself and the transition probability of the state to another state. The numerical values of each transition probability are different between the first transition matrix Λ1 and the second transition matrix Λ2. Specifically, the transition probability to another state in the first transition matrix Λ1 is lower than the transition probability to another state in the second transition matrix Λ2. In other words, the self-transition probability in the first transition matrix Λ1 is higher than the self-transition probability in the second transition matrix Λ2. Therefore, when the first transition matrix Λ1 is applied to the second HMM, transitions between states are less likely to occur than when the second transition matrix Λ2 is applied to the second HMM. In other words, the frequency with which each time point on the time axis is estimated as a sounding time point decreases.

The reference data generating unit 43 of the fifth embodiment applies a transition matrix Λ, which is calculated for each state by a weighted sum of a first transition matrix Λ1 and a second transition matrix Λ2, to each state of the second HMM. The transition matrix Λ is expressed, for example, by the following formula (1). The coefficient α in formula (1) is a numerical value in the range of 0 to 1.
Λ＝α・Λ1+(1−α)・Λ2 (1)

15, the preparation screen 60 displays an operation image 64 that the user U can operate using the operation device 14. The reference data generating unit 43 controls the coefficient α applied to the specification of the performance time point in response to an instruction from the user U on the operation image 64. As can be understood from the formula (1), the larger the coefficient α, the greater the influence of the first transition matrix Λ1 on the transition matrix Λ, and the smaller the influence of the second transition matrix Λ2 on the transition matrix Λ. Therefore, it becomes difficult for a transition between states in the second HMM to occur. That is, the probability that each time point of the reference signal Zr2 is determined to be a sounding time point decreases. As can be understood from the above explanation, there is a tendency that the larger the coefficient α, the smaller the number of sounding times specified for the reference signal Zr2. In other words, the smaller the coefficient α, the greater the number of sounding times. The user U operates the operation image 64 while checking the preparation screen 60, thereby adjusting the coefficient α so that the number of sounding times displayed by the instruction image 63 is an appropriate number. Note that the method of specifying the sounding time point is not limited to the above example.

The reference data generating unit 43 identifies pitch transitions by analyzing the reference signal Zr2 (Sc4). For example, an estimation model that learns the relationship between the frequency characteristics of the reference signal Zr2 and the pitch transitions by machine learning is used to identify the pitch transitions. The estimation model is, for example, a deep neural network such as a convolutional neural network or a recurrent neural network. For example, control data including a spectrum generated by a constant Q conversion of the reference signal Zr2 is supplied to the estimation model. The estimation model outputs a time series of pitches (i.e., pitch transitions) that correlate with the control data. Note that the method of identifying pitch transitions is not limited to the above examples. In addition, the order of emphasizing the acoustic components of the musical tones (Sc1), identifying the performance period (Sc2), identifying the sounding time point (Sc3), and identifying the pitch transitions (Sc4) may be changed arbitrarily. For example, the acoustic components of the musical tones of the performance device 50 may be suppressed from the reference signal Zr1 by using a numerical value (e.g., the posterior probability of the pitch) calculated in identifying the pitch transitions (Sc4).

The reference data generating unit 43 determines whether or not an instruction has been received from the user U (Sc5). Specifically, it determines whether or not the user U has instructed to change the threshold value τ or the coefficient α. If an instruction from the user U has been received (Sc5: YES), the reference data generating unit 43 identifies the performance period, the sounding time point, and the pitch transition by processing (Sc1 to Sc4) that applies the changed threshold value τ or coefficient α.

It is assumed that the processing load for identifying the sounding time exceeds the processing load for identifying the performance period. In consideration of the above circumstances, a configuration is assumed in which the identification of the sounding time is performed less frequently than the identification of the performance period. For example, the reference data generating unit 43 repeats the identification of the performance period (Sc2) during the process in which the user U changes the threshold τ by operating the operation image 62. For example, the identification of the performance period (Sc2) is repeated during the period in which the user U drags the operation image 62 to change the threshold τ. On the other hand, the reference data generating unit 43 does not identify the sounding time during the process in which the user U changes the coefficient α by operating the operation image 64, but performs the identification of the sounding time (Sc3) when the change of the coefficient α is completed (i.e., the coefficient α after the change is confirmed). For example, the sounding time is identified at the stage when the user U finishes the operation of dragging the operation image 62 to change the coefficient α. The above configuration has the advantage that the performance period changes quickly in response to the instruction of the user U, while the processing load required for identifying the sounding time is reduced.

If no instruction is received from the user U (Sc5: NO), the reference data generation unit 43 determines whether an instruction to save the reference data R has been received from the user U (Sc6). If no instruction is received (Sc6: NO), the reference data generation unit 43 transitions to step Sc5. On the other hand, if an instruction to save is received from the user U (Sc6: YES), the reference data generation unit 43 saves the reference data R, which specifies the current performance period, sounding time, and pitch transition, in the storage device 12 (Sc7).

The reference data R generated by the reference data generation process Sc exemplified above is applied to the playback control process Sa. The specific procedure of the playback control process Sa in the fifth embodiment is the same as that of each of the above-mentioned embodiments. For example, the performance analysis unit 31 estimates the playing position X in parallel with the performance by comparing the audio signal Z representing the musical tone generated by the instrument 200 in the performance of the performance part by the user U2 (second performance) with the reference data R generated by the reference data generation process Sc. In addition, the playback control unit 32 causes the performance device 50 to play each note of the playback part specified by the performance data D acquired by the first recording unit 41. Specifically, the playback control unit 32 causes the playback of each note of the playback part to follow the performance by the user U2 according to the result of the estimation by the performance analysis unit 31.

In the above explanation, a single performance device 50 was used to obtain the performance data D during the preparation period and for automatic performance by the playback control process Sa. However, a playback device 20 that executes automatic performance by the playback control process Sa may be used separately from the performance device 50 used to obtain the performance data D during the preparation period. In other words, the automatic performance function is not essential for the performance device 50. Also, different musical instruments 20 may be used to obtain the reference signal Zr1 during the preparation period and for performance in parallel with the playback control process Sa.

Furthermore, as in the above-mentioned embodiments, the playback control unit 32 continues playing the note N1 that is being played at the time the first instruction Q1 is made by the user U2, even after the first instruction Q1 is made, until the end point of the note N1 specified by the performance data D. Furthermore, after the playback of the note N1 is stopped, the playback control unit 32 starts playing the note N2 immediately after the note N1, triggered by the second instruction Q2 made by the user U. Therefore, the same effect as the above-mentioned embodiments is achieved in the fifth embodiment as well.

Furthermore, according to the fifth embodiment, reference data R is generated based on the result of user U2 playing the musical instrument 200. Therefore, for any piece of music for which reference data R is not prepared, reference data R that reflects the performance of user U2 can be generated.

The control device 11 of the fifth embodiment causes the display device 15 to display the playback screen 70 of FIG. 16 in parallel with the playback control process Sa. The playback image 70 includes an operation image 71, a playback image 72, an instruction image 73, an instruction image 74, and an operation image 75.

The operation image 71 is an image that allows the user U to instruct the performance device 50 to start automatic performance (start of playback control process Sa). The playback image 72 is an image that represents the current playback position Y of the performance device 50. Specifically, the playback image 72 is composed of a time axis 721 that represents the entire section of the music piece, and an instruction image 722 that represents the current playback position Y. The instruction image 722 moves along the time axis 721 as the automatic performance progresses.

The indication image 73 is an image for notifying the user U of the sounding time specified by the reference data R. Specifically, the control device 11 changes the display mode of the indication image 73 when the playback position Y reaches each sounding time specified by the reference data R. For example, the indication image 73 is momentarily enlarged when the playback position Y coincides with the sounding time. By visually checking the indication image 73, the user U can confirm each sounding time specified by the reference data R. Therefore, for the performance of user U2 in the preparation period, the excess or deficiency of the sounding time estimated by the reference data generation process Sc (i.e., the appropriateness of the process of identifying the sounding time) can be visually confirmed.

The indication image 74 is an image for notifying the user U whether the volume of the musical tone produced by the instrument 200 is appropriate for estimating the performance position X with high accuracy. If the volume σ of the audio signal Z is too high or too low, there is a tendency for the accuracy of the estimation of the performance position X by the performance analysis unit 31 to decrease. Taking the above tendency into consideration, the control device 11 changes the display mode of the indication image 74 according to the volume σ of the audio signal Z.

Specifically, the control device 11 maintains the display state of the instruction image 74 in the first state when the volume σ of the audio signal Z is equal to or greater than the threshold σL and equal to or less than the threshold σH. The threshold σH is a numerical value that exceeds the threshold σL. Each of the thresholds σL and σH is set experimentally or statistically so that the performance position X is estimated with a target accuracy when the volume σ of the audio signal Z is a numerical value that is equal to or greater than the threshold σL and equal to or less than the threshold σH.

In addition, when the volume σ of the audio signal Z falls below the threshold σL, the control device 11 changes the display mode of the instruction image 74 to a second mode different from the first mode. That is, when the volume σ is too low for estimating the playing position X with high accuracy, the instruction image 74 is displayed in the second mode. On the other hand, when the volume σ of the audio signal Z exceeds the threshold σH, the control device 11 changes the display mode of the instruction image 74 to a third mode different from the first mode. That is, when the volume σ is too high for estimating the playing position X with high accuracy, the instruction image 74 is displayed in the third mode. As can be understood from the above explanation, the decrease in the estimation accuracy of the playing position X is notified to the user U by the change in the display mode of the instruction image 74. Note that there is no difference between the second mode and the third mode.

In the above explanation, we have focused on the volume σ of the audio signal Z, but the index that serves as the basis for changing the display mode of the instruction image 74 is not limited to the volume σ. For example, the control device 11 may control the display mode of the instruction image 74 according to the likelihood of the pitch estimated in step Sc4 of the reference data generation process Sc.

The operation image 75 in FIG. 16 is an image that allows the user U to adjust the performance speed of the automatic performance. By operating the operation image 75, the user U can specify the performance speed of the automatic performance by multiplying it by a predetermined reference value (i.e., a relative value). The playback control unit 32 uses the numerical value specified by the user U as the reference value for the performance speed, and controls the automatic performance by the performance device 50 so that the playback position Y follows the performance position X.

F: Modifications Specific modifications to the above-mentioned embodiments are given below. Two or more of the following embodiments may be combined as long as they are not mutually contradictory.

(1) In each of the above-mentioned embodiments, the instruction receiving unit 33 receives an operation to transition the operation device 14 from a released state to an operating state as the first instruction Q1, but the form of the first instruction Q1 is not limited to the above examples. For example, a specific motion by the user U is detected as the first instruction Q1. To detect the motion of the user U, various detection devices such as an imaging device or an acceleration sensor are used. For example, the instruction receiving unit 33 identifies various motions such as the motion of the user U raising one hand, the motion of lifting the musical instrument 200, or the motion of breathing (for example, the motion of inhaling) as the first instruction Q1. The breathing by the user U is, for example, a breath when playing a wind instrument as the musical instrument 200. The operation speed V1 in the second embodiment is comprehensively expressed as the speed of the motion of the user U identified as the first instruction Q1.

Specific data (hereinafter referred to as "first data") representing the first instruction Q1 may be included in the performance data D. The first data is, for example, data representing a fermata in a piece of music. The instruction receiving unit 33 determines that the first instruction Q1 has occurred when the playback position Y reaches the point of the first data. As can be understood from the above explanation, the first instruction Q1 is not limited to an instruction from the user U. Note that, when the dispersion degree Δ exceeds the threshold value Δth in the editing process Sb, the editing processing unit 34 may add the first data to the note N1.

(2) In each of the above-described embodiments, the instruction receiving unit 33 receives an operation to transition the operating device 14 from an operating state to a released state as the second instruction Q2, but the form of the second instruction Q2 is not limited to the above examples. For example, similar to the first instruction Q1 in the first embodiment, the instruction receiving unit 33 may receive an operation to transition the operating device 14 from a released state to an operating state as the second instruction Q2. In other words, two operations including stepping on and releasing the movable part 141 may be detected as the first instruction Q1 and the second instruction Q2.

A specific movement of the user U may be detected as the second instruction Q2. To detect the movement of the user U, various detection devices, such as an imaging device or an acceleration sensor, are used. For example, the instruction receiving unit 33 identifies various movements, such as the movement of the user U lowering one hand, the movement of lowering the instrument 200, or the movement of breathing (for example, the movement of exhaling), as the second instruction Q2. Breathing by the user U is, for example, a breath when playing a wind instrument as the instrument 200. The operation speed V2 in the second embodiment is comprehensively expressed as the speed of the movement of the user U identified as the second instruction Q2.

Specific data (hereinafter referred to as "second data") representing the second instruction Q2 may be included in the performance data D. The second data is, for example, data representing a fermata in a piece of music. The instruction receiving unit 33 determines that the second instruction Q2 has occurred when the playback position Y reaches the point in time of the second data. As can be understood from the above explanation, the second instruction Q2 is not limited to an instruction from the user U.

As shown in the above example, a configuration is envisaged in which one of a series of mutually paired actions by user U is accepted as a first instruction Q1, and the other is accepted as a second instruction Q2. For example, the action of user U raising one hand is accepted as a first instruction Q1, and the action of lowering the hand following that action is accepted as a second instruction Q2. Also, the action of user U lifting the instrument 200 is accepted as a first instruction Q1, and the action of lowering the instrument 200 following that action is accepted as a second instruction Q2. Similarly, the action of user U inhaling is accepted as a first instruction Q1, and the action of exhaling following that action is accepted as a second instruction Q2.

However, the first instruction Q1 and the second instruction Q2 do not have to be the same type of action by the user U. In other words, separate actions that the user U can perform independently of each other may be detected as the first instruction Q1 and the second instruction Q2. For example, the instruction receiving unit 33 may detect an operation on the operating device 14 as the first instruction Q1, and detect another action, such as an action of lifting the musical instrument 200 or an action of breathing, as the second instruction Q2.

(3) In each of the above-mentioned embodiments, an automatic musical instrument is exemplified as the playback device 20, but the configuration of the playback device 20 is not limited to the above examples. For example, a sound source system including a sound source device that generates an audio signal of a musical tone in response to an instruction from the playback control system 10, and a sound emission device that plays the musical tone represented by the audio signal may be adopted as the playback device 20. The sound source device is realized as a hardware sound source or a software sound source. The same applies to the performance device 50 in the fifth embodiment.

(4) In each of the above embodiments, the playback device 20 is controlled using performance data D that specifies the time series of multiple notes in a piece of music, but the format of data used to control the playback device 20 is not limited to the above examples. For example, waveform data representing the waveforms of multiple sounds that make up the playback part may be used to control the playback device 20. The waveform data is composed of a time series of multiple samples. The playback control unit 32 supplies the playback device 20 with a sample of the waveform data that corresponds to the playback position Y. The playback device 20 is a sound emission system that reproduces the sound represented by the time series of samples supplied from the playback control system 10.

The waveform data is added with sound generation point data representing the start point of each of the multiple sounds (hereinafter referred to as "sound generation point"). When the instruction receiving unit 33 receives the first instruction Q1, the playback control unit 32 causes the playback device 20 to play the sound represented by the waveform data until the point immediately before the sound generation point located immediately after the first instruction Q1, and waits for the second instruction Q2 after the playback. That is, the playback of the sound (first sound) being played at the time of the occurrence of the first instruction Q1 is continued until the end point of the first sound. On the other hand, when the instruction receiving unit 33 receives the second instruction Q2, the playback control unit 32 causes the playback device 20 to play the sound (second sound) represented by the waveform data from the sound generation point immediately after the section that has already been played (the sound generation point located immediately after the first instruction Q1). That is, after the playback of the first sound is stopped, the playback of the second sound immediately after the first sound is started in response to the second instruction Q2. In the above explanation, modified examples of the first to fourth embodiments equipped with the playback device 20 have been given, but the format of the data for controlling the automatic performance by the performance device 50 of the fifth embodiment is similarly not limited to the performance data D, and for example, the waveform data exemplified above may be used.

As can be understood from the above explanation, the performance data D exemplified in each of the above-mentioned forms and the waveform data exemplified in the modified examples are collectively expressed as sound data representing a time series of multiple sounds.

(5) The playback control system 10 may be realized by a server device that communicates with a terminal device such as a smartphone or a tablet terminal. The terminal device includes a sound collection device 13 that generates an audio signal Z according to a performance by the user U, and a playback device 20 (or a performance device 50 in the fifth embodiment) that plays a piece of music according to an instruction from the playback control system 10. The terminal device transmits the audio signal Z generated by the sound collection device 13 and a first instruction Q1 and a second instruction Q2 according to the user U's actions to the playback control system 10 via a communication network. The playback control system 10 causes the playback device 20 of the terminal device to play the playback part of the music according to the performance position X estimated from the audio signal Z and the first instruction Q1 and the second instruction Q2 received from the terminal device. The performance analysis unit 31 may be mounted on the terminal device. The terminal device transmits the performance position X estimated by the performance analysis unit 31 to the playback control system 10. In the above configuration, the performance analysis unit 31 is omitted from the playback control system 10.

(6) As described above, the functions of the playback control system 10 exemplified above are realized by the cooperation of one or more processors constituting the control device 11 and the program stored in the storage device 12. The program according to the present disclosure can be provided in a form stored in a computer-readable recording medium and installed in a computer. The recording medium is, for example, a non-transitory recording medium, and a good example is an optical recording medium (optical disk) such as a CD-ROM, but also includes any known type of recording medium such as a semiconductor recording medium or a magnetic recording medium. Note that a non-transitory recording medium includes any recording medium except a transient, propagating signal, and does not exclude volatile recording media. In addition, in a configuration in which a distribution device distributes a program via a communication network, the storage device that stores the program in the distribution device corresponds to the non-transitory recording medium described above.

G: Supplementary Note From the above-described exemplary embodiments, the following configurations, for example, can be understood.

A playback control method according to one aspect (aspect 1) of the present disclosure is a method for playing back a plurality of sounds using sound data representing a time series of the plurality of sounds, which plays back a first sound of the plurality of sounds corresponding to a first instruction, and after playback of the first sound is stopped, starts playback of a second sound of the plurality of sounds immediately after the first sound in response to a second instruction from the user. According to the above aspect, after playback of the first sound corresponding to the first instruction is stopped, playback of the second sound immediately after the first sound is started in response to a second instruction from the user. Therefore, the interval between the first sound and the second sound (e.g., the duration of the rest period) can be appropriately controlled according to the time points of the first instruction and the second instruction.

"Sound data" is data of any format that represents a time series of multiple sounds. For example, the concept of "sound data" includes performance data that specifies the sounding period (specifically, sounding and muting) for each of multiple notes, or waveform data that represents the waveforms of multiple sounds on a time axis.

The "first instruction" is, for example, an instruction according to an action by a user, or an instruction added to the sound data. An instruction according to an action by a user is, for example, an instruction given by the user operating an operating device. The first instruction may also be generated when the user performs a specific action (for example, lifting an instrument or taking a breath while playing a wind instrument). An instruction added to the sound data is, for example, a performance instruction of various kinds, such as a fermata, which means the extension of a note or rest.

The "second instruction" is, for example, an instruction corresponding to an action by the user. An instruction corresponding to an action by the user is, for example, an instruction given by the user operating an operating device. In addition, the second instruction may be generated when the user performs a specific action (for example, lifting an instrument or taking a breath while playing a wind instrument).

"Starting playback of the second sound in response to the second instruction" means that playback of the second sound is started on the condition that the second instruction is received, and the relationship between the time of the second instruction and the time at which playback of the second sound is started is not important. For example, in addition to the mode in which playback of the second sound is started at or immediately after the second instruction, the mode in which playback of the second sound is started a predetermined time after the second instruction is received is also included in the mode in which "playback of the second sound is started in response to the second instruction."

"Reproduction" of multiple sounds means emitting each sound as a sound wave. Therefore, for example, the concept of "reproduction" includes automatic performance in which an automatic musical instrument such as a player piano reproduces multiple sounds, or the emission of multiple sounds by a sound source device and a sound emission device.

In a specific example (aspect 2) of aspect 1, the first sound is one of the multiple sounds being played at the time when the first instruction is issued, and in playing the first sound, the playback of the first sound continues after the first instruction is issued until the end point of the first sound represented by the sound data. In the above aspect, even after the first instruction is issued, the playback of the first sound continues until the end point of the first sound represented by the sound data. Therefore, compared to a configuration in which the playback of the first sound is stopped at the time of the first instruction, it is possible to appropriately continue the first sound according to the sound data.

In a specific example of aspect 1 (aspect 3), the multiple sounds constitute a piece of music, and the point in time at which the user is playing the piece of music is estimated in parallel with the playback of the multiple sounds, and in the playback of the multiple sounds, the playback of the multiple sounds is made to follow the performance of the piece of music according to the result of the estimation. In the above aspect, since the playback of the multiple sounds follows the performance of the piece of music by the user, it is possible to appropriately reflect the user's intentions (e.g., performance expression) or preferences in the playback of the multiple sounds.

In a specific example (Aspect 4) of any of Aspects 1 to 3, the sound data is performance data that specifies a sounding period for each of the multiple sounds, and in playing the first sound, after the first instruction is generated, the playback of the first sound continues until the end of the sounding period specified for the first sound by the performance data. In the above aspects, even after the first instruction is generated, the playback of the first sound continues until the end of the sounding period specified for the first sound by the performance data. Therefore, compared to a configuration in which the playback of the first sound is stopped at the time of the first instruction, it is possible to appropriately continue the first sound over the sounding period specified by the performance data.

In a specific example (Aspect 5) of any one of Aspects 1 to 4, the first instruction and the second instruction are generated in response to an operation of an operating device by the user. According to the above aspect, the user can change the interval between the first sound and the second sound to an appropriate time length according to the user's intention or preference by operating the operating device.

In a specific example (aspect 6) of aspect 5, the first instruction is generated by the user's operation of transitioning the operating device from the first state to the second state, and the second instruction is generated by the user's operation of transitioning the operating device from the second state to the first state. According to the above aspect, the first instruction is generated by the user transitioning the operating device from the first state to the second state, and after the second state is maintained, the second instruction is generated by the user transitioning the operating device from the second state to the first state at a time point desired by the user. Therefore, compared to a configuration in which an operation of transitioning the operating device from the first state to the second state is required for each of the first instruction and the second instruction, the operation of the operating device by the user is simplified.

In a specific example (aspect 7) of aspect 5 or aspect 6, the duration of the first sound is controlled according to the speed of operation of the operating device. According to the above aspect, the user can adjust the duration of the first sound according to the speed of operation of the operating device. In addition, since the operating device for providing the first and second instructions is also used to adjust the duration of the first sound, there is also the advantage that the operation by the user is simplified compared to a configuration in which the user operates separate devices for each of providing the first and second instructions and adjusting the duration of the first sound.

In a specific example (aspect 8) of any of aspects 5 to 7, the time at which playback of the second sound starts is controlled according to the speed of operation of the operation device. According to the above aspects, the user can adjust the start point of playback of the second sound according to the speed of operation of the operation device. In addition, since the operation device for giving the first instruction and the second instruction is also used to adjust the start point of the second sound, there is also the advantage that the operation by the user is simplified compared to a configuration in which the user operates separate devices for each of giving the first instruction and the second instruction and adjusting the start point of the second sound.

A playback control method according to one aspect (aspect 9) of the present disclosure is a method for playing back a plurality of sounds using sound data representing a time series of the plurality of sounds, and continues a first sound among the plurality of sounds that is being played back at the time when a first instruction is issued until the end point of the first sound represented by the sound data. In the above aspect, even after the first instruction is issued, playback of the first sound continues until the end point of the first sound represented by the sound data. Therefore, compared to a configuration in which playback of the first sound is stopped at the time of the first instruction, it is possible to appropriately continue the first sound according to the sound data.

A playback control system according to one aspect (aspect 10) of the present disclosure includes a playback control unit that plays back a plurality of sounds using sound data representing a time series of the plurality of sounds, and the playback control unit plays back a first sound among the plurality of sounds that corresponds to a first instruction, and after playback of the first sound is stopped, starts playback of a second sound among the plurality of sounds immediately after the first sound in response to a second instruction from the user. According to the above aspect, after playback of the first sound corresponding to the first instruction is stopped, playback of the second sound immediately after the first sound is started in response to a second instruction from the user. Therefore, the interval between the first sound and the second sound (e.g., the duration of the rest period) can be appropriately controlled according to the time points of the first instruction and the second instruction.

A program according to one aspect (aspect 11) of the present disclosure is a program that causes a computer to function as a playback control unit that uses sound data representing a time series of a plurality of sounds to play back the plurality of sounds, and the playback control unit plays a first sound of the plurality of sounds that corresponds to a first instruction, and after playback of the first sound is stopped, starts playback of a second sound of the plurality of sounds immediately after the first sound in response to a second instruction from the user. According to the above aspect, after playback of the first sound corresponding to the first instruction is stopped, playback of the second sound immediately after the first sound is started in response to a second instruction from the user. Therefore, the interval between the first sound and the second sound (e.g., the duration of the rest period) can be appropriately controlled according to the time points of the first instruction and the second instruction.

In order to estimate the time (performance position) in a piece of music that a user is playing, reference data that can be compared with the user's performance is required. However, there may be situations in which reference data is not available for the desired piece of music.

In consideration of the above circumstances, a playback control system according to one aspect (aspect 12) of the present disclosure includes a first recording unit that acquires performance data specifying a plurality of sounds instructed to a playback device, a second recording unit that acquires a first audio signal representing a musical tone generated by a musical instrument in a first performance of the music using the musical instrument, a reference data generating unit that generates reference data representing at least the time when each musical tone is generated in the first performance from the first audio signal, a performance analysis unit that estimates the time when the musical tone is being played in the music in parallel with the second performance by comparing the second audio signal representing the musical tone generated by the musical instrument in a second performance of the music using the musical instrument with the reference data, and a playback control unit that reproduces the plurality of sounds specified by the performance data and causes the reproduction of the plurality of sounds to follow the second performance according to the result of the estimation by the performance analysis unit. According to the above configuration, reference data representing the time when each musical tone is generated in the performance is generated from the first audio signal representing the musical tone generated by the musical instrument in a performance of the music using the musical instrument. Therefore, it is not necessary to prepare reference data in advance for a desired music piece.

In a specific example (aspect 13) of aspect 12, the playback control unit continues a first sound of the multiple sounds that is being played at the time when a first instruction is issued until the end point of the first sound represented by the sound data, and after the playback of the first sound is stopped, starts playback of a second sound of the multiple sounds immediately after the first sound in response to a second instruction from the user. According to the above aspect, after the playback of the first sound corresponding to the first instruction is stopped, playback of the second sound immediately after the first sound is started in response to a second instruction from the user. Therefore, the interval between the first sound and the second sound (e.g., the duration of the rest period) can be appropriately controlled according to the time points of the first instruction and the second instruction.

100...playback system, 200...musical instrument, 10...playback control system, 11...control device, 12...storage device, 13...sound collection device, 14...operation device, 141...moving part, 20...playback device, 21...driving mechanism, 22...sound generation mechanism, 50...performance device, 31...performance analysis unit, 32...playback control unit, 33...instruction reception unit, 34...editing processing unit, 41...first recording unit, 42...second recording unit, 43...reference data generation unit.

Claims (6)

A method for reproducing a plurality of sounds by using sound data representing a time series of the plurality of sounds constituting a piece of music, comprising:
estimating a point in time within the piece of music that a user is playing in parallel with the playback of the plurality of sounds;
According to the result of the estimation, the reproduction of the plurality of sounds is synchronized with the performance of the musical piece;
In the reproduction of the plurality of sounds,
playing a first sound corresponding to a first instruction among the plurality of sounds;
and after stopping the playback of the first sound, starting playback of a second sound of the plurality of sounds immediately after the first sound in response to a second instruction from a user. A method for reproducing a plurality of sounds by using sound data representing a time series of the sounds, comprising:
continuing a first sound among the plurality of sounds that is being reproduced at the time when a first instruction is issued until an end point of the first sound represented by the sound data;
after stopping the reproduction of the first sound, starting reproduction of a second sound immediately after the first sound among the plurality of sounds in response to a second instruction from a user ;
the first instruction and the second instruction are generated in response to an operation of an operation device by the user,
The duration of the first sound is controlled in accordance with the speed of the operation of the operating device.
A computer-implemented playback control method. A method for reproducing a plurality of sounds by using sound data representing a time series of the sounds, comprising:
continuing a first sound among the plurality of sounds that is being reproduced at the time when a first instruction is issued until an end point of the first sound represented by the sound data;
after stopping the reproduction of the first sound, starting reproduction of a second sound immediately after the first sound among the plurality of sounds in response to a second instruction from a user ;
the first instruction and the second instruction are generated in response to an operation of an operation device by the user,
A time point at which the reproduction of the second sound is started is controlled according to a speed of the operation of the operation device.
A computer-implemented playback control method. the first instruction is generated by an operation by the user to transition the operation device from a first state to a second state,
The playback control method according to claim 2 or 3 , wherein the second instruction is generated by an operation by the user to transition the operation device from the second state to the first state. the sound data is performance data that specifies a sound generation period for each of the plurality of sounds;
5. The playback control method according to claim 1, wherein, in the playback of the first sound, after the first instruction is generated, the playback of the first sound is continued until an end point of the sound generation period specified for the first sound by the performance data. a first recording unit for acquiring performance data specifying a plurality of sounds instructed to a performance device;
a second recording unit that acquires a first audio signal representing a musical tone generated by a musical instrument during a first performance of a piece of music using the musical instrument;
a reference data generating unit that generates reference data representing at least a sounding time point of each musical tone generated by the first performance from the first audio signal;
a performance analysis unit that estimates a time point in the musical piece being played in parallel with the second performance by comparing a second audio signal representing a musical tone produced by the musical instrument in a second performance of the musical piece using the musical instrument with the reference data;
a playback control unit that plays back a plurality of sounds designated by the performance data, and causes the playback of the plurality of sounds to follow the second performance in accordance with a result of estimation by the performance analysis unit;
The playback control unit
a first sound among the plurality of sounds that is being reproduced at the time when a first instruction is issued is continued until an end point of the first sound represented by the performance data;
and after stopping the playback of the first sound, starting playback of a second sound of the plurality of sounds immediately after the first sound in response to a second instruction from a user.

Images