JP7811964B2 - Control device, system, control method, and program - Google Patents

Description

The present invention relates to a control device, a system, a control method, and a program.

Patent document 1 describes "an ensemble device that is placed at a first location to realize an ensemble performance between a first performer at a first location and a second performer at a second location different from the first location via a network."
[Prior art documents]
[Patent Documents]
[Patent Document 1] JP 2016-200712 A

One embodiment of the present invention provides a control device. The control device may include a memory unit that stores information about a performance data processing device that processes performance data of a musical instrument performance received via a network to generate processed performance data. The control device may include a correspondence unit that uses information about the performance data processing device stored in the memory unit to associate the performance data communication device with the performance data processing device, the performance data processing device having a transmission unit that transmits the performance data, to have the performance data processing device process the performance data transmitted by the transmission unit.

In the control device, the storage unit may store information on a plurality of performance data processing devices that perform different processing of performance data. The association unit may associate the performance data processing device specified by the performance data communication device with the performance data communication device among the plurality of performance data processing devices.

Any of the control devices may include a destination designation receiving unit that receives designation of a destination to which the processed performance data is to be sent. The association unit may control the transmission of the processed performance data via the network to the destination designated by the destination designation receiving unit.

Any of the control devices may include a communication status acquisition unit that acquires the communication status of the communication path from the performance data processing device to the destination. The control device may include a processing adjustment unit that adjusts the processing of the performance data by the performance data processing device based on the communication status. The processing adjustment unit may control the content of the processing so that the data volume of the processed performance data is less than the data volume corresponding to the communication delay indicated by the communication status.

In any of the above control devices, the association unit may associate a first performance data communication device with a first performance data processing device, and may associate a second performance data communication device with a second performance data processing device. The control device may include a receiving unit that receives from the first performance data processing device first processed performance data, which the first performance data processing device has processed the performance data received from the first performance data communication device, and receives from the second performance data processing device second processed performance data, which the second performance data processing device has processed the performance data received from the second performance data communication device. The control device may include an ensemble data generation unit that generates ensemble data by combining the first processed performance data and the second processed performance data received by the receiving unit.

In any of the control devices, the association unit may associate the performance data communication devices with the performance data processing device. The association unit may cause the performance data processing device to process the performance data received from each of the performance data communication devices and synthesize the processed performance data to generate ensemble data.

One embodiment of the present invention provides a system comprising any one of the control devices described above and a performance data generation device. The performance data generation device may have a pseudo-generation unit that generates pseudo-performance data of a specified piece of music played by a specified user. The system may also have an ensemble data generation unit that generates ensemble data by combining processed performance data obtained by processing performance data received from the performance data communication device by the performance data processing device associated with the performance data communication device by the association unit, with pseudo-performance data pseudo-generated by the pseudo-generation unit.

In the system, the performance data generation device may have a learning model storage unit that stores performance data generated by a user playing a piece of music on an instrument and a learning model that takes score data as input and outputs performance data, the learning model being generated by machine learning using training data including score data for the piece of music. In the system, the performance data generation device may have a score data receiving unit that receives score data. In the system, the pseudo-generation unit may input the score data received by the score data receiving unit into the learning model corresponding to the specified user, and obtain the performance data output from the learning model as pseudo-performance data.

In any of the above systems, the learning model storage unit may store ensemble data generated by a first user and a second user playing together, and a learning model generated by machine learning using learning data including performance data of the first user in the ensemble, the learning model taking performance data of the first user as input and performance data of the second user as output. The pseudo-performance generation unit may input the performance data of the first user to the learning model and obtain the performance data output from the learning model as pseudo-performance data.

One embodiment of the present invention provides a control method executed by a computer. The control method may include a storage step of storing, in a storage unit, information about a performance data processing device that processes performance data of a musical instrument performance received via a network to generate processed performance data. The control method may also include an association step of associating a performance data communication device with a performance data processing device using information about the performance data processing device stored in the storage unit, the performance data communication device having a transmission unit that transmits the performance data, so that the performance data processing device can process the performance data transmitted by the transmission unit.

According to one embodiment of the present invention, a program for causing a computer to execute the control method is provided.

Note that the above summary of the invention does not list all of the necessary features of the present invention. Subcombinations of these features may also constitute inventions.

An example of a system 90 is shown schematically. 10 shows an example of a process flow in the system 90. 10 shows an example of a process flow in the system 90. 10 shows an example of a process flow in the system 90. 10 shows an example of a process flow in the system 90. 10 shows an example of a process flow in the system 90. 10 shows an example of a process flow in the system 90. 10 shows an example of a process flow in the system 90. An example of a system 90 is shown schematically. 2 shows an example of a functional configuration of the control device 300. 1 shows an example of the functional configuration of a performance data generating device 400. An example of the hardware configuration of a computer 1200 that functions as the performance data communication device 100, the performance data processing device 200, the control device 300, the performance data generation device 400, or the ensemble data generation unit 92 is shown in the figure.

The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention as claimed. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

In ensemble performances using existing remote ensemble services, for example, if a user wanted to use an effector or other device to process the sound of an electronic guitar or other instrument in order to participate in the ensemble, the sound processing had to be performed in the user's local environment, and then the processed performance data had to be transmitted over the Internet. As a result, in order to perform the desired sound processing, the user had to purchase and own relatively expensive sound processing equipment such as an effector or amp simulator.

The control device according to this embodiment has a configuration that contributes to solving these problems. For example, the control device controls the connection between a sound processing device and a user terminal on a network, causing the sound processing device to process performance data received from the user terminal via the network. This allows performers to achieve their desired sound processing without having to purchase and own multiple sound processing devices. Furthermore, users other than performers, such as recording engineers, PA (Public Address) engineers, and listeners, can also link the data of their favorite performers to their own usage environment and process performances, broadening the scope of usage patterns.

Figure 1 shows an example of a system 90. The system 90 includes a performance data communication device 100, a performance data processing device 200, and a control device 300. The performance data communication device 100, the performance data processing device 200, and the control device 300 are interconnected via a network 80 so that they can communicate with each other.

The performance data communication device 100 has the function of transmitting performance data 10, which is generated when a user 50 plays a musical instrument 500, via a network 80. The type of musical instrument 500 is not particularly limited. The musical instrument 500 may be an electronic instrument such as an electronic guitar, electronic bass, electronic drum, electronic violin, or electronic piano. The musical instrument 500 may also be an acoustic instrument such as a guitar, bass, drum, violin, or piano. In this case, a pickup device may be attached to the acoustic instrument, and the performance data communication device 100 may generate the performance data 10 by acquiring an electrical signal from the pickup device. The performance data communication device 100 may also generate the performance data 10 by collecting the sound of the musical instrument 500 with a microphone and acquiring the electrical signal.

The performance data communication device 100 may have a performance input unit 130 and a conversion unit 150. The performance data communication device 100 may receive an analog signal of a performance using the musical instrument 500 via the performance input unit 130, and the conversion unit 150 may convert the analog signal into a digital signal, thereby generating performance data 10. The performance data communication device 100 has a transmission unit 110. The transmission unit 110 transmits the performance data 10. The performance input unit 130 may receive an analog signal from the musical instrument 500. The performance input unit 130 may receive an analog signal from a pickup device attached to the musical instrument 500. The performance input unit 130 may receive an analog signal from a microphone.

The performance input unit 130 may accept singing by the user 50. In this case, the performance data communication device 100 may generate vocal data by accepting an analog signal of the user 50's singing via the performance input unit 130 and converting the analog signal into a digital signal via the conversion unit 150. The transmission unit 110 may transmit the vocal data.

The performance input unit 130 may accept audio input via field recording. The subject of field recording is not particularly limited, but for example, the performance input unit 130 may accept as field sounds the sounds of natural environments such as the sea or mountains, or the sounds of a busy urban area. The performance input unit 130 may also accept as field sounds the sounds played outdoors using an instrument 500. The performance input unit 130 may also accept as field sounds the sounds sung outdoors by a user 50. In these cases, the performance data communication device 100 may generate field sound data by receiving an analog signal of the field sound via the performance input unit 130 and converting the analog signal into a digital signal via the conversion unit 150. The transmission unit 110 may transmit the field sound data.

The performance data processing device 200 processes the performance data 10 received via the network 80 to generate processed performance data 20. The processing of the performance data 10 by the performance data processing device 200 may be processing that changes the acoustic characteristics of the performance. For example, the processing of the performance data 10 by the performance data processing device 200 may include distortion processing, oscillation processing, reverberation processing, dynamics adjustment processing, filter processing, amplifier simulation processing, harmony processing, volume processing, synthesizer processing, panning processing, tempo processing, etc. For example, the processing of the performance data 10 by the performance data processing device 200 may include distortion, tremolo, vibrato, delay, reverb, compressor, filter, wah, preamp, simulator, octaver, pitch shifter, harmonizer, chorus, equalizer, booster, guitar synthesizer, etc.

The processing of the performance data 10 by the performance data processing device 200 may be processing that takes advantage of the communication characteristics of the network 80. For example, the processing of the performance data 10 by the performance data processing device 200 may be processing that realizes a feedback performance technique that takes advantage of the communication characteristics of the network 80. For example, the processing of the performance data 10 by the performance data processing device 200 may be processing that realizes a feedback performance technique that utilizes network echo. For example, the control device 300 controls the performance data processing device 200 to return the performance data 10 to the performance data communication device 100 via multiple different communication paths. As a result, the timing at which the performance data 10 reaches the performance data communication device 100 differs for each communication path, so when the performance data 10 is output from the performance data communication device 100, it sounds as if it has been subjected to feedback processing.

In this case, the control device 300 may control the performance data processing device 200 to return processed performance data 20, instead of performance data 10, to the performance data communication device 100 via multiple different communication paths. For example, by having the control device 300 control the performance data processing device 200 to return processed performance data 20, which is performance data 10 that has been subjected to distortion processing, to the performance data communication device 100 via multiple different communication paths, the processed performance data will sound as if it has been subjected to both distortion processing and feedback processing when output from the performance data communication device 100.

For example, the processing of the performance data 10 by the performance data processing device 200 is processing that realizes a feedback performance technique by having the performance data processing device 200 transmit multiple pieces of performance data 10 to the network 80 at different transmission timings. For example, the control device 300 may control the performance data processing device 200 so that after the performance data processing device 200 transmits performance data 10 to the network 80, it transmits performance data 10 identical to the performance data 10 to the network 80 each time a predetermined delay interval has elapsed. As a result, each of the multiple pieces of performance data 10 arrives at the performance data communication device 100 with a timing difference, and when output from the performance data communication device 100, it sounds as if feedback processing has been applied.

In this case, the processing of the performance data 10 by the performance data processing device 200 may be processing that realizes a feedback performance technique by the performance data processing device 200 transmitting multiple pieces of processed performance data 20 to the network 80 at different transmission timings instead of multiple pieces of performance data 10. For example, after the performance data processing device 200 transmits processed performance data 20 to the network 80, the control device 300 may control the performance data processing device 200 to transmit processed performance data 20 that is identical to the processed performance data 20 to the network 80 each time a predetermined delay interval has elapsed.

For example, the processing of performance data 10 by the performance data processing device 200 is a processing that realizes a feedback performance technique by performing a loop process in which the processed performance data 20 is input again into the performance data processing device 200. For example, the control device 300 controls the performance data processing device 200 to execute a loop process in which the processed performance data 20 generated by processing the performance data 10 is transmitted to the network 80, and the performance data processing device 200 inputs the processed performance data 20 itself and transmits it to the network 80 together with the processed performance data 20 generated by processing the next performance data 10. As a result, the processed performance data 20 arrives at the performance data communication device 100 together with the previous processed performance data 20, so that when it is output from the performance data communication device 100, it sounds as if it has been subjected to feedback processing.

The control device 300 may control the performance data processing device 200 to further process the processed performance data 20 input to the performance data processing device 200. For example, the control device 300 controls the performance data processing device 200 to transmit the processed performance data 20 generated by processing the performance data 10 over the network 80, and to process the processed performance data 20 together with the next performance data 10 and transmit the processed performance data 20 obtained by processing the processed performance data 20 together with the next performance data 10 over the network 80. The control device 300 may control the performance data processing device 200 to start loop processing in response to an instruction from the performance data communication device 100, and may control the performance data processing device 200 to end loop processing in response to an instruction from the performance data communication device 100.

The performance data processing device 200 may include a receiving unit 220 and a performance data processing unit 230. The receiving unit 220 may receive performance data 10 via the network 80. The performance data processing unit 230 may process the performance data 10 received by the receiving unit 220 to generate processed performance data 20. The performance data processing device 200 may include a transmitting unit 210. The transmitting unit 210 may transmit the processed performance data 20 generated by the performance data processing unit 230.

The control device 300 may control the various devices that make up the system 90. The control device 300 may control the communication connections between the various devices that make up the system 90.

The control device 300 stores information about the performance data processing device 200 (sometimes referred to as processing device information). The processing device information may include processing device identification information that can identify the performance data processing device 200. The processing device information may include network information that can identify the performance data processing device 200 on the network 80. The network information may include, for example, an IP address assigned to the performance data processing device 200. The processing device information may include processing content information that indicates the processing content performed by the performance data processing device 200. The control device 300 may store processing device information for multiple performance data processing devices 200. For example, the control device 300 may store a processing device list that includes processing device information for multiple performance data processing devices 200 that can be used by the performance data communication device 100.

The control device 300 may store information about the performance data communication device 100 (sometimes referred to as communication device information). The communication device information may include communication device identification information that can identify the performance data communication device 100. The communication device information may include network information that can identify the performance data processing device 200 on the network 80. The network information includes, for example, an IP address assigned to the performance data communication device 100. The communication device information may also include user identification information that can identify the user of the performance data communication device 100.

The control device 300 associates the performance data communication device 100 with the performance data processing device 200 using the processing device information stored in the control device 300, in order to have the performance data 10 processed by the performance data processing device 200. The control device 300 may, for example, associate the performance data processing device 200 specified by the performance data communication device 100 with the performance data communication device 100. The control device 300 may, for example, send a processing device list to the performance data communication device 100, and associate the performance data processing device 200 selected by the performance data communication device 100 with the performance data communication device 100.

The control device 300's association of the performance data communication device 100 with the performance data processing device 200 may involve, for example, establishing a communication connection between the performance data communication device 100 and the performance data processing device 200. The control device 300 may establish a communication connection between the performance data communication device 100 and the performance data processing device 200 by sending an instruction to establish a communication connection to at least one of the performance data communication device 100 and the performance data processing device 200. The control device 300's association of the performance data communication device 100 with the performance data processing device 200 may involve the control device 300 managing the correspondence between the performance data communication device 100 and the performance data processing device 200. In this case, the control device 300 may relay communication between the performance data communication device 100 and the performance data processing device 200.

The transmitting unit 110 of the performance data communication device 100, which has established a communication connection with the performance data processing device 200 via the control device 300, may transmit the performance data 10 to the performance data processing device 200 via the network 80. The receiving unit 220 of the performance data processing device 200 receives the processed performance data 20. The performance data processing unit 230 may process the performance data 10 received by the receiving unit 220 to generate processed performance data 20. The transmitting unit 210 may transmit the processed performance data 20 to the performance data communication device 100 via the network 80.

When the control device 300 relays communication between the performance data communication device 100 and the performance data processing device 200, the transmitting unit 110 of the performance data communication device 100 may transmit the performance data 10 to the control device 300 via the network 80. The control device 300 may transmit the received performance data 10 to the performance data processing device 200 via the network 80. The receiving unit 220 of the performance data processing device 200 receives the performance data 10. The performance data processing unit 230 may process the performance data 10 received by the receiving unit 220 to generate processed performance data 20. The transmitting unit 210 may transmit the processed performance data 20 to the control device 300 via the network 80. The control device 300 may transmit the processed performance data 20 to the performance data communication device 100 via the network 80.

The performance data communication device 100 may have a receiving unit 120, a converting unit 160, and a performance output unit 140. The receiving unit 120 may receive the processed performance data 20 transmitted by the transmitting unit 210. The converting unit 160 converts the processed performance data 20 into an analog signal. The converting units 150 and 160 may be integrated. The performance output unit 140 outputs the analog signal converted by the converting unit 160 as sound via a performance output device 600. The performance output device 600 is, for example, a speaker, headphones, or earphones. This allows the user 50 to listen to the processed performance data 20 that has been acoustically processed from their own performance, without having to prepare an acoustic processing device such as an effector in the environment where the musical instrument 500 is played.

Network 80 may include a mobile communication network. The mobile communication network may be compliant with 5G (5th Generation). The mobile communication network may be compliant with LTE (Long Term Evolution). The mobile communication network may be compliant with 6G (6th Generation) or later communication systems. Network 80 may include the Internet. Network 80 may include the cloud. The Internet may include a LAN (Local Area Network).

The performance data communication device 100 may be connected to the network 80 wirelessly. The performance data communication device 100 may be connected to the network 80 via a wireless base station. The performance data communication device 100 may be connected to the network 80 via a Wi-Fi (registered trademark) access point. The performance data communication device 100 may be connected to the network 80 via a wired connection.

The performance data processing device 200 may be wirelessly connected to the network 80. The performance data processing device 200 may be connected to the network 80 via a wireless base station. The performance data processing device 200 may be connected to the network 80 via a Wi-Fi access point. The performance data processing device 200 may be wired to the network 80. The performance data processing device 200 may be located, for example, in an MEC (Multi-access Edge Computing). If the mobile communications network of the network 80 has a hierarchical management system consisting of a management platform, a lower-level distribution platform, and a lower-level wireless base station, the performance data processing device 200 may be located on the distribution platform or on the management platform. The management platform is sometimes called a Core Brain, and the distribution platform is sometimes called a Regional Brain. Note that the distribution platform may have multiple hierarchical levels. For example, if a two-tiered distributed infrastructure is placed under a management infrastructure, the management infrastructure may be called the Core Brain, the distributed infrastructure at the tier below it may be called the Regional Brain, and the distributed infrastructure at the tier below that may be called the Sub-Regional Brain.

The control device 300 may be wirelessly connected to the network 80. The control device 300 may be connected to the network 80 via a wireless base station. The control device 300 may be connected to the network 80 via a Wi-Fi access point. The control device 300 may be wired to the network 80. The control device 300 may be located in, for example, an MEC. If the mobile communications network of the network 80 has a hierarchical management system consisting of a management platform, a distributed platform at a lower level, and a wireless base station at a lower level, the control device 300 may be located in either the distributed platform or the management platform.

As described above, the control device 300 may cause the performance data processing device 200 to process the performance data 10 transmitted by the performance data communication device 100, and may control the performance data processing device 200 to transmit the processed performance data 20 to the performance data communication device 100. The control device 300 may cause the performance data processing device 200 to process the performance data 10 transmitted by the performance data communication device 100, and may control the performance data processing device 200 to transmit the processed performance data 20 to another device. For example, the control device 300 may accept a designation of a destination to which the processed performance data 20 is to be transmitted, and may control the processing performance data 20 to be transmitted to the designated destination via the network 80.

For example, the control device 300 establishes a communication connection between the performance data communication device 100 and the performance data processing device 200, and also establishes a communication connection between the performance data processing device 200 and a specified destination. The performance data communication device 100 transmits performance data 10 to the performance data processing device 200 with which the communication connection has been established, and the performance data processing device 200 processes the received performance data 10 to generate processed performance data 20, which it then transmits to the destination.

For example, the control device 300 receives performance data 10 from the performance data communication device 100, transmits it to the performance data processing device 200, and receives processed performance data 20 from the performance data processing device 200. The control device 300 then transmits the processed performance data 20 to a specified destination.

The control device 300 may adjust the processing of the performance data 10 by the performance data processing device 200 based on the communication status of the communication path from the performance data processing device 200 to the destination of the processed performance data 20. When the control device 300 relays communication between the performance data processing device 200 and the destination, the communication status between the performance data processing device 200 and the destination may be the communication status of the communication path from the performance data processing device 200 to the control device 300 and the communication status of the communication path from the control device 300 to the destination.

For example, when the performance data processing device 200 processes the performance data 10 transmitted by the performance data communication device 100 and transmits the processed performance data 20 to the performance data communication device 100, if the total time taken for all processes from when the musical instrument 500 produces a sound to when the performance output device 600 outputs the sound exceeds a certain value, the performance may feel unnatural, and the user 50 may feel stressed. Let's say that the user 50 feels stressed if this total time exceeds 10 milliseconds.

As an example of this case, it is known from the communication conditions that the time required for transmitting the processed performance data 20 from the performance data processing device 200 to the performance data communication device 100 is 4 milliseconds. Working backwards based on this, it is possible that the total time can be kept to 10 milliseconds or less if the time required for the processing of the performance data 10 by the performance data processing device 200 is kept to 3 milliseconds or less. In this case, the control device 300 may adjust the processing of the performance data 10 by the performance data processing device 200 so that the time required for this processing is 3 milliseconds or less. For example, the control device 300 may reduce the time required for the processing process by limiting the number of types of processing applied simultaneously. This reduces the discrepancy between the actual performance timing and the sound output timing due to communication delays, improving the user's musical experience.

The control device 300 may control the content of the processing so that the data volume of the processed performance data 20 is less than the data volume corresponding to the communication delay indicated by the communication status. For example, if the type of processing involves interpolation of sampling intervals, the control of the content of the processing may involve adjusting the amount of interpolation. For example, the control device 300 may reduce the data volume of the processed performance data 20 by reducing the amount of interpolation. For example, if the type of processing involves changing the number of channels of the processed performance data 20, the control device 300 may control the content of the processing so that the data volume of the processed performance data 20 is less than the data volume corresponding to the communication delay indicated by the communication delay, improving the user's musical experience.

Figure 2 shows an example of the processing flow in the system 90. Here, we explain the processing flow when the control device 300 associates a performance data processing device 200 specified by a performance data communication device 100 with the performance data communication device 100, among multiple performance data processing devices 200 that perform different processing of the performance data 10.

The control device 300 stores processing device information for multiple performance data processing devices 200. The control device 300 establishes a communication connection between the performance data communication device 100 and a performance data processing device 200 specified by the performance data communication device 100 among the multiple performance data processing devices 200.

The performance data communication device 100, for example, specifies a performance data processing device 200 selected by a user 50 of the performance data communication device 100. As a specific example, when the user 50 plays the guitar, the user 50 selects, from among multiple performance data processing devices 200, a performance data processing device 200 that can execute the desired effect.

The performance data communication device 100 transmits performance data 10 to the performance data processing device 200 via the network 80. The performance data processing device 200 processes the received performance data 10 to generate processed performance data 20, and transmits the processed performance data 20 to the performance data communication device 100 via the network 80. The performance data communication device 100 outputs the received processed performance data 20 as sound using the performance output device 600. The performance data communication device 100 and the performance data processing device 200 continuously perform these processes while the user 50 is playing the musical instrument 500.

When another performance data processing device 200 is specified by the performance data communication device 100, the control device 300 establishes a communication connection between the other performance data processing device 200 and the performance data communication device 100.

By having the system 90 execute the above-described processing, the user 50 can select the desired processing content as appropriate and obtain processed performance data 20 in which the desired processing has been applied to the performance data 10 that the user performed.

Figure 3 shows an example of the processing flow in system 90. System 90 may generate ensemble data 30 by processing and synthesizing performance data 10 from multiple users 50. Here, we will explain the processing flow when system 90 processes and synthesizes performance data 11 from a performance by user 51 and performance data 12 from a performance by user 52.

The control device 300 associates the performance data processing device 201 specified by the performance data communication device 101 with the performance data communication device 101, and associates the performance data processing device 202 specified by the performance data communication device 102 with the performance data communication device 102. The control device 300 receives processed performance data 21 from the performance data processing device 201, which is generated by processing the performance data 11 received by the performance data processing device 201 from the performance data communication device 101. The control device 300 receives processed performance data 22 from the performance data processing device 202, which is generated by processing the performance data 12 received by the performance data processing device 202 from the performance data communication device 102. The control device 300 generates ensemble data 30 by combining the received processed performance data 21 and processed performance data 22. The control device 300 transmits the ensemble data 30 to the performance data communication device 100 and the performance data processing device 200. This allows each user to obtain ensemble data that has been modified as desired from their own performance.

The control device 300 may control the performance data 11 and the performance data 12 to be combined without processing and generate the ensemble data 30. For example, the control device 300 receives the performance data 11 from the performance data communication device 101 and receives the performance data 12 from the performance data communication device 102. The control device 300 combines the performance data 11 and the performance data 12 to generate the ensemble data 30 and transmits it to the performance data communication device 101 and the performance data communication device 102. Note that the control device 300 may also process and combine only one of the performance data 11 and the performance data 12 to generate the ensemble data 30.

Figure 4 shows an example of the processing flow in system 90. System 90 may generate multiple types of ensemble data by processing and combining performance data 10 from multiple users 50. Here, we will explain the processing flow when system 90 processes performance data 11 from a performance by user 51, processes performance data 12 from a performance by user 52 in a different way, and combines the data to generate two types of ensemble data.

The control device 300 associates the performance data processing device 201 specified by the performance data communication device 101 with the performance data communication device 101, and associates the performance data processing device 202 specified by the performance data communication device 101 with the performance data communication device 102. The control device 300 associates the performance data processing device 201 specified by the performance data communication device 102 with the performance data communication device 101, and associates the performance data processing device 209 specified by the performance data communication device 102 with the performance data communication device 102. The control device 300 receives processed performance data 21 from the performance data processing device 201, which is processed performance data 11 received by the performance data processing device 201 from the performance data communication device 101. The control device 300 receives processed performance data 22 from the performance data processing device 202, which is processed performance data 12 received by the performance data processing device 202 from the performance data communication device 102. The control device 300 receives processed performance data 29 from the performance data processing device 209, which is generated by processing the performance data 12 received by the performance data processing device 209 from the performance data communication device 102. The control device 300 generates ensemble data 31 by combining the received processed performance data 21 and processed performance data 22. The control device 300 generates ensemble data 32 by combining the received processed performance data 21 and processed performance data 29. The control device 300 transmits the ensemble data 31 to the performance data communication device 101, and transmits the ensemble data 32 to the performance data communication device 102. This allows each of the different users participating in the ensemble to obtain ensemble data for the ensemble in which sound processing has been applied to the performance of each user according to their own preferences.

Here, an example has been described in which the control device 300 associates the performance data processing device 202 with the performance data communication device 102 as specified by the performance data communication device 101 in order to generate the ensemble data 31, but this is not limited to this. The specification of which performance data processing device 200 is associated with which performance data communication device 100 may be made by any of the performance data communication devices 100. For example, in order to generate the ensemble data 31, the control device 300 may associate the performance data processing device 202 specified by the performance data communication device 102 instead of the performance data communication device 101 with the performance data communication device 102.

Figure 5 shows an example of the processing flow in the system 90. The system 90 may be available to a user 59 who does not play the musical instrument 500. The user 59 may be, for example, a recording engineer, a PA engineer, a listener, etc. The user 59 may instruct the control device 300 from the performance data communication device 109 as to which performance data 10 of which user 50 (performer) should be processed by which performance data processing device 200, and how to synthesize the generated processed performance data 20. Note that, while the example described here is one in which the user 59 uses the performance data communication device 109, this is not limiting. The user 59 may use any device that is capable of communication and capable of audio output of performance data and ensemble data.

The control device 300 associates multiple performance data communication devices 100 with multiple performance data processing devices 200 based on instructions from the user 59. In the example shown in FIG. 5, the control device 300 associates the performance data communication device 101 with the performance data processing device 201, and associates the performance data communication device 102 with the performance data processing device 202. The control device 300 receives processed performance data 21 from the performance data processing device 201, which is generated by processing the performance data 11 received by the performance data processing device 201 from the performance data communication device 101. The control device 300 receives processed performance data 22 from the performance data processing device 202, which is generated by processing the performance data 12 received by the performance data processing device 202 from the performance data communication device 102. The control device 300 synthesizes the received processed performance data 21 and processed performance data 22 to generate ensemble data 31. The control device 300 transmits the ensemble data 31 to the performance data communication device 101, the performance data communication device 102, and the performance data communication device 109. In this way, by generating ensemble data by associating the performance data communication device 100 with the performance data processing device 200 in accordance with instructions from persons other than the performers, such as a recording engineer or PA engineer, the recording engineer, PA engineer, etc. can use the ensemble sound source to create ensemble works that incorporate innovative sound processing and synthesis techniques.

Figure 6 shows an example of the processing flow in system 90. System 90 may generate multiple types of ensemble data 30 from multiple types of performance data 10. Figure 6 illustrates the processing flow when multiple types of ensemble data 30 are generated from performance data 11 of user 51 and performance data 12 of user 52 in accordance with instructions from users 51 and 52 who are performers, and user 59 who does not play the instrument 500.

In the example shown in FIG. 6, the control device 300 controls the performance data communication device 101 to transmit ensemble data 31, which is a combination of processed performance data 21 obtained by processing performance data 11 using the performance data processing device 201 and processed performance data 22 obtained by processing performance data 12 using the performance data processing device 202. The control device 300 controls the performance data communication device 102 to transmit ensemble data 32, which is a combination of processed performance data 21 obtained by processing performance data 11 using the performance data processing device 201 and processed performance data 29 obtained by processing performance data 12 using the performance data processing device 209. The control device 300 controls the performance data communication device 109 to transmit ensemble data 33, which is a combination of processed performance data 21 obtained by processing performance data 11 using the performance data processing device 201 and processed performance data 22 obtained by processing performance data 12 using the performance data processing device 202.

In the example shown in FIG. 6, the control device 300 generates ensemble data 33 using a different synthesis method than that used to generate ensemble data 31. The control device 300 may generate ensemble data 31 by synthesizing processed performance data 21 and processed performance data 22 using a synthesis method instructed by the performance data communication device 101, and may generate ensemble data 33 by synthesizing processed performance data 21 and processed performance data 22 using a synthesis method instructed by the performance data communication device 109.

For example, the control device 300 generates ensemble data 31 by synthesizing the processed performance data 21 and the processed performance data 22 using a synthesis method instructed by the user 51 from the performance data communication device 101 that emphasizes the performance of the user 51. The control device 300 generates ensemble data 33 by synthesizing the processed performance data 21 and the processed performance data 22 using a synthesis method instructed by the user 59 from the performance data communication device 109 that optimizes the overall harmony of the volume balance between the processed performance data 21 and the processed performance data 22. The synthesis method instructed by the user 59 from the performance data communication device 109 may be a synthesis method that adjusts the volume balance for each range of the played instruments in the processed performance data 21 and the processed performance data 22. The synthesis method may also be a synthesis method that synthesizes the processed performance data 21 and the processed performance data 22 so as to achieve effective sound effects when output by a specific performance output device 600, depending on the specifications of the performance output device 600.

In the above example, user 59 processes performance data 11 by specifying the same performance data processing device 201 as the performance data processing device 201 specified by user 51, but user 59 may process performance data 11 by specifying a performance data processing device 200 different from the performance data processing device 201 specified by user 51, depending on the type of processing user 59 desires. User 59 processes performance data 12 by specifying the same performance data processing device 202 as the performance data processing device 202 specified by user 52, but user 59 may process performance data 12 by specifying a performance data processing device 200 different from the performance data processing device 202 specified by user 52, depending on the type of processing user 59 desires.

In the above example, the control device 300 synthesizes the processed performance data 21 and the processed performance data 22 to generate ensemble data 33 in response to an instruction from the performance data communication device 109 by the user 59. However, the method of synthesizing ensemble data 33 is not limited to this. For example, the control device 300 associates the performance data processing device 201 instructed by the user 59 from the performance data communication device 109 with the performance data communication device 109, and controls the processed performance data 21 to be sent to the performance data communication device 109, and associates the performance data processing device 202 instructed by the user 59 from the performance data communication device 109 with the performance data communication device 109, and controls the processed performance data 22 to be sent to the performance data communication device 109.

The control device 300 associates the performance data processing device 201 with the performance data communication device 109 by, for example, establishing a communication connection between the performance data processing device 201 and the performance data communication device 109. The control device 300 may establish a communication connection between the performance data processing device 201 and the performance data communication device 109 by sending an instruction to establish a communication connection to at least one of the performance data processing device 201 and the performance data communication device 109. The control device 300 associates the performance data processing device 201 with the performance data communication device 109 by having the control device 300 manage the correspondence between the performance data processing device 201 and the performance data communication device 109. In this case, the control device 300 may relay communication between the performance data processing device 201 and the performance data communication device 109.

The performance data processing device 201, which has established a communication connection with the performance data communication device 109 via the control device 300, may transmit the processed performance data 21 to the performance data communication device 109 via the network 80. The performance data communication device 109 receives the processed performance data 21.

When the control device 300 relays communication between the performance data processing device 201 and the performance data communication device 109, the performance data processing device 201 may transmit the processed performance data 21 to the control device 300 via the network 80. The control device 300 may transmit the received processed performance data 21 to the performance data communication device 109 via the network 80. The performance data communication device 109 receives the processed performance data 21.

The above explanation has been given on the correspondence between the performance data processing device 201 and the performance data communication device 109 by the control device 300, but the same applies to the correspondence between the performance data processing device 202 and the performance data communication device 109 by the control device 300.

In this way, the performance data communication device 109 receives the processed performance data 21 and the processed performance data 22. The performance data communication device 109 may synthesize the processed performance data 21 and the processed performance data 22 received by the performance data communication device 109 to generate ensemble data 33. The processed performance data 21 and the processed performance data 22 received by the performance data communication device 109 may be synthesized by a mixer connected to the performance data communication device 109 to generate ensemble data 33.

Figure 7 shows an example of the processing flow in system 90. In system 90, the performance data processing device 200, rather than the control device 300, may generate the ensemble data 30. In the example shown in Figure 7, the control device 300 associates multiple performance data communication devices (performance data communication device 101, performance data communication device 102, ..., performance data communication device 109) with the performance data processing device 200, and causes the performance data processing device 200 to process the performance data (performance data 11, performance data 12, ..., performance data 19) received from each of the multiple performance data communication devices 100 and synthesize the multiple processed performance data (processed performance data 21, processed performance data 22, ..., processed performance data 29) to generate ensemble data 30. The control device 300 may control the ensemble data 30 to be transmitted to the multiple performance data communication devices (performance data communication device 101, performance data communication device 102, ..., performance data communication device 109).

Figure 8 shows an example of the processing flow in system 90. In the example shown in Figure 8, the control device 300 associates multiple performance data communication devices (performance data communication device 101, performance data communication device 102, ..., performance data communication device 109) with the performance data processing device 200, and causes the performance data processing device 200 to process the performance data (performance data 11, performance data 12, ..., performance data 19) received from each of the multiple performance data communication devices 100 and synthesize multiple processed performance data (processed performance data 21, processed performance data 22, ..., processed performance data 29) to generate multiple different ensemble data (ensemble data 31, ensemble data 32, ..., ensemble data 39). The control device 300 may perform control so that ensemble data is transmitted to each of the multiple performance data communication devices (performance data communication device 101, performance data communication device 102, ..., performance data communication device 109) in accordance with the specifications of each of the multiple performance data communication devices (performance data communication device 101, performance data communication device 102, ..., performance data communication device 109).

Figure 9 shows an example of a system 90. In the example shown in Figure 9, the system 90 includes a performance data generation device 400 and an ensemble data generation unit 92.

The performance data generation device 400 has a pseudo-generation unit that generates pseudo-performance data of a specified user (performer) playing a specified piece of music. The ensemble data generation unit 92 generates ensemble data 30 by combining processed performance data 20, which is generated by a performance data processing device 200 associated with the performance data communication device 100 by the control device 300 and processing the performance data 10 received from the performance data communication device 100, with pseudo-performance data 40 that is pseudo-generated by the performance data generation device 400.

In the example shown in FIG. 9, the ensemble data generation unit 92 is arranged independently of the performance data processing device 200 and the control device 300, but the performance data processing device 200 may have the ensemble data generation unit 92, or the control device 300 may have the ensemble data generation unit 92.

For example, in accordance with instructions from the performance data communication device 100, the control device 300 associates the performance data communication device 100 with the performance data processing device 200 and specifies a performer and a piece of music to the performance data generation device 400. The performance data communication device 100 transmits performance data 10 of the piece of music performed by the user 50 to the performance data processing device 200. The performance data processing device 200 then processes the performance data 10 to generate processed performance data 20 and transmits the processed performance data 20 to the ensemble data generation unit 92. The pseudo-generation unit of the performance data generation device 400 generates pseudo-performance data of the specified performer playing the specified piece of music and transmits the pseudo-performance data 20 to the ensemble data generation unit 92. The ensemble data generation unit 92 combines the received processed performance data 20 and pseudo-performance data 40 to generate ensemble data 30 and transmits it to the performance data communication device 100. This allows the user 50 to apply desired processing to their own performance and have a simulated experience of playing in an ensemble with a famous performer, etc.

In addition, ensemble data may be generated by combining the performance data 10 and the pseudo performance data 40 without processing the performance data 10. For example, the control device 300 specifies performers and music to the performance data generation device 400 in accordance with instructions from the performance data communication device 100. The performance data communication device 100 transmits performance data 10 of the music performed by the user 50 to the ensemble data generation unit 92. The pseudo generation unit of the performance data generation device 400 pseudo-generates pseudo performance data 40 of the specified performer playing the specified music, and transmits it to the ensemble data generation unit 92. The ensemble data generation unit 92 synthesizes the received performance data 10 and pseudo performance data 40 to generate ensemble data 30, which is then transmitted to the performance data communication device 100.

The system 90 may generate ensemble data combining performances by multiple users 50 and a simulated performance. For example, if one of the multiple users 50 who are ensemble members is absent, the performance data generation device 400 simulates the ensemble by generating performance data for the absent user 50. In this case, for example, the control device 300, in accordance with instructions from at least one of the multiple users 50, associates a performance data communication device 100 with a performance data processing device 200 for each of the multiple users 50 who are present, and specifies the absent user 50 and the piece of music to be played in the ensemble to the performance data generation device 400. The multiple performance data processing devices 200 process the performance data 10 received from the multiple performance data communication devices 100 to generate processed performance data 20, which are then transmitted to the ensemble data generation unit 92. The pseudo-generation unit of the performance data generation device 400 generates pseudo-performance data 40 in which a specified user 50 plays a specified piece of music, and transmits it to the ensemble data generation unit 92. The ensemble data generation unit 92 combines multiple pieces of processed performance data 20 and the pseudo-performance data 40 to generate ensemble data 30, which it transmits to multiple performance data communication devices 100. In this way, even if one of the ensemble members is absent from the ensemble, a pseudo-ensemble can be realized, providing an environment in which ensemble practice can be carried out.

Figure 10 shows an example of the functional configuration of the control device 300. The control device 300 includes a memory unit 310, a receiving unit 320, a transmitting unit 330, a matching unit 340, a destination designation receiving unit 350, an ensemble data generating unit 360, a communication status acquiring unit 370, and a processing adjustment unit 380. Note that it is not essential for the control device 300 to include all of these units. In addition to these "units," the control device 300 may further include any "units" for realizing the functions of the control device 300.

The memory unit 310 stores various types of information. The memory unit 310 stores processing device information. The memory unit 310 may store information on multiple processing devices. The memory unit 310 stores communication device information. The memory unit 310 may store information on multiple communication devices.

The receiving unit 320 and the transmitting unit 330 communicate with various devices via the network 80. For example, the receiving unit 320 and the transmitting unit 330 communicate with the performance data communication device 100 via the network 80. For example, the receiving unit 320 and the transmitting unit 330 communicate with the performance data processing device 200 via the network 80. For example, the receiving unit 320 and the transmitting unit 330 communicate with the performance data generation device 400 via the network 80. For example, the receiving unit 320 and the transmitting unit 330 communicate with the ensemble data generation unit 92 via the network 80.

The association unit 340 associates the performance data communication device 100 with the performance data processing device 200 using information stored in the storage unit 310. The association unit 340 associates the performance data communication device 100 with the performance data processing device 200 using communication device information and processing device information so that the performance data 10 transmitted by the performance data communication device 100 can be processed by the performance data processing device 200.

The associating unit 340 may associate the performance data communication device 100 with the performance data processing device 200 in accordance with external specifications. For example, the associating unit 340 may associate the performance data communication device 100 with the performance data processing device 200 in accordance with specifications from the performance data communication device 100. The associating unit 340 may associate the performance data communication device 100 with the performance data processing device 200 specified by the performance data communication device 100, out of the multiple performance data processing devices 200.

The correspondence unit 340 may establish a communication connection between the performance data communication device 100 and the performance data processing device 200. The correspondence unit 340 may manage the correspondence between the performance data communication device 100 and the performance data processing device 200. The correspondence unit 340 may control the receiving unit 320 and the transmitting unit 330 to relay communication between the performance data communication device 100 and the performance data processing device 200 that are in a correspondence relationship.

The destination designation receiving unit 350 accepts designation of a destination to which the processed performance data 20 is to be transmitted. The destination designation receiving unit 350 accepts, for example, from a performance data communication device 100, designation of a destination to which the processed performance data 20, which is obtained by processing the performance data 10 transmitted by the performance data communication device 100, is to be transmitted. The destination designation receiving unit 350 accepts, for example, from a performance data communication device 100, designation of a destination to which the processed performance data 20, which is obtained by processing the performance data 10 transmitted by a performance data communication device 100 other than the performance data communication device 100, is to be transmitted.

The correspondence assignment unit 340 controls the transmission of the processed performance data 20 to the destination specified by the destination specification acceptance unit 350. If a communication connection has been established between the performance data processing device 200 and the destination, the correspondence assignment unit 340 may cause the performance data processing device 200 to transmit the processed performance data 20 to the destination via the communication connection. If a communication connection has not been established between the performance data processing device 200 and the destination, the correspondence assignment unit 340 may establish a communication connection and cause the performance data processing device 200 to transmit the processed performance data 20 to the destination via the communication connection. If the correspondence assignment unit 340 manages the correspondence between the performance data processing device 200 and the destination, the correspondence assignment unit 340 may receive the processed performance data 20 from the performance data processing device 200 and transmit it to the destination. If the correspondence between the performance data processing device 200 and the transmission destination is not being managed, the association unit 340 may newly manage the correspondence between the performance data processing device 200 and the transmission destination, and perform control so that the processed performance data 20 is received from the performance data processing device 200 and transmitted to the transmission destination.

The association unit 340 may cause the performance data processing device 200 to generate ensemble data. For example, the association unit 340 may associate multiple performance data communication devices 100 with the performance data processing device 200, and cause the performance data processing device 200 to process multiple pieces of performance data 10 received from the multiple performance data communication devices 100 to generate multiple pieces of processed performance data 20, and to generate ensemble data by synthesizing the multiple pieces of processed performance data 20. The association unit 340 may also cause the performance data processing device 200 to generate ensemble data by synthesizing multiple pieces of performance data 10 received from the multiple performance data communication devices 100. The association unit 340 may also cause the performance data processing device 200 to generate ensemble data by synthesizing a portion of the multiple pieces of performance data 10 received from the multiple performance data communication devices 100 with processed performance data 20 obtained by processing the remaining portions.

The ensemble data generation unit 360 generates ensemble data 30. The ensemble data generation unit 360 may generate ensemble data 30 by combining multiple pieces of processed performance data 20. For example, the receiving unit 320 receives multiple pieces of processed performance data 20 that have been created by multiple performance data processing devices 200 processing multiple pieces of performance data 10 transmitted by multiple performance data communication devices 100, and the ensemble data generation unit 360 generates ensemble data 30 by combining the multiple pieces of processed performance data 20 received by the receiving unit 320. As a specific example, the association unit 340 associates the first performance data communication device 100 with the first performance data processing device 200, and associates the second performance data communication device 100 with the second performance data processing device 200, the receiving unit 320 receives from the first performance data processing device 200 first processed performance data 20 that is obtained by processing the performance data 10 that the first performance data processing device 200 received from the first performance data communication device 100, and the second performance data processing device 200 receives from the second performance data processing device 200 second processed performance data 20 that is obtained by processing the performance data 10 that the second performance data processing device 200 received from the second performance data communication device 100, and the ensemble data generation unit 360 generates ensemble data by combining the first processed performance data 20 and the second processed performance data 20 received by the receiving unit 320.

The ensemble data generating unit 360 may generate ensemble data by combining multiple pieces of performance data 10. For example, the receiving unit 320 receives multiple pieces of performance data 10 transmitted by multiple performance data communication devices 100, and the ensemble data generating unit 360 combines the pieces of performance data 10 received by the receiving unit 320 from the multiple performance data communication devices 100 to generate ensemble data 30.

The ensemble data generation unit 360 may generate ensemble data 30 by combining one or more pieces of performance data 10 and one or more pieces of processed performance data 20.

The ensemble data generation unit 360 may generate ensemble data 30 by combining one or more pieces of processed performance data 20 with pseudo performance data 40 generated by the pseudo generation unit of the performance data generation device 400. For example, the receiving unit 320 receives one or more pieces of processed performance data 20 that have been processed by one or more performance data processing devices 200 from one or more pieces of performance data 10 transmitted by one or more performance data communication devices 100, and the pseudo performance data 40, and the ensemble data generation unit 360 generates ensemble data 30 by combining the one or more pieces of processed performance data 20 and the pseudo performance data 40 received by the receiving unit 320. The ensemble data generation unit 360 may generate ensemble data 30 by combining one or more pieces of processed performance data 20 with multiple pieces of pseudo performance data 40.

The ensemble data generation unit 360 may generate ensemble data 30 by combining one or more pieces of performance data 10 with pseudo performance data 40 generated by the pseudo generation unit of the performance data generation device 400. For example, the receiving unit 320 receives one or more pieces of performance data 10 and pseudo performance data 40 transmitted by one or more performance data communication devices 100, and the ensemble data generation unit 360 generates ensemble data 30 by combining the one or more pieces of performance data 10 and pseudo performance data 40 received by the receiving unit 320. The ensemble data generation unit 360 may generate ensemble data 30 by combining one or more pieces of performance data 10 and multiple pieces of pseudo performance data 40.

The ensemble data generation unit 360 may generate ensemble data 30 by combining one or more pieces of performance data 10, one or more pieces of processed performance data 20, and multiple pieces of pseudo performance data 40. The ensemble data generation unit 360 may generate ensemble data 30 by combining one or more pieces of performance data 10, one or more pieces of processed performance data 20, and multiple pieces of pseudo performance data 40.

The communication status acquisition unit 370 acquires the communication status of the communication path from the performance data processing device 200 to the destination of the processed performance data 20. Examples of communication status include, but are not limited to, communication speed, communication delay, communication traffic, and communication capacity.

When a communication connection has been established between the performance data processing device 200 and the transmission destination, the communication status acquisition unit 370 may acquire the communication status of the communication path between the performance data processing device 200 and the transmission destination. The communication status acquisition unit 370 may acquire the communication status of the communication path between the performance data processing device 200 and the transmission destination from at least one of the performance data processing device 200 and the transmission destination.

When the control device 300 relays communication between the performance data processing device 200 and the destination, the communication status acquisition unit 370 may acquire the communication status of the communication path between the performance data processing device 200 and the control device 300, and the communication status of the communication path between the control device 300 and the destination.

The processing adjustment unit 380 may adjust the processing of the performance data 10 by the performance data processing device 200 based on the communication status acquired by the communication status acquisition unit 370. For example, consider a case where performance data 10 produced by a user 50 playing an instrument 500 is transmitted from the performance data communication device 100, processed by the performance data processing device 200, and synthesized by the performance data processing device 200, the control device 300, or the ensemble data generation unit 92 to generate ensemble data 30, which is received by the performance data communication device 100 and output as audio from the performance output device 600. In this case, the processing adjustment unit 380 causes the performance data processing device 200 to adjust the processing of the performance data 10 so that the total time (sometimes referred to as total processing time) required for all processes from when the user 50 sounds the instrument 500 to when the performance output device 600 outputs as audio the ensemble data 30 derived from that sound does not exceed a predetermined threshold corresponding to the time at which the user 50 begins to feel stressed. For example, the processing adjustment unit 380 causes the performance data processing device 200 to adjust the processing of the performance data 10 so as to reduce the time required for the performance data processing device 200 to process the performance data 10 out of the total processing time. For example, the processing adjustment unit 380 limits the number of types of processing that the performance data processing device 200 can perform simultaneously.

The processing adjustment unit 380 may control the content of the processing so that the data volume of the processed performance data 20 is less than the data volume corresponding to the communication delay indicated by the communication status. For example, if the type of processing of the performance data 10 by the performance data processing device 200 involves interpolation of the sampling interval of the performance data 10, the processing adjustment unit 380 causes the performance data processing device 200 to adjust the amount of interpolation. For example, the processing adjustment unit 380 causes the performance data processing device 200 to reduce the amount of interpolation. For example, if the type of processing of the performance data 10 by the performance data processing device 200 involves changing the number of channels of the performance data 10, the processing adjustment unit 380 causes the performance data processing device 200 to adjust the number of channels. For example, the processing adjustment unit 380 causes the performance data processing device 200 to reduce the number of channels.

Figure 11 shows an example of the functional configuration of the performance data generation device 400. The performance data generation device 400 comprises a learning model generation unit 410, a learning model storage unit 420, a musical score data receiving unit 430, and a pseudo-generation unit 440. Note that it is not essential for the performance data generation device 400 to have all of these units. In addition to these "units," the performance data generation device 400 may further comprise any "unit" for realizing the functions possessed by the performance data generation device 400.

The learning model generation unit 410 generates a learning model. The learning model generation unit 410 may generate a learning model that receives musical score data as input and performance data as output. For example, the learning model generation unit 410 generates the learning model through machine learning using performance data generated by a user playing a piece of music on an instrument and training data including musical score data for the piece of music. The training data may include a large amount of performance data and musical score data generated by a user playing various pieces of music. The learning model can simulate performance data of a piece of music based on input musical score data played by the user. The learning model generation unit 410 may generate a learning model for each of multiple users.

The learning model generation unit 410 may perform machine learning using ensemble data generated by multiple users playing together and learning data including performance data by some of the users in the ensemble to generate a learning model that takes performance data by some of the users as input and outputs ensemble data by the multiple users. For example, the learning model generation unit 410 may input performance data by a first user, generated by machine learning using ensemble data generated by a first user and a second user playing together and learning data including the performance data by the first user in the ensemble, and generate a learning model that outputs ensemble data by the first user and the second user. The learning data may include a large amount of ensemble data generated by the first user and the second user playing various pieces of music together and performance data of the first user. According to this learning model, even when the second user is absent, performance data of the second user can be simulated to generate ensemble data with the first user.

The learning model generation unit 410 may perform machine learning using ensemble data generated by multiple users playing together and learning data including performance data by some of the users in the ensemble, to generate a learning model that takes performance data by some of the users as input and performance data by the remaining users as output. For example, the learning model generation unit 410 may generate a learning model that takes performance data by a first user as input and performance data by the second user as output, the learning model being generated by machine learning using ensemble data generated by a first user and a second user playing together and learning data including performance data by the first user in the ensemble. The learning data may include a large amount of ensemble data generated by the first user and the second user playing various pieces of music together, as well as performance data of the first user. This learning model makes it possible to generate simulated performance data 40 of the second user playing together with the first user, even when the second user is absent.

The learning model storage unit 420 stores learning models. The learning model storage unit 420 may store a learning model that takes sheet music data as input and performance data as output. The learning model storage unit 420 may store multiple learning models that take sheet music data as input and output performance data of the user playing the music of the sheet music data, respectively, corresponding to multiple users. The learning model storage unit 420 may store a learning model that takes sheet music data as input and output performance data of a famous performer playing the music of the sheet music data. The learning model storage unit 420 may store a learning model that takes performance data by some of the multiple users as input and output ensemble data of the multiple users. The learning model storage unit 420 may store a learning model that takes performance data by some of the multiple users as input and output performance data of the remaining users of the multiple users. The learning model storage unit 420 may store learning models that take performance data by some of the multiple users as input and output performance data of the remaining users of the multiple users. The learning model storage unit 420 may store learning models generated by the learning model generation unit 410. The learning model storage unit 420 may store learning models generated by other devices.

The score data receiving unit 430 receives score data for the music piece to be simulated from an external source. For example, the score data receiving unit 430 receives score data from the performance data communication device 100. For example, the score data receiving unit 430 receives score data from the control device 300. For example, the score data receiving unit 430 receives score data from the performance data processing device 200.

The pseudo generation unit 440 generates pseudo performance data. The pseudo generation unit 440 may generate pseudo performance data of a specified user playing a specified piece of music. The pseudo generation unit 440 may accept designation of a user and piece of music from the performance data communication device 100. The pseudo generation unit 440 may accept designation of a user and piece of music from the control device 300. The pseudo generation unit 440 may accept designation of a user and piece of music from the performance data processing device 200. The pseudo generation unit 440 may use the score data of the specified piece of music received by the score data receiving unit 430. The pseudo generation unit 440 may input the score data of the specified piece of music into a learning model corresponding to the specified user, which is stored in the learning model storage unit 420, and obtain the performance data output from the learning model as pseudo performance data 40.

The pseudo-generation unit 440 may generate pseudo-performance data for the remaining users from the performance data of some of the multiple users. The pseudo-generation unit 440 may acquire the performance data of some of the multiple users from their performance data communication devices 100, input the data to the corresponding learning models stored in the learning model storage unit 420, and acquire the performance data output from the learning models as pseudo-performance data 40 for the remaining users. For example, the pseudo-generation unit 440 may input the performance data of a first user out of a first and second user into learning models corresponding to the first and second users, and acquire the performance data output from the learning models as pseudo-performance data 40 for the second user.

The pseudo generation unit 440 may generate pseudo ensemble data. For example, the pseudo generation unit 440 may generate pseudo ensemble data for multiple users from the performance data of some of the multiple users. The pseudo generation unit 440 may acquire the performance data of some of the multiple users from their performance data communication devices 100, input the data to the corresponding learning models stored in the learning model storage unit 420, and acquire the ensemble data output from the learning models as pseudo ensemble data for the multiple users. For example, the pseudo generation unit 440 may input the performance data of a first user out of a first and second user into learning models corresponding to the first and second users, and acquire the ensemble data output from the learning models as pseudo ensemble data for the first and second users.

FIG. 12 shows a schematic diagram of an example of the hardware configuration of a computer 1200 functioning as the performance data communication device 100, the performance data processing device 200, the control device 300, the performance data generation device 400, or the ensemble data generation unit 92. A program installed on the computer 1200 can cause the computer 1200 to function as one or more "units" of the device according to this embodiment, or to execute operations or one or more "units" associated with the device according to this embodiment, and/or to execute a process or steps of the process according to this embodiment. Such a program may be executed by the CPU 1212 to cause the computer 1200 to perform specific operations associated with some or all of the blocks in the flowcharts and block diagrams described herein.

The computer 1200 according to this embodiment includes a CPU 1212, a GPU 1213, a RAM 1214, and a graphics controller 1216, which are interconnected by a host controller 1210. The computer 1200 also includes input/output units such as a communications interface 1222, a storage device 1224, a DVD drive, and an IC card drive, which are connected to the host controller 1210 via an input/output controller 1242. The DVD drive may be a DVD-ROM drive, a DVD-RAM drive, or the like. The storage device 1224 may be a hard disk drive, a solid-state drive, or the like. The computer 1200 also includes a ROM 1230 and legacy input/output units such as a keyboard, which are connected to the input/output controller 1242 via an input/output chip 1240.

The CPU 1212 operates according to programs stored in the ROM 1230 and RAM 1214, thereby controlling each unit. The graphics controller 1216 acquires image data generated by the CPU 1212 into a frame buffer provided in the RAM 1214 or into the graphics controller itself, and causes the image data to be displayed on the display device 1218.

The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD drive reads programs or data from a DVD-ROM or the like and provides them to the storage device 1224. The IC card drive reads programs and data from an IC card and/or writes programs and data to an IC card.

ROM 1230 stores therein a boot program and the like that is executed by computer 1200 upon activation, and/or programs that depend on the hardware of computer 1200. I/O chip 1240 may also connect various I/O units to I/O controller 1242 via USB ports, parallel ports, serial ports, keyboard ports, mouse ports, etc.

The programs are provided on a computer-readable storage medium such as a DVD-ROM or IC card. The programs are read from the computer-readable storage medium, installed in storage device 1224, RAM 1214, or ROM 1230, which are also examples of computer-readable storage media, and executed by CPU 1212. The information processing described in these programs is read by computer 1200, resulting in cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be configured by implementing the operation or processing of information in accordance with the use of computer 1200.

For example, when communication is performed between computer 1200 and an external device, CPU 1212 may execute a communication program loaded into RAM 1214 and instruct communication interface 1222 to perform communication processing based on the processing described in the communication program. Under the control of CPU 1212, communication interface 1222 reads transmission data stored in a transmission buffer area provided in RAM 1214, storage device 1224, DVD-ROM, or a recording medium such as an IC card, and transmits the read transmission data to the network, or writes received data received from the network to a reception buffer area or the like provided on the recording medium.

The CPU 1212 may also cause all or a necessary portion of a file or database stored on an external recording medium such as the storage device 1224, a DVD drive (DVD-ROM), an IC card, etc. to be read into the RAM 1214, and perform various types of processing on the data on the RAM 1214. The CPU 1212 may then write the processed data back to the external recording medium.

Various types of information, such as various types of programs, data, tables, and databases, may be stored on the recording medium and may undergo information processing. CPU 1212 may perform various types of processing on data read from RAM 1214, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search/replacement, etc., as described throughout this disclosure and specified by the program's instruction sequence, and write the results back to RAM 1214. CPU 1212 may also search for information in files, databases, etc. on the recording medium. For example, if multiple entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored on the recording medium, CPU 1212 may search for an entry whose attribute value of the first attribute matches a specified condition from among the multiple entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

The programs or software modules described above may be stored on computer-readable storage media on or near computer 1200. Recording media such as a hard disk or RAM provided within a server system connected to a dedicated communications network or the Internet can also be used as computer-readable storage media, thereby providing the programs to computer 1200 via the network.

The blocks in the flowcharts and block diagrams in this embodiment may represent stages of a process in which an operation is performed or "parts" of a device responsible for performing the operation. Particular stages and "parts" may be implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable storage medium, and/or a processor provided with computer-readable instructions stored on a computer-readable storage medium. Dedicated circuitry may include digital and/or analog hardware circuitry, and may include integrated circuits (ICs) and/or discrete circuits. Programmable circuitry may include reconfigurable hardware circuitry including AND, OR, XOR, NAND, NOR, and other logical operations, flip-flops, registers, and memory elements, such as field programmable gate arrays (FPGAs) and programmable logic arrays (PLAs).

A computer-readable storage medium may include any tangible device capable of storing instructions that are executed by a suitable device. As a result, a computer-readable storage medium having instructions stored thereon comprises an article of manufacture, including instructions that can be executed to create means for performing the operations specified in the flowcharts or block diagrams. Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, etc. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), electrically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disc (DVD), Blu-ray disc, memory stick, integrated circuit card, etc.

The computer-readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk®, JAVA®, C++, etc., and conventional procedural programming languages such as the "C" programming language or similar programming languages.

The computer-readable instructions may be provided to a general-purpose computer, a special-purpose computer, or a processor of another programmable data processing device, or a programmable circuit, either locally or via a local area network (LAN) or a wide area network (WAN) such as the Internet, so that the processor or programmable circuit of the programmable data processing device, such as a computer, executes the computer-readable instructions to generate means for performing the operations specified in the flowchart or block diagram. Here, the computer may be a personal computer (PC), tablet computer, smartphone, workstation, server computer, general-purpose computer, special-purpose computer, etc., or may be a computer system in which multiple computers are connected. Such a computer system in which multiple computers are connected is also called a distributed computing system, and is a broad definition of computer. In a distributed computing system, multiple computers collectively execute a program by each executing a portion of the program and passing data between computers as needed during program execution.

Examples of processors include computer processors, central processing units (CPUs), processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc. A computer may have one processor or multiple processors. In a multiprocessor system with multiple processors, each processor executes a portion of a program and passes data between processors as needed during program execution, allowing the multiple processors to collectively execute the program. For example, in multitasking, each of the multiple processors may execute a portion of each task in small chunks by switching tasks at each time slice. In this case, which portion of a program each processor executes changes dynamically. Which portion of a program each of the multiple processors executes may also be statically determined by multiprocessor-aware programming.

The present invention has been described above using embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be clear to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the claims that such modifications and improvements can also be included within the technical scope of the present invention.

The order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before" or "prior to," and it should be noted that processes can be performed in any order unless the output of a previous process is used in a subsequent process. Even if the operational flow in the claims, specifications, and drawings is described using terms such as "first," "next," etc. for convenience, this does not mean that the processes must be performed in that order.

10 Performance data, 11 Performance data, 12 Performance data, 19 Performance data, 20 Processed performance data, 21 Processed performance data, 22 Processed performance data, 29 Processed performance data, 30 Ensemble data, 31 Ensemble data, 32 Ensemble data, 33 Ensemble data, 39 Ensemble data, 40 Pseudo performance data, 50 User, 51 User, 52 User, 59 User, 80 Network, 90 System, 92 Ensemble data generation unit, 100 Performance data communication device, 101 Performance data communication device, 102 Performance data communication device, 109 Performance data communication device, 110 Transmission unit, 120 Reception unit, 130 Performance input unit, 140 Performance output unit, 150 Conversion unit, 160 Conversion unit, 200 Performance data processing device, 201 Performance data processing device, 202 Performance data processing device, 209 performance data processing device, 210 transmission unit, 220 reception unit, 230 performance data processing unit, 300 control device, 310 storage unit, 320 reception unit, 330 transmission unit, 340 correspondence unit, 350 destination designation reception unit, 360 ensemble data generation unit, 370 communication status acquisition unit, 380 processing adjustment unit, 400 performance data generation device, 410 learning model generation unit, 420 learning model storage unit, 430 musical score data reception unit, 440 simulation generation unit, 500 musical instrument, 600 performance output device, 1200 computer, 1210 host controller, 1212 CPU, 1213 GPU, 1214 RAM, 1216 graphics controller, 1218 display device, 1222 communication interface, 1224 storage device, 1230 ROM, 1240 I/O chip, 1242 I/O controller

Claims (15)

a storage unit for storing information about a performance data processing device that processes performance data of a musical instrument performance received via a network to generate processed performance data;
a destination specification receiving unit that receives, from a device other than the performance data communication device having the transmitting unit that transmits the performance data, a specification of a destination of the performance data to which the performance data is to be transmitted and a specification of a destination of the processed performance data to which the processed performance data is to be transmitted;
a correspondence unit that associates the performance data communication device with the performance data processing device using information about the performance data processing device stored in the storage unit, so that the performance data transmitted by the transmission unit of the performance data communication device can be processed by the performance data processing device;
The correspondence unit controls the performance data to be transmitted via the network to a destination of the performance data whose designation has been accepted by the destination designation acceptance unit, and controls the processed performance data to be transmitted via the network to a destination of the processed performance data whose designation has been accepted by the destination designation acceptance unit . the storage unit stores information on a plurality of the performance data processing devices, each of which processes performance data differently;
2. The control device according to claim 1, wherein the destination designation receiving unit receives a designation of a destination to which the performance data is to be transmitted from among the plurality of performance data processing devices. the association unit associates the first performance data communication device with the first performance data processing device, and associates the second performance data communication device with the second performance data processing device;
The control device
a receiving section for receiving, from the first performance data processing device, first processed performance data obtained by processing the performance data received from the first performance data communication device, and for receiving, from the second performance data processing device, second processed performance data obtained by processing the performance data received from the second performance data communication device;
3. The control device according to claim 1 , further comprising: an ensemble data generating unit that generates ensemble data by combining the first processed performance data and the second processed performance data received by the receiving unit. 4. The control device according to claim 3, wherein the ensemble data generating unit generates ensemble data by synthesizing the first processed performance data and the second processed performance data based on a synthesis method instructed by a device other than the performance data communication device. 3. The control device according to claim 1, wherein the association unit associates the plurality of performance data communication devices with the performance data processing device, and causes the performance data processing device to process the performance data received from each of the plurality of performance data communication devices and synthesize the plurality of processed performance data to generate ensemble data. 6. The control device according to claim 5, wherein the association unit causes the performance data processing device to process the performance data received from each of the plurality of performance data communication devices based on a synthesis method instructed by another device other than the performance data communication device, and to generate ensemble data by synthesizing the plurality of processed performance data. The control device according to claim 1 or 2 ;
A system comprising: a performance data generating device;
The performance data generating device
a pseudo-generation unit that generates pseudo-performance data of a designated user playing a designated piece of music;
The system comprises:
The system further comprises an ensemble data generation unit that generates ensemble data by combining processed performance data, which is obtained by the performance data processing device associated with the performance data communication device by the association unit processing the performance data received from the performance data communication device, with pseudo performance data that is pseudo-generated by the pseudo generation unit. The performance data generating device
a learning model storage unit that stores performance data generated by a user playing a musical piece on a musical instrument and a learning model that receives musical score data as input and outputs performance data, the learning model being generated by machine learning using learning data including musical score data of the musical piece;
a musical score data receiving unit for receiving musical score data;
The system described in claim 7, wherein the pseudo-generation unit inputs the musical score data received by the musical score data receiving unit into the learning model corresponding to the specified user, and obtains the performance data output from the learning model as pseudo-performance data. a storage unit for storing information about a performance data processing device that processes performance data of a musical instrument performance received via a network to generate processed performance data;
a destination designation receiving unit that receives designation of a destination to which the processed performance data is to be transmitted;
a communication status acquisition unit that acquires a communication status of a communication path from the performance data processing device to the transmission destination;
a processing adjustment unit that adjusts processing of the performance data by the performance data processing device based on the communication status ;
a correspondence unit that associates the performance data communication device with the performance data processing device using information about the performance data processing device stored in the storage unit, in order to have the performance data processing device process the performance data transmitted by the transmission unit of the performance data communication device that has the transmission unit that transmits the performance data;
Equipped with
The association unit controls the processed performance data to be transmitted via the network to the destination designated by the destination designation receiving unit . 10. The control device according to claim 9 , wherein the processing adjustment section controls the content of the processing so that the amount of data of the processed performance data is less than the amount of data corresponding to the communication delay indicated by the communication status. a storage unit for storing information about a performance data processing device that processes performance data of a musical instrument performance received via a network to generate processed performance data;
a correspondence unit that associates the performance data communication device with the performance data processing device using information about the performance data processing device stored in the storage unit, in order to have the performance data processing device process the performance data transmitted by the transmission unit of the performance data communication device that has the transmission unit that transmits the performance data;
a control device comprising:
A performance data generating device
A system comprising:
The performance data generating device
a pseudo-generation unit that generates pseudo-performance data of a specified user playing a specified piece of music;
a learning model storage unit that stores ensemble data generated by a first user and a second user playing together, and a learning model that receives performance data by the first user as input and outputs performance data by the second user, the learning model being generated by machine learning using learning data including performance data by the first user in the ensemble ;
and
the pseudo-performance generation unit inputs performance data by the first user into the learning model, and acquires performance data output from the learning model as pseudo-performance data;
The system comprises:
The system further includes an ensemble data generation unit that generates ensemble data by combining processed performance data, which is generated by the performance data processing device associated with the performance data communication device by the association unit, and performance data received from the performance data communication device, with pseudo performance data that is pseudo-generated by the pseudo generation unit . 1. A computer-implemented control method comprising:
a storing step of storing in a storage unit information about a performance data processing device that processes performance data of a performance using a musical instrument, the performance data being received via the network, and generates processed performance data;
a destination designation receiving step of receiving, from a device other than the performance data communication device having a transmitting unit for transmitting the performance data, designation of a destination of the performance data to which the performance data is to be transmitted and designation of a destination of the processed performance data to which the processed performance data is to be transmitted;
a correspondence step of associating the performance data communication device with the performance data processing device by using information about the performance data processing device stored in the storage unit, so that the performance data transmitted by the transmission unit of the performance data communication device is processed by the performance data processing device;
The correspondence step controls the performance data to be transmitted via the network to a destination of the performance data designated in the destination designation receiving step, and controls the processed performance data to be transmitted via the network to a destination of the processed performance data designated in the destination designation receiving step . 1. A computer-implemented control method comprising:
a storing step of storing in a storage unit information about a performance data processing device that processes performance data of a performance using a musical instrument, the performance data being received via the network, and generates processed performance data;
a destination designation receiving step of receiving a designation of a destination to which the processed performance data is to be transmitted;
a communication status acquisition step of acquiring a communication status of a communication path from the performance data processing device to the transmission destination;
a processing adjustment step of adjusting the processing of the performance data by the performance data processing device based on the communication status; a correspondence step of associating the performance data communication device with the performance data processing device using information about the performance data processing device stored in the storage unit, in order to have the performance data processing device process the performance data transmitted by the transmission unit of the performance data communication device having the transmission unit for transmitting the performance data;
Equipped with
The control method, wherein the associating step controls the processed performance data so that the processed performance data is transmitted via the network to the destination designated in the destination designation receiving step. 1. A computer-implemented control method comprising:
a storing step of storing in a storage unit information about a performance data processing device that processes performance data of a performance using a musical instrument, the performance data being received via the network, and generates processed performance data;
a correspondence step of associating the performance data communication device with the performance data processing device using information about the performance data processing device stored in the storage unit, in order to have the performance data processing device process the performance data transmitted by the transmission unit of the performance data communication device, which has a transmission unit for transmitting the performance data;
a pseudo-generation step of pseudo-generating performance data of a specified piece of music played by a specified user;
Equipped with
the pseudo-generation step inputs the performance data of the first user to a learning model that receives ensemble data generated by the first user and a second user playing together and generates training data including the performance data of the first user in the ensemble, and outputs performance data of the second user, and acquires the performance data output from the learning model as pseudo-performance data;
The control method includes:
The control method further comprises an ensemble data generation step of generating ensemble data by combining processed performance data obtained by processing the performance data received from the performance data communication device by the performance data processing device associated with the performance data communication device in the association step with the pseudo performance data generated in the pseudo generation step. A program for causing a computer to execute the control method according to any one of claims 12 to 14 .