JP7851546B2 - Music performance device using a mobile device - Google Patents

Description

This invention relates to a music playback device using a mobile device such as a smartphone.

Conventionally, as shown in Patent Document 1, for example, there are known musical toys that simulate an automatic piano. These toys involve printing lines representing musical notes onto a roll of paper or the like to create a musical score, and then using an image input means to take still images as input from digital video data created by photographing, recording, or other methods of capturing the movement of this roll of paper. An image processing means is also provided to detect the recorded musical notes from the input still images through image processing. Based on the detected notes, the toy plays the musical score by producing sounds corresponding to the notes.

Furthermore, as an advanced form of this toy, as shown in Patent Document 2, a method and apparatus are known for electronic devices such as mobile terminals that detect the rotation speed of a gesture from a rotation operation on a touch panel recognized as a gesture event at a single contact point, determine the performance position of a musical score based on the detected rotation speed, and perform the musical score based on the determined performance position information.

If a method or device implemented on a mobile terminal device, as shown in Patent Document 2, is implemented, it is conceivable that multiple people could easily bring these devices together and perform music using multiple devices, i.e., play in an ensemble. However, when actually performing in an ensemble, the only things displayed on each performer's mobile terminal display are thin lines of varying positions and lengths that move from top to bottom of the screen according to the rotation speed of the performer's own gestures, as disclosed in Figures 1 and 7 of Patent Document 2. This makes it difficult for the performers to understand which part of the musical score they are playing, and even if a leader is designated and the other performers (referred to as ensemble members) try to match the timing and speed of their own playing by listening to the leader's performance, as is done in typical ensembles, it is difficult to accurately synchronize.

On the other hand, with the widespread use of network technology, techniques are now known that allow multiple musicians to perform together using network communication functions, even when they are located in different places.

For example, Patent Document 3 discloses a technology relating to ensemble and automated performance by multiple people via a network. This technology uses a performance control unit to select at least one performance part and necessary automated performance parts to realize ensemble performance over a network.

Furthermore, Patent Document 4 discloses a technology for transmitting control information of a designated electronic musical instrument via a network and displaying the performance status of electronic musical instruments at other locations on a display.

Furthermore, Patent Documents 5, 6, and 7 also disclose technologies related to real-time ensemble systems using network functions.

Furthermore, Patent Documents 8 and 9 disclose methods for correcting performance delays caused by network communication during real-time ensemble performances using network functionality.

Patent documents 10 and 11 disclose a method for coordinating timing when performing real-time ensemble playing using network functions. This method involves predicting movement states, generating performance information according to the predicted movement states, and providing notification actions using information other than visual/auditory signals, such as operation information input from other devices.

Furthermore, Patent Document 12 discloses a technique for displaying diverse musical scores by displaying a standard five-line staff on multiple layers, setting display modes for various elements such as musical notes on each layer, and overlaying the layers.

On the other hand, Apple provides a framework called Apple Multipeer Connectivity (hereinafter abbreviated as Multipeer Connectivity), which is a mechanism that enables the discovery of services provided by nearby terminal devices via Wi-Fi networks and Bluetooth personal area networks, and enables message-based data communication and file transfers through those services. Using this, it is possible to transmit performance position information of a musical score on a specific terminal device (hereinafter referred to as the leader terminal device) that is playing as the leader of an ensemble, using the technology disclosed in Patent Document 2, to at least one other terminal device (hereinafter referred to as the member terminal device) via data communication.

On the other hand, member terminal devices receive performance position information of the sheet music from the reader terminal device via a communication means utilizing the Multipeer Connectivity framework. Based on the received performance position information, they can play the sheet music stored within their own terminal device. Therefore, this method enables ensemble playing between the performance part and the automated performance part without using the performance control unit described in Patent Document 3. Furthermore, the performance position data used for data communication over this network is a simple data that can be represented by a single value, as described later, and does not require complex data such as phrases or audio information as shown in Patent Documents 5, 6, and 7.

Furthermore, since the data used for communication via this network is only one-way information from the reader terminal device to the member terminal devices, the delay caused by network communication is only in one direction. Because the member terminal devices store their own musical scores, by adjusting the performance position of the score played by the member terminal device to be ahead of the performance position of the score received by the reader terminal device by the amount of the communication delay, it becomes possible to compensate for the performance delay caused by network communication without using techniques such as those described in Patent Documents 8 and 9.

Furthermore, in cases where the reader terminal device outputs sound using, for example, an external speaker via Bluetooth connection, the delay in data transfer from the reader terminal device to the external speaker may be longer than the delay in network communication due to the performance. However, even in such cases, the present invention can address this by setting a slower reading position for the musical score played by the member terminal device.

In this way, it became possible to automatically synchronize the timing and speed of each member terminal's performance with the leader terminal's performance. However, the objective of allowing member terminal players to easily manually synchronize their performance with the leader terminal's performance has not yet been achieved.

Furthermore, the technologies described in Patent Documents 10 and 11 require means to predict movement states and means to notify users of operation information input from other devices using information other than visual/auditory signals, in order to synchronize timing when performing real-time ensemble playing using network functions. This has the drawback of complicating the device, and a simpler method of implementation is needed.

Furthermore, while Patent Document 12 discloses a technique for displaying a standard musical score using multiple layers and overlaying them, even if this technique is used to overlay the leader terminal's performance onto a member terminal's performance, as already mentioned, the only things displayed on the mobile terminal's display, which is the subject of this invention, are thin lines of varying positions and lengths moving from top to bottom of the screen. Since the leader's score and the members' scores are usually different, this technique was insufficient for members to visually recognize whether their timing and speed were in sync with or out of sync with the leader's performance.

Japanese Patent Application No. 2020-210850 Japanese Patent Application No. 2021-096161 Special Publication No. 2016-521381 Japanese Patent Application Publication No. 6-35456 Japanese Patent Publication No. 2005-258267 Patent No. 3277875 Japanese Patent Publication No. 2001-356764 Japanese Patent Publication No. 2005-195982 Japanese Patent Publication No. 2007-178860 Japanese Patent Publication No. 2006-162904 Japanese Patent Publication No. 2005-250148 Japanese Patent Publication No. 2009-230007

The problem that the present invention aims to solve is to provide a method and apparatus that enables multiple portable terminal devices or the like to easily synchronize their performance with the performance of a device set as the leader of the ensemble when performing together using a music performance device that includes a music score input means for reading electronically represented music score data, an event detection means for detecting performance events that occur in conjunction with a performer's performance, and a performance position determination means for determining the performance position of the music score data from the events detected by the event detection means.

The present invention is characterized by comprising: a first display means that, in one device set as the leader of the ensemble by the leader setting means, transmits performance position data determined by the performance position determination means via a communication means provided in the same device; in at least one other device set as a non-leader by the leader setting means, uses the performance position data received by the communication means as the display position of reference performance score data by the display position determination means, and continuously displays the reference performance score data as a score image; and a second display means that displays the score image displayed in accordance with the performance of the performer in the non-leader device, superimposed on the reference performance score image so that it is visible, continuously and independently of the first display means.

According to the present invention, the first display means continuously displays sheet music data images that the performer should play based on performance position data transmitted from the reader terminal device, allowing the displayed images to be used as a reference for the performance. The second display means continuously overlays sheet music images of the performer's own performance using the same sheet music data onto the reference sheet music images, making it possible to visually recognize how far ahead or behind the performer's own performance is compared to the reference performance.

By recognizing how far ahead or behind the reference performance is, and by continuously adjusting their own playing speed based on this recognition, it becomes possible to synchronize the timing and speed of the member terminals' performances with the leader terminal's performance during ensemble playing.

Figure 1 is an explanatory diagram showing the state when acting as the leader of the ensemble in the first implementation method. (Example 1) Figure 2 is an explanatory diagram showing the state when acting as a member of an ensemble in the first implementation method. (Example 1) Figure 3 is an explanatory diagram showing the hardware and software configuration in Example 1. Figure 4 is a flowchart showing the operation of the first software in Example 1. Figure 5 is a flowchart showing the operation of the second software in Example 1. Figure 6 is a flowchart showing the operation of the third software in Example 1. Figure 7 is an explanatory diagram illustrating a typical musical score. Figure 8 is an explanatory diagram illustrating the digital musical score image in Example 1. Figure 9 is the first explanatory diagram illustrating the overlay of digital musical score images in Example 1. Figure 10 is a second explanatory diagram illustrating the overlay of digital musical score images in Example 1. Figure 11 is a third explanatory diagram illustrating the overlay of digital musical score images in Example 1. Figure 12 is a flowchart showing the operation of the second software in Example 2.

The objective of providing a method and apparatus for performing an ensemble using multiple musical performance devices, each equipped with a musical score input means for reading electronically represented musical score data, an event detection means for detecting performance events that occur during performance, and a performance position determination means for determining the performance position of the musical score data from the events detected by the event detection means, is to enable the timing and speed of performances by devices set as members of an ensemble to easily match the performance of a device set as the leader. This is achieved by providing a first display means that continuously displays the reference musical score data as a musical score image, and a second display means that continuously displays the musical score data displayed in accordance with the performance of the performer in the device set as the leader, overlaid on the reference musical score image displayed by the first display means, so that it is visible.

Figure 1 is an explanatory diagram showing the state when the device of the present invention is operating as the leader of an ensemble in the first method of implementation, where 100 is the smartphone body and 101 is the image output unit using a liquid crystal panel. The liquid crystal panel is a touch panel and also serves as an information input unit 102 for user operation. The area indicated by the dashed frame 1011 is a musical score image display unit that is read appropriately from the first musical score input means in the smartphone body and displays the movement of the musical score in real time according to the progress of the performance. The dashed line 1020 is a virtual timing line indicating the performance position, and it indicates that a sound is produced when the note is on that line. The black lines indicated by 2005, 2006, and 2007 (there are other similar black lines, but they are omitted for explanation) are musical notes in the musical score image. 2010 and 2011 are marker lines recorded at both ends of the musical score image for detecting the position of the notes displayed on the panel 101. In this embodiment, these marker lines are used in the image processing means described later, but they are not essential for producing sound from notes using digitally recorded sheet music. The area indicated by the dashed frame 1012 is the keyboard display section. When there is a note at the playing position, the white and black keys are displayed in gray to indicate that the corresponding key has been pressed. When the note that was at the playing position moves away over time, the white and black keys are displayed in their original colors to indicate that the corresponding key has been released. 300 is a single finger used for gesture operation on the liquid crystal panel 101, and 301 is an arrow that simulates the rotational gesture made by that finger 300. When finger 300 is rotated in the direction of arrow 301, the notes indicated by the black lines 2005, 2006, and 2007 move in the direction of arrow 302 according to the rotation speed. When a note lies on the virtual timing line 1020, which represents the performance position, it produces a sound. When a note moves off the timing line 1020, the sound stops, thus achieving performance. 3000 schematically represents the communication between this device and other devices that are members of the ensemble via data communication. In this embodiment, it transmits performance position information corresponding to the performance progress via gesture control.

Figure 2 is an explanatory diagram showing the state of the device of the present invention when it is operating as a member of an ensemble in the first method of implementation. The device itself is the same as in Figure 1, so the physical configuration is the same, but because the operation is different, the image displayed on the score image display unit 1011, indicated by the dashed frame on the liquid crystal panel 101, is different from that in Figure 1. Also, 3000 schematically represents that this device is communicating with the device acting as the leader of the ensemble via a data communication means, similar to Figure 1, and in this case, it is receiving performance position information transmitted by the other device acting as the leader of the ensemble. The black lines shown in 2001, 2002, and 2003 are notes on a reference score image that are appropriately read and displayed by the second score input means based on performance position information received from another device acting as the leader of the ensemble. The white lines shown in 2001-2, 2002-2, and 2003-2 are notes on a score image played by a performer that are appropriately read and displayed by the first score input means based on a rotational gesture operation by finger 300. These are displayed by inverting the original score image to black and white and setting its transparency to 75%, overlaying it on top of the reference score image. Both the black and white lines move in the direction of arrow 302 as the performance progresses. However, the events that trigger their movement are independent: the former is performance position information received via data communication from a device acting as the leader of the ensemble, while the latter is an event based on the performer's own gestures. Therefore, when a note represented by the white line lies on the virtual timing line 1020 (the performance position), the sound is produced; when the note moves off the timing line 1020, the sound is silenced, thus realizing the performer's participation in the ensemble.

Figure 3 is a configuration diagram showing the hardware and software of an embodiment of the present invention. 1 is the CPU (Central Processing Unit), which realizes the present invention by performing various processes in various means according to a program stored in non-volatile memory (not shown). 2 is a leader setting means for setting whether the present invention's device operates as a leader in an ensemble or as a member (non-leader) of an ensemble. This setting is configured from the device's various operation setting screens (not shown), and the setting is stored in non-volatile memory (not shown). 3 is a gesture event detection means (abbreviated as gesture detection means) which is a performance event detection means in this embodiment. It recognizes a rotational gesture made by a finger 300 on the touch panel 102 and detects the position coordinates (x, y) of the finger 300 on the touch panel 102 at regular time intervals. From the detected position coordinates (x, y) of the finger 300, the CPU 1 detects the rotational speed using a rotational speed detection means 4 configured by a software program. The method for detecting the rotational speed is disclosed in Patent Document 2 and is therefore omitted here.

Similarly, 5 is a performance position determination means configured by a software program, and similarly, it uses the method disclosed in Patent Document 2 to determine the performance position of the musical score from the rotation speed detected by the CPU 1. 6 is a first musical score input means for reading musical score image data from the musical score data electronically stored in a musical score storage means (not shown) within the device body, which is composed of non-volatile memory, etc., by trimming it to the required size (details will be described later) based on the performance position determined by the performance position determination means 5, and then displaying it to the image processing means 7, the first display means 12, or the second display means 13.

7 is an image processing means that identifies the position of a note at the performance timing (corresponding to the timing line 1020) from the musical score image data read by the first musical score input means 6 using the method shown in Patent Documents 1 and 2, and detects whether or not a note is present at the identified position. 11 is a sound output means that, based on the information of whether or not a note is present at the specific performance timing position detected by the image processing means 7 and its history, has the function of playing or stopping the sound of the corresponding note with a preset timbre and volume, according to the CPU 1's decision to play or stop the sound of the corresponding note.

8 is a data communication means that transmits and receives data via a network between multiple devices having the same configuration as the present invention. For example, when this device is set as the leader of the ensemble by the leader setting means 2, the data communication means 8 transmits the performance position determined by the performance position determination means 5 as performance position data to the devices set as members of the ensemble sequentially. On the other hand, when this device is set as a member of the ensemble by the leader setting means 2, it receives the performance position data transmitted from the device set as the leader and uses that data as input to the display position determination means 9, which will be described later.

9 is a display position determination means. When the device is set as a member of an ensemble, it is used to display the sheet music data electronically stored in the sheet music storage means within the device on the LCD panel 101 within the device, based on the performance position data of the ensemble leader received by the data communication means 8. The received performance position data of the ensemble leader directly becomes the display position of the sheet music data. 10, similar to the first sheet music input means 6, is a second sheet music input means for reading image data from the sheet music data electronically stored in the sheet music storage means within the device, after trimming it to the required size for the first display means 12 (described later) based on the display position of the sheet music determined by the display position determination means 9.

12 is the first display means. Based on the settings in the leader setting means 2, if this device is set as the leader of the ensemble, it displays the musical score data image read by the first musical score input means 6 directly on the musical score image display unit 1011 on the liquid crystal panel 101. If this device is set as a member of the ensemble, it displays the musical score data image read by the second musical score input means 10 directly on the musical score image display unit 1011 on the liquid crystal panel 101.

13 is a second display means, operating independently of the operation of the first display means 12. Only when this device is set as a member of the ensemble based on the settings in the leader setting means 2, it overlays the sheet music data image read by the first sheet music input means 10 onto the sheet music image display unit 1011 on the liquid crystal panel 101, so that the image already displayed by the first display means 12 is visible.

Figure 4 is a flowchart showing the operation of the first software in this embodiment of the present invention, illustrating the operation when the device is set as the leader in an ensemble performance by the leader setting means 2 in this embodiment.

First, in step S1, the CPU 1 waits for a gesture event to be detected by the gesture detection means 3. In this embodiment, when a gesture operation by finger 300 begins, an event occurs at intervals of 1/60th of a second. The position where finger 300 first touches the panel is set as the origin O: (x, y) = (0, 0), and thereafter, the relative coordinates from the origin are detected each time until finger 300 leaves the panel. Once a gesture is detected, the process proceeds to step S2, where the rotation speed detection means 4 determines the rotation speed caused by the gesture.

The method for determining the rotation speed by the velocity detection means 4 from the position coordinates of finger 300 on the panel via a gesture event is disclosed in Patent Document 2, so it will be omitted here.

Once the rotation speed detection means 4 determines the rotation speed resulting from the gesture operation by the finger 300, the CPU 1 determines the performance position using the performance position determination means 5 in step S3.

Here, we will explain the musical score image, which is a digital representation of a musical score. Figures 5 and 6 illustrate how a typical musical score is represented and played as a digital musical score image according to the present invention. Figure 5 shows a typical musical score, and the notes 2004, 2005, 2006, 2007, and 2008 in the score are quarter notes of different pitches. The symbol 400 is a tempo marking, which indicates that the score should be played at a speed of 80 quarter notes per minute. Figure 6 is the beginning of the musical score image obtained by converting the musical score shown in Figure 5 into a digital image. Its coordinates are with the positive x-axis on the right and the positive y-axis on the top, and the notes contained in the ellipse shown at 401 in Figure 5 are shown as 2004, 2005, 2006, 2007, and 2008. 2010 and 2011 are marker lines recorded at both ends of the digital musical score image for note position detection. In this embodiment, the width of the musical score image is 384 pixels, the width of each note is 2 pixels, and the gap between adjacent notes is 1 pixel. Since 85 notes (= 7 octaves [1 octave includes 12 notes including semitones] + 1 note) can be played, the space between marker lines 2010 and 2011 is 256 pixels = (2 + 1) × 85 + 1 (the + 1 represents the gap between the marker lines). In this embodiment, marker lines 2010 and 2011 are placed on the musical score image to detect the position of the notes. However, if the position of a given note on the x-axis is predetermined, these marker lines 2010 and 2011 are not essential. Note that Figure 6 only shows the beginning of the musical score shown in Figure 5; however, the actual musical score data is electronically stored in the musical score storage means within the device.

In Figure 6, quarter note 2004 is located three octaves above the lowest note in this embodiment, corresponding to the 36th note (12 notes x 3 octaves) from the lowest note. Therefore, there is a gap of 109 pixels = (2 + 1) x 36 + 1 between it and marker line 2010 in Figure 6. Similarly, there are gaps of 115, 121, and 124 pixels between quarter notes 2005, 2006, and 2007 and marker line 2010, respectively.

Furthermore, in Figure 6, the lengths in the y-axis direction of quarter notes 2004, 2005, 2006, 2007, and 2008 correspond to the length of a quarter note. In this embodiment, the distance from the beginning of quarter note 2004 to the beginning of quarter note 2005 is 180 pixels. Similarly, the distances between 2005 and 2006, 2006 and 2007, and 2007 and 2008 are also 180 pixels. Although not shown here, the length of an eighth note would be half that, 90 pixels; a sixteenth note would be 45 pixels; and a half note would be 360 pixels.

In Figure 5, the performance of the musical score progresses from left to right, as indicated by the arrow 500, following the score. On the other hand, in the case of the musical score image, it progresses from bottom to top, as indicated by the arrow 500 in Figure 6. The dashed line 501 in Figures 5 and 6 indicates the performance position, and by advancing this dashed line in the direction of the arrow 500 at regular time intervals, the performance on the musical score and musical score image will progress. Therefore, the distance that the dashed line 501 advances is determined according to the rotation speed at the timing when a gesture event occurs (at intervals of 1/60th of a second in this embodiment). For example, in this embodiment, the tempo marking 400 in Figure 5 indicates playing at a speed of counting 80 quarter notes per minute, so on the digital musical score image in Figure 6, it should advance 14,400 pixels (= 180 pixels × 80 times) per minute. Therefore, since gesture events occur at intervals of 1/60th of a second, following the instruction of speed symbol 400, we can see that we only need to move 4 pixels (= 14400 pixels / 60 seconds / 60 times) for each gesture event. On the other hand, the rotation speed V obtained in step S2 has been measured and found to have an approximate average value of 15.0, so if we find the constant C such that the number of pixels is 4 when the speed V is 15.0, we get 4 pixels = 15.0 × C, so C = 4 / 15.0 = 0.2667.
Therefore, if N is the number of pixels that move when the rotation speed V,
This is how it works [ROUND() represents rounding to the nearest decimal place]. Therefore, the number of pixels N, which is the distance the dashed line travels at speed V, can be determined. This number of pixels N can then be added to the performance position at the time of the previous gesture event to obtain the new performance position. In this manner, the CPU 1 determines the performance position using the performance position determination means 5 in step S3.

After determining the performance position in step S3, the process proceeds to step S4, where image data is trimmed and read from the first score input means 6 based on the performance position determined in step S3. Here, "trimming and reading" means partially reading the score image based on the performance position so that it can be displayed on the score display unit 1011 on the LCD panel 101. For example, if the performance position is indicated by the dashed line 501 in Figure 6, then only the portion enclosed by the dashed line 502 (384 pixels vertically and 384 pixels horizontally), with the dashed line placed vertically in the center, is read.

In step S4, the sheet music data read out as trimmed image data is processed by the image processing means 7 in the next step S5 to determine the position of each of the 85 notes at the performance position (on the dashed line 501 in Figure 6) determined in step S3, and to detect the presence or absence of notes at each position. The details of the image processing performed by the image processing means 7 in this embodiment to determine the position of each note from the sheet music data and to detect the presence or absence of notes at each determined position are as described in Patent Document 1.

Based on the information detected in step S5 regarding the presence or absence of musical notes at each position, in step S6, the CPU 1 compares this information with the information stored in the memory means (not shown) regarding the presence or absence of musical notes at each position during the previous gesture event to determine whether new musical notes have appeared at each position or whether previously appearing musical notes have disappeared. Based on this determination, the CPU 1 instructs the sound output means 11 to play or stop the sound of the musical note corresponding to each position at a predetermined timbre and volume.

Once the instructions to the sound output means 11 are complete, the process proceeds to step S7. The CPU 1 passes the musical score data read in step S4 to the first display means 12, which then displays the received musical score data image directly on the musical score image display unit 1011 on the liquid crystal panel 101.

Finally, in step S8, the CPU 1 transmits the performance position determined in step S3 as performance position data to multiple other devices having the same configuration as the present invention, which are connected via the data communication means 8 and configured as members of the ensemble.

With the above steps complete, the series of processes triggered by the gesture event are finished. The process returns to S1 and waits for the gesture detection means 7 to detect another gesture event.

In step S8, performance position data is transmitted via the data communication means 8, but volume and other information may be added and transmitted simultaneously. Furthermore, in step S1, the following steps proceed triggered by a gesture event from the gesture detection means 3, but for example, an event triggered by a timer occurring at 1/60 second intervals may be used, and the following steps may proceed with the rotation speed determined in step S2 remaining constant. In this way, the leader's performance in an ensemble can be automated.

Figure 7 is a flowchart showing the operation of the second software in the same embodiment of the present invention, illustrating the operation when the device is set as a member in an ensemble by the leader setting means 2 in this embodiment, and data is received by the data communication means 8.

First, in step S9, CPU 1 waits for data to be received from data communication means 8. If data is received by data communication means 8, the process proceeds to step S10, where display position determination means 9 determines the display position of the musical score data from the received data. At this time, the data received from communication means 8 is performance position data from another device set as the leader in the ensemble. Therefore, this value is used as the reference display position for its own musical score data. A correction value is added or subtracted to account for delays in inter-device communication within the network, and this value is determined as the display position of the musical score, i.e., the performance position that serves as the reference for its own performance.

Once the display position of the musical score data is determined in step S10, the process proceeds to step S11, where the image data is trimmed and read from the second musical score input means 10 based on the display position determined in step S10. This reading method is similar to the method described in step S4 for trimming and reading image data from the first musical score input means 6. While step S4 involved trimming and reading image data based on the performance position determined in step S3, step S11 involves trimming and reading the musical score data based on the display position determined in step S10.

The process then proceeds to step S12. The CPU 1 passes the musical score data read in step S11 to the first display means 12. The first display means 12 then displays the received musical score data image directly on the musical score image display unit 1011 on the liquid crystal panel 101.

As described above, the series of processes that occur when this device is set as a member in an ensemble and receives data via the data communication means 8 are completed. Therefore, the process returns to S9 and waits for the data communication means 8 to receive data again.

Figure 8 is a flowchart showing the operation of the third software in this embodiment of the present invention. This software operates within the same device, in parallel with the operation of the second software shown in Figure 7, when the device is set as a member in an ensemble by the leader setting means 2 in this embodiment. It illustrates the operation when a gesture event is detected by the gesture detection means 3.

First, in step S13, CPU 1 waits for a gesture event to be detected by the gesture detection means 3. If a gesture is detected, the process proceeds to step S14, where the rotation speed detection means 4 determines the rotation speed caused by the gesture.

Once the rotation speed detection means 4 determines the rotation speed resulting from the gesture operation by the finger 300, the CPU 1 determines the performance position using the performance position determination means 5 in step S15.

After determining the performance position in step S15, the process proceeds to step S16, where image data is trimmed and read from the first score input means 6 based on the performance position determined in step S15.

In step S16, the sheet music data read out as trimmed image data is processed by the image processing means 7 in the next step S17 to determine the position of each of the 85 notes at the performance position determined in step S15, and to detect the presence or absence of notes at each position.

Based on the information detected in step S17 regarding the presence or absence of musical notes at each position, in step S18, the CPU 1 compares this information with the information stored in the memory means regarding the presence or absence of musical notes at each position during the previous gesture event to determine whether new musical notes have appeared at each position or whether previously appearing musical notes have disappeared. Based on this determination, the CPU 1 instructs the sound output means 11 to play or stop the sound of the musical note corresponding to each position at a predetermined timbre and volume. These operations from steps S13 to S18 are identical to the operations from steps S1 to S6 of the first software operation shown in Figure 4.

If the instruction to the sound output means 11 is completed in step S18, the process proceeds to step S19, where the musical score data read out as trimmed image data in step S16 is passed to the second display means 13. Here, the second display means 13, independently of the first display means 12, overlays and displays the passed musical score data image so that the musical score data image already displayed on the musical score image display unit 1011 on the liquid crystal panel 101 by the first display means is visible.

This section explains the overlay display of musical score data images. Figure 9 shows how the musical score image data read in step S11 and the musical score image data read in step S16 are overlaid. In the figure, 2100 represents the musical score image data read in step S11, 2101 represents the musical score image data read in step S16 with the colors inverted and the transparency set to 75%, and 2102 represents image data with a virtual timing line 1020 indicating the performance position drawn on a transparent background. These are overlaid from bottom to top in the order of 2100 → 2101 → 2102, as shown in Figures 10 and 11.

In Figure 10, the black lines indicated by 2001, 2002, 2003, and 2005 represent musical notes in the score image data read in step S11. These are reference notes for performance, displayed based on the performance position of the reader terminal device received by the data communication means. On the other hand, the white lines indicated by 2001-2, 2002-2, 2003-2, and 2005-2 represent musical notes in the score image data read in step S16 in accordance with the performer's gesture operation. Lines 2001, 2002, 2003, and 2005 advance independently in the direction of the arrows indicated by 302 in the figure, as the performance of the reader terminal device progresses, while lines 2001-2, 2002-2, 2003-2, and 2005-2 advance independently in the direction of the arrows indicated by 302 in the figure, as the performance of the performer themselves progresses on the member terminal device. Incidentally, since notes 2001, 2002, 2003, and 2005, as well as notes 2001-2, 2002-2, 2003-2, and 2005-2, were originally read from the same musical score data, if the timing and speed of the performance on the reader terminal device and the performer's own performance on the member terminal device match, they should overlap and progress in the same place. However, in Figure 10, the white notes are behind the black notes in the direction of progression, indicating that the notes 2001-2, 2002-2, 2003-2, and 2005-2, which display the performer's performance, are behind the reference notes 2001, 2002, 2003, and 2005. Similarly, in Figure 11, the white notes are ahead of the black notes in the direction of progression, indicating that the notes 2001-2, 2002-2, 2003-2, and 2005-2, which represent the performer's playing, are ahead of the reference notes 2001, 2002, 2003, and 2005. By overlaying the score image data representing the performer's playing with the score image data representing the reference playing and displaying them sequentially, it becomes possible to visually recognize whether the performer's playing is ahead or behind the reference playing.

As a result of the above, the series of processes triggered by the gesture event generated by the third software is completed, so we return to step S13 and wait for the gesture detection means 7 to detect another gesture event.

Furthermore, when the performance position is determined by the performance position determination means 5 in step S15, the gesture detection means 3 uses the position where the finger 300 first touches the panel as the origin, and detects events at intervals of 1/60 seconds until the finger 300 leaves the panel. For the first and last events, it can also detect information indicating the start and end of the event. Therefore, in determining the performance position in step S15 of the processing flow where a gesture start event occurs, instead of simply setting the performance position to 0 pixels, copying the display position of the reference image determined by the display position determination means 9 immediately before that point and using that as the initial value for the performance position, and then using this value as the basis for determining the performance start position in step S15 for each gesture event detection, makes it easier to synchronize the timing of the performer's performance.

As described above, this embodiment provides a first display means for displaying a reference performance score image, and a second display means for overlaying the performer's performance score image onto the reference performance score video image. By independently displaying both images in accordance with the progress of the performance, it becomes possible to visually recognize whether the performer's own performance is ahead or behind the reference performance. Furthermore, by adjusting the rotation speed of the performer's own performance image using gesture control to match the reference performance image, it becomes possible to easily realize ensemble playing with multiple devices, which was previously difficult.

In this embodiment, the gesture was described as a single-point rotation gesture using a finger 300 on a touch panel. However, any single-point gesture using a stylus or similar device is acceptable, as long as the gesture can be recognized.

Furthermore, in this embodiment, the method employed is to detect musical notes from a digital musical score image by image processing using the image processing means 7. However, the musical score data can be digital data such as MIDI data, rather than a digital image. For example, in the case of MIDI data, the data is stored in groups called tracks in the form of delta time and event repetitions. Delta time is the time determined by the length of a quarter note and the time unit specified separately, and represents the elapsed time since the previously specified delta time. Events specify the pitch, volume, and on/off status of the note at that time. Therefore, if the performance position is represented by time rather than the number of pixels, and the MIDI data is converted into a digital musical score image as shown in this invention and displayed, the exact same functionality can be achieved. Furthermore, in this embodiment, performance position data transmitted from the device acting as the leader of the ensemble using the data communication means 8 was adopted as a reference. However, by generating periodic events using a timer within the devices acting as members of the ensemble, thereby replacing step S9 in the flowchart showing the operation of the second software in Figure 7, and adding processing equivalent to steps S17 and S18 in the flowchart showing the operation of the third software in Figure 8 after step S12, an ensemble performance by a single performer within the same device can also be achieved (69).

Figure 12 shows a second embodiment of the present invention, adding steps S20, S21, and S22 between steps S11 and S12 of the flowchart showing the operation of the second software in Figure 7 of Embodiment 1. Since the gesture detection means 3 can identify the start and end events of a gesture, in step S20 it determines whether the device is performing a gesture operation. If it is performing a gesture, it proceeds to step S17, as in Figure 7; otherwise, it proceeds to steps S21 and S21. Steps S21 and S21 are the same processes as steps S17 and S18 of the flowchart showing the operation of the third software in Figure 8. By adding this step, the device set as a member of the ensemble by the leader setting means 3 will operate as an automatic playing device that automatically plays according to the ensemble leader's performance when the performer is not performing a gesture operation. In this case, if the data received via the data communication means 8 includes data other than the performance position data, it can be used; for example, if volume information is included, it can be reflected in step S22.

By detecting rotational speed using electrical signals instead of gestures to determine the performance position, and then playing the musical score based on the determined position, this system can be applied not only to mobile devices but also to personal computer-based devices and standalone music performance devices. Furthermore, the operation of visually matching the video image of the musical score can also be used as a game.

1. CPU (Central Processing Unit)
2 Leader setting means 3 Gesture detection means 4 Rotation speed determination means 5 Performance position determination means 6 First score input means 7 Image processing means 8 Data communication means 9 Display position determination means 10 Second score input means 11 Sound output means 12 First display means 13 Second display means

Claims (2)

A music performance device comprising a music score input means for reading electronically represented music score data, an event detection means for detecting performance events, and a performance position determination means for determining the performance position of the music score data from the events detected by the event detection means, wherein the device further comprises a first display means for continuously displaying the music score data of a reference performance as a music score image, and a second display means for continuously displaying the same music score data, which is displayed in accordance with the performer's performance , superimposed on the music score image of the reference performance so that it is visible, and independently of the first display means, wherein, as the reference performance, one device set as the leader of the ensemble by a leader setting means transmits the performance position data determined by the performance position determination means via a communication means provided in that device, and at least one other device set as a non-leader by the leader setting means uses the performance position data received by the communication means as the display position of the music score data of the reference performance by a display position determination means . The music performance device according to claim 1, characterized in that the event detection means is a gesture detection means that recognizes rotational operations on a touch panel as gestures.