JP4529489B2 - Image generating apparatus and musical instrument - Google Patents

Description

  The present invention relates to a technique for transmitting a state of a musical instrument performance operation.

  In the learning of keyboard instruments, not only the musical sounds output by the instrument, but also the fingering represented by fingering and touch are important. For this reason, the teacher visually checks the keyboard and student's hands to confirm whether or not an appropriate key is operated with an appropriate finger. However, since direct visual inspection is not possible with distance learning using a network, it is necessary to provide teachers with an alternative confirmation method.

As one of such confirmation methods, the student photographs the keyboard with the camera, transmits the captured image data via the network, and receives and displays the transmitted image data on the teacher side. A method is conceivable. According to this method, the teacher can check the shape of the student's hand by looking at the displayed live-action image. In general, since the teacher can estimate the finger usage from the shape of the hand, according to this method, the teacher can confirm whether or not the finger usage is appropriate.
However, in reality, since the data size and display size of image data are limited, it is difficult for the teacher to specify the key operated by the student. In addition, the key operated by the student in the live-action image may be hidden in the student's hand. That is, even if this method is adopted, it is difficult to confirm whether or not an appropriate key is operated with an appropriate finger.

As another confirmation method, the method currently disclosed by patent documents 1-3 is mentioned.
Patent Document 1 discloses an apparatus that uses performance information representing operation contents such as key-on / key-off of a teacher for student key-press guidance. This device determines whether or not the student's performance is correct by comparing with the performance of the teacher. In this determination, performance information is first compared, then finger usage information representing finger usage is compared, and then key depression location information representing a key depression location is compared. The finger usage information and the key-pressed part information can be obtained by performing image recognition processing on the photographed image.
By applying this method, the teacher can confirm whether or not an appropriate key is operated. However, since the information necessary for teachers to identify the fingering of students is diverse and can vary from teacher to teacher, it is difficult to provide a device that extracts these information from image data without omission. .

Patent Document 2 discloses an apparatus that displays an image representing a musical instrument performance on an HMD (Head Mounted Display) worn by the performer. In this display, a background image layer such as a spectator, a musical instrument image layer, and a fingering image layer determined from the switch state of the musical instrument are overlaid. The musical instrument image is a pseudo image obtained by rotating and moving a prestored three-dimensional image and then projecting it onto a plane, and the direction and amount of rotation and movement are determined using a real image of the instrument.
In this apparatus, the photographed image is only used to determine the direction and amount of rotation / movement of the three-dimensional image, and is not displayed. That is, the information necessary for the teacher to specify the fingering of the student is not provided.

Patent Document 3 discloses a device that guides a performance by emitting a key according to music data. This device detects the coordinates of a light-emitting key in the field of view of a player wearing a goggle-type liquid crystal display with a transparent background, and uses the detected coordinates to express a fingering image represented by finger data in song data Controls the liquid crystal display so that it overlaps the light emission key in the field of view.
However, this method is based on the premise that the keyboard being operated is in a position where it can be directly seen, and is not applicable to an environment where direct viewing is impossible.
JP 2002-182553 A JP 2000-293172 A JP 2000-352773 A

  The present invention has been made in view of the above-described circumstances, and it is possible to confirm whether or not an appropriate instrument operation unit is operated with an appropriate finger even in an environment where direct visual inspection is impossible. The purpose is to provide technology that can be used.

The present invention includes a musical sound input means for inputting a musical sound signal, and an operation section specifying means for specifying an operation section to be operated on the musical instrument when the musical instrument emits a musical sound represented by the musical sound signal input by the musical sound input means. An image input means for inputting an image signal representing a photographed image, and a position for specifying the position of the operation unit of the instrument in the photographed image represented by the image signal using the image signal input by the image input means A live action image represented by an image signal input by the image input unit using the specifying unit and the operation unit specified by the operation unit specifying unit using the position specified by the position specifying unit with respect to the operation unit. An image generation apparatus having an image generation means for generating an image signal shown in an enhanced manner is provided.
The image generation apparatus is configured to display, in a live-action image represented by an input image signal, an operation unit to be operated on the musical instrument when the musical instrument emits a musical sound represented by the input musical sound signal. An image signal to be emphasized using the position of the operation unit of the musical instrument in the image is generated.

In the musical instrument having an operation unit, the present invention is input by an operation unit specifying unit that specifies the operation unit being operated, an image input unit that inputs an image signal representing a photographed image, and the image input unit. The position specifying means for specifying the position of the photographed image of the operation section in the photographed image represented by the image signal and the operation section specified by the operation section specifying means are specified by the position specifying means for the operation section. A musical instrument having an image generation unit that generates an image signal that is emphasized in a real image represented by the image signal input by the image input unit using the position.
This musical instrument is shown in the live-action image represented by the input image signal, with the operation unit being operated being emphasized using the position of the operation unit of the musical instrument in the live-action image represented by the image signal. An image signal is generated.

According to the present invention, it is possible to obtain an image signal that shows an operation portion of a musical instrument to be operated or operated in an enhanced manner in a photographed image. If the photographed image is a photographed image around the operation unit of the instrument, the shape of the hand operating the instrument, and the finger usage can be confirmed by visually recognizing the photographed image. Furthermore, in this live-action image, the operation unit specified using the musical tone signal is highlighted, so that the operated instrument operation unit can be confirmed. In addition, since the generated image signal can be used in a remote place, the present invention can cope with an environment where direct visual inspection is impossible.
Therefore, according to the present invention, by using the generated image signal, it is confirmed whether or not an appropriate instrument operation unit is operated with an appropriate finger even in an environment where direct visual observation is impossible. Can do.
Further, if the operation amount of the operation unit is specified and an image signal is generated by emphasizing the operation unit in a manner corresponding to the operation amount, direct visual inspection is not possible by using this image signal. Even in a possible environment, it is possible to confirm whether or not an appropriate instrument operating unit is appropriately operated with an appropriate finger.

Embodiments of the present invention will be described below with reference to the drawings.
[Constitution]
FIG. 1 is a block diagram showing the configuration of a remote learning system for performing remote learning of musical instrument performance between students and teachers.
As shown in this figure, a teacher side equipment 2 used for teachers is arranged on the teacher side of the network 1, and a student side equipment 3 used for students is arranged on the student side. The teacher-side equipment 2 and the student-side equipment 3 communicate with each other via the network 1.

First, the configuration of the teacher-side facility 2 will be described.
The keyboard instrument 21 includes a keyboard 211, which is played by a teacher and outputs musical sounds.
The microphone 22 collects the musical sound output from the keyboard instrument 21 and outputs audio data.
The camera 23 captures the periphery of the keyboard 211 and outputs image data in units of frames.
The terminal 24 receives the audio data from the microphone 22 and the image data from the camera 23 and transmits them to the communication partner terminal via the network 1. On the other hand, it receives image data and audio data from the partner terminal via the network 1, displays the received image data on the display 241, and outputs the received audio data to the speaker 25.

Next, the configuration of the student side equipment 3 will be described.
The keyboard instrument 31 is compliant with the MIDI (Musical Instruments Digital Interface) standard and includes an 88-key keyboard 311 that is played by students and outputs MIDI data and musical sounds. At a predetermined position near the keyboard 311, a reference image B is formed to indicate a reference position, a reference direction, and a reference length in image data obtained by photographing the keyboard 311 with the camera 33 (see FIG. 6). The reference image B has different feature values for each orientation so that the reference direction can be understood. The formation method of the reference image B is arbitrary. For example, it may be formed by printing, may be formed by placing paper on which the reference image B is printed on the keyboard instrument 31, or formed as a three-dimensional shape on the surface of the keyboard instrument 31 during the manufacturing process of the keyboard instrument 31. May be.

The image generation device 36 receives the image data from the camera 33 and the MIDI data from the keyboard instrument 31, and generates and outputs image data using these data. The configuration of the image generation device 36 will be described later.
The video mixer 37 receives the image data from the camera 33 and the image data from the image generation device 36, generates the synthesized image data by transparently synthesizing the latter image data with the former image data, and generates the synthesized image data. Is output.

  The functions of the microphone 32, the camera 33, the speaker 35, and the terminal 34 are the same as those of the same name in the teacher side equipment 2. However, the terminal 34 inputs not the image data from the camera 33 but the composite image data from the video mixer 37. That is, the image data displayed on the display 241 by the terminal 24 of the teacher-side equipment 2 is not captured by the camera 33 of the student-side equipment 3, but is composite image data output from the video mixer 37 of the student-side equipment 3. .

FIG. 2 is a diagram illustrating the configuration of the image generation device 36.
As shown in FIG. 3, the keyboard information table 361 includes, for each key of the keyboard 311, a key number for identifying the key, an operation state indicating whether or not the key is depressed, an operation amount indicating the keystroke strength, and a camera. The key coordinates in the image data photographed at 33 are stored. The key number is an integer from 1 to 128, the operation state is “ON” / “OFF”, and the operation amount is an integer from 0 to 127. In the initial state, the operation states in the keyboard information table 361 are all “off”, and the operation amounts are all zero.

  The key specifying unit 362 receives the MIDI data from the keyboard instrument 31 and updates the operation state and operation amount in the keyboard information table 361 according to the content of the note-on message in the input MIDI data. On the other hand, the operation state in the keyboard information table 361 is updated according to the content of the note-on message in the input MIDI data.

  The overlay image generation unit 363 receives the image data from the camera 33, specifies the reference position, reference direction, and reference length in the input image data, and updates the coordinates in the keyboard information table 361 using these. On the other hand, using the input image data and the keyboard information table 361, overlay image data in which an identification image of a color corresponding to the operation amount of the key is located at the coordinate of the key whose operation state is “ON” is generated and output. To do.

[Operation]
Next, the operation of the remote learning system configured as described above will be described.
First, the operation for the student to confirm the performance of the teacher will be described.
The microphone 22 of the teacher-side equipment 2 collects sound and outputs sound data, and the camera 23 images the periphery of the keyboard 211 of the keyboard instrument 21 played by the teacher and outputs image data. These data are input to the terminal 24 and transmitted to the terminal 34 via the network 1. The terminal 34 receives these data, outputs the audio data to the speaker 35, and displays the image data on the display 341. As a result, the musical sound output from the keyboard instrument 21 is emitted from the speaker 35, and a live-action image around the keyboard 211 is displayed on the display 341.

Next, an operation for the teacher to confirm the performance of the student will be described. However, description of operations similar to those described above will be omitted.
First, it is assumed that the student has simultaneously pressed the keys C, E, and G (key numbers 76, 80, and 83) of the keyboard 311 of the keyboard instrument 31. MIDI data is output from the keyboard musical instrument 31 by this keystroke. This MIDI data includes three note-on messages. Each note-on message stores a note number (96, 100, 103) and a velocity (40, 50, 60).

  The key specifying unit 362 inputs this MIDI data, and first determines whether or not a note-on message is included in this MIDI data (step SA1 in FIG. 4). Since this MIDI data includes a note-on message, the determination result is “YES”. Therefore, the key specifying unit 362 updates the keyboard information table 361 using the note-on message in the MIDI data (step SA2). In this update, as shown in FIG. 5, the operation state of the key (C, E, G key) of the key number (76, 80, 83) corresponding to the note number in the above three note-on messages is “ON”. On the other hand, the velocity values (40, 50, 60) in the three note-on messages are set as the operation amounts of these three keys.

  Next, the key specifying unit 362 determines whether or not a note-off message is included in the MIDI data (step SA3). Since this MIDI data does not include a note-off message, the determination result is “NO”. Therefore, the key specifying unit 362 ends the process for this MIDI data.

  On the other hand, the camera 33 shoots the periphery of the keyboard 311 of the keyboard instrument 31 and outputs image data, and also outputs the image data immediately after the key is pressed. This image data is input to the overlay image generation unit 363 and the video mixer 37. An image when this image data is displayed is shown in FIG. This image is a real image around the keyboard 311 and includes the reference image B. In this live-action image, the student's hand overlaps the keyboard 311.

  When the image data is input, the overlay image generation unit 363 first specifies the reference position PS, the reference direction DR, and the reference length (L) in the image data (see step SB1 in FIG. 7 and FIG. 8). In this specification, the position PS of the reference image B in the image data is set as the reference position, and the direction in which the feature amount of the reference image B becomes a predetermined feature amount in the image data is set as the reference direction DR. The length of the predetermined section in the image B is set as the reference length (L). The reference direction DR is a direction parallel to the longitudinal direction of the key closest to the reference image B in this image data, and passes through the center of the key when the reference position PS is a base point. The reference length (L) matches the length (semitone width) obtained by dividing the width of the keyboard 311 in the image data by the number of keys (88).

  Next, the overlay image generation unit 363 calculates the coordinates of each key of the keyboard 311 in the image data using the reference position PS, the reference direction DR, and the reference length (L) (step SB2). The principle of this calculation will be described with reference to FIG. In this figure, the key in the reference direction DR when the reference position PS is the base point is the key K1. In this case, the coordinates of the key K1 are positions advanced by a predetermined number (k) times the reference length (L) in the reference direction DR from the reference position PS. Further, the coordinates of the key K2 that is five adjacent to the key K1 are positions advanced by a distance five times the reference length (L) in the direction orthogonal to the reference direction DR from the coordinates of the key K1.

  Next, the overlay image generation unit 363 updates the keyboard information table 361 using the calculated coordinates (step SB3). By this update, the coordinate value stored in the keyboard information table 361 in association with each key of the keyboard 311 is changed to the latest value.

  Next, the overlay image generation unit 363 refers to the keyboard information table 361 and determines whether there is a key whose operation state is “ON” (step SB4). At this time, the operation state of the C, E, and G keys (key numbers 76, 80, and 83) is “ON”, so the determination result is “YES”. Therefore, the overlay image generation unit 363 reads the coordinates of the C, E, and G keys and the operation amount from the keyboard information table 361 (step SB5).

  Next, the overlay image generation unit 363 generates and outputs overlay image data having the same size as the input image data using the read coordinates and operation amount (step SB6), and performs processing on the input image data. finish. An image when this overlay image data is displayed is shown in FIG. In this overlay image data, identification images P1 to P3 of colors corresponding to the operation amounts of the keys are located at the coordinates of the three keys, and the brightness of pixels that do not constitute these identification images P1 to P3 is 0. It has become. This overlay image data is input to the video mixer 37.

  The video mixer 37 transparently combines the image data from the camera 33 with the overlay image data from the overlay image generation unit 363 to generate composite image data, and outputs this. In this transparent composition, the overlay image data from the image generation device 36 is superimposed on the front surface of the image data from the camera 33, and the image data seen from the front side is composed when the pixel having 0 lightness in the latter image is made a transparent pixel. Let it be image data.

  The composite image data output from the video mixer 37 is input to the terminal 34 and transmitted to the terminal 24 via the network 1. The terminal 24 receives this composite image data and displays it on the display 241. As a result, the image shown in FIG. In this image, for the three keys, identification images P1 to P3 of colors corresponding to the key strike strength of the key are superimposed on the actual photographed image of the key.

  Thereafter, the key specifying unit 362 does not update the keyboard information table 361 until the next key is pressed or released by the student (step SA1: NO, SA3: NO). Therefore, the overlay image generation unit 363 repeats the same processing as described above (steps SB1 to SB3, SB4: YES, SB5, SB6). As a result, an image similar to that shown in FIG. However, since the student moves his hand in preparation for the next keystroke or key release while the above three keys are pressed, the background image of the identification images P1 to P3 continues to change.

  Thereafter, when another key is pressed prior to the release of the three keys, the key specifying unit 362 and the overlay image generating unit 363 repeat the same processing as described above (steps SA1: YES, SA2, SA3: NO, SB1 to SB3, SB4: YES, SB5, SB6) and the display 241 display more identification images.

  Here, it is assumed that when the image shown in FIG. 10 is displayed on the display 241, the student releases the above three keys simultaneously. When the key is released, MIDI data is output from the keyboard instrument 31. This MIDI data includes three note-off messages. As the note number in the note-off message, 96 is stored for the C key, 100 is stored for the E key, and 103 is stored for the G key.

  This MIDI data is input to the key specifying unit 362. Since the MIDI data does not include the note-on message but includes the note-off message, the key specifying unit 362 updates the keyboard information table 361 using the note-off message in the MIDI data ( Step SA1: NO, SA3: YES, SA4). In this update, as shown in FIG. 3, the operation state of the key (C, E, G key) of the key number (76, 80, 83) corresponding to the note number in the three note-off messages is “off”. " As a result, all the operation states in the keyboard information table 361 are “off”.

  On the other hand, when the image data output from the camera 33 is input immediately after the key release, the overlay image generation unit 363 updates the coordinates in the keyboard information table 361 by the same process as described above (steps SB1 to SB3). Next, the overlay image generation unit 363 refers to the keyboard information table 361 and determines whether there is a key whose operation state is “ON” (step SB4). At this time, since all the operation states in the keyboard information table 361 are “off”, the determination result is “NO”. Therefore, the overlay image generation unit 363 generates and outputs overlay image data having the same size as the input image data and all pixels are transparent pixels (step SB7), and performs processing on the input image data. finish. This overlay image data is input to the video mixer 37.

  The video mixer 37 transparently combines the image data from the camera 33 with the overlay image data from the overlay image generation unit 363 to generate composite image data, and outputs this. The composite image data output from the video mixer 37 is finally displayed on the display 241. As a result, the actual image itself around the keyboard 311 is displayed on the display 241 (see FIG. 6).

  As described above, according to the present system, on the display 241 of the teacher-side equipment 2, in the live-action images around the keyboard 311 of the student-side equipment 3, the real-action images of the keys pressed by the students are displayed for each key. An image emphasized by a color identification image corresponding to the operation amount is displayed. Therefore, the teacher can confirm whether or not an appropriate key is operated with an appropriate finger use only by referring to the display 241.

  By the way, when the installation position of the camera 33 is changed, the position of the key in the image data photographed by the camera 33 is changed. However, in this system, every time the overlay image generation unit 363 inputs image data from the camera 33, the reference position, the reference direction, and the reference length are specified and the coordinates in the keyboard information table 361 are updated. The display position of the identification image is always an appropriate position.

[Deformation]
The above-described embodiments may be modified as listed below.
For example, the same processing as the keyboard may be performed for the pedal of the keyboard instrument.
For example, the key type (white key / black key) is also stored in the keyboard information table 361, and if the pressed key is a black key, the identification image for the black key is not displayed or the black key is displayed. You may make it display on the position which overlaps with a key. Further, in these forms, the coordinates of each white key are calculated using the reference image in which the width of the white key is the length of the predetermined section, and the identification image for the white key is the center of the width of the white key It may be displayed in the vicinity.
For example, not only the reference position, the reference direction, and the reference length, but also the degree of distortion of the reference image B may be used for calculating the coordinates of each key. With this configuration, it is possible to cope with the case where the orientation of the camera 33 with respect to the keyboard 311 is variable.

For example, if the installation method of the camera 33 is limited, only the reference position and the reference direction may be used, only the reference position and the reference length may be used, or only the reference position may be used. Also good.
For example, the key coordinates may be obtained by recognizing the key image by a known image recognition technique. In this embodiment, the reference image B may be unnecessary, or the key image recognition process may be simplified by obtaining at least one of the reference position, the reference direction, and the reference length using the reference image B. Good.

  For example, a number representing the keystroke strength may be displayed as an identification image, or an identification image of the same color may be displayed even if the keystroke strength is different. The identification image to be displayed may be changed according to the key. For example, if the pitch corresponding to the key on which the identification image is to be displayed is “mi”, the character “mi” may be used as the identification image. Alternatively, the transmission composition may be performed so that the actual photographed image can be seen through the identification image.

For example, the identification image may be displayed at the position of the key that has not been operated, and the identification image may not be displayed at the position of the key that has been operated. In addition, the display position of the identification image may be a position near the corresponding key, not the position itself.
For example, the function of the image generation device 36 may be realized by a program. In this case, a computer that executes this program functions as an apparatus equivalent to the image generation apparatus 36.

  For example, the video mixer 37 may be integrated with the image generation device 36. Further, in the image generation device 36, the overlay image generation unit 363 may generate and output composite image data. In this case, the video mixer 37 is unnecessary. The image generating device 36 may be integrated with the keyboard instrument 31, or the image generating device 36 integrated with the video mixer 37 may be integrated with the keyboard instrument 31.

  By the way, the MIDI data including the note-on message or the note-off message represents the pitch of the musical sound. That is, MIDI data is a kind of musical tone signal. From this point of view, the key specifying unit 362 specifies a key to be pressed when a keyboard instrument emits a musical tone having a pitch represented by the input musical tone signal. Therefore, for example, the key specifying unit 362 can be modified to input a musical sound signal collected by a microphone and specify an operation unit (key) operated from the musical sound signal. In this mode, not only keyboard instruments that are not compatible with MIDI, but also learning of any musical instrument (for example, wind instrument) that can specify the operation unit to be operated from the pitch of the musical sound can be handled. In this embodiment, the operation amount of the operation unit is specified based on the magnitude (sound pressure / volume) of the collected tone signal.

It is a block diagram which shows the structure of the distance learning system which concerns on one Embodiment of this invention. It is a block diagram which shows the structure of the image generation apparatus 36 in the system. 4 is a diagram schematically showing the contents of a keyboard information table 361 in the image generating device 36. FIG. 5 is a flowchart showing a flow of processing performed by a key specifying unit 362 in the image generation device 36. 4 is a diagram schematically showing the contents of a keyboard information table 361 in the image generating device 36. FIG. It is a figure which shows the display image of the image data which the camera 33 in the system image | photographed. 5 is a flowchart showing a flow of processing performed by an overlay image generation unit 363 in the image generation apparatus 36. FIG. 10 is a diagram for explaining a principle of calculating coordinates in an overlay image generation unit 363. It is a figure which shows the display image of the overlay image data which the overlay image generation part 363 output. It is a figure which shows the image displayed on the display 241 of the terminal 24 which comprises the system.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Network, 2 ... Teacher side equipment, 24, 34 ... Terminal, 241, 341 ... Display, 3 ... Student side equipment, 31 ... Keyboard instrument, 311 ... Keyboard, 33 ... Camera, 36 ... Image generation device, 361 ... Keyboard Information table, 362... Key specifying unit, 363... Overlay image generating unit, 37.

Claims (4)

For a musical instrument having a plurality of operation units, each operation unit operated by a performer is converted into a sound generated by the operation of the operation unit or a signal generated from the instrument by the operation of the operation unit. An operation unit specifying means for specifying based on;
A live-action image photographing means for photographing a peripheral live-action image including each operation unit;
Position specifying means for specifying each position of each of the operation units based on the photographed image taken by the photographed image photographing means;
Overlay image generation means for determining which of the positions specified by the position specification means the position of the operation part specified by the operation part specification means and generating an identification image at the determined position;
Image combining means for generating and outputting a composite image by superimposing the identification image generated by the overlay image generating means on a corresponding position of the live-action image captured by the live-action image capturing means;
An image generation apparatus comprising: The operation amount specification that specifies the operation amount of each operation unit operated by the performer based on the sound of the instrument generated by the operation of the operation unit or the signal generated from the instrument by the operation of the operation unit Further comprising means,
The image generation apparatus according to claim 1, wherein the overlay image generation unit generates the identification image so as to have an aspect corresponding to the operation amount specified by the operation amount specification unit. In a musical instrument having a plurality of operation units,
Operation unit specifying means for specifying each operation unit operated by a performer based on the sound of the musical instrument generated by the operation of the operation unit;
A live-action image photographing means for photographing a peripheral live-action image including each operation unit;
Position specifying means for specifying each position of each of the operation units based on the photographed image taken by the photographed image photographing means;
Overlay image generation means for determining which of the positions specified by the position specification means the position of the operation part specified by the operation part specification means and generating an identification image at the determined position;
Image combining means for generating and outputting a composite image by superimposing the identification image generated by the overlay image generating means on a corresponding position of the live-action image captured by the live-action image capturing means;
A musical instrument characterized by comprising: In a musical instrument having a plurality of operation units,
An operation unit specifying means for specifying each operation unit operated by a performer based on a signal generated from the musical instrument by operation of the operation unit;
A live-action image photographing means for photographing a peripheral live-action image including each operation unit;
Position specifying means for specifying each position of each of the operation units based on the photographed image taken by the photographed image photographing means;
Overlay image generation means for determining which of the positions specified by the position specification means the position of the operation part specified by the operation part specification means and generating an identification image at the determined position;
Image combining means for generating and outputting a composite image by superimposing the identification image generated by the overlay image generating means on a corresponding position of the live-action image captured by the live-action image capturing means;
A musical instrument characterized by comprising:

Images