JP6589356B2 - Display control device, electronic musical instrument, and program - Google Patents

Description

  The present invention relates to a technique for supporting operation of an electronic musical instrument that generates a synthesized singing sound.

  In an electronic musical instrument such as an electronic keyboard device, in addition to generating an electronic sound that assumes a musical instrument sound or the like, a singing sound is synthesized and generated. Such a singing sound (to be distinguished from an actual singing, hereinafter referred to as “synthetic singing sound”) is, for example, combined with speech segments corresponding to characters constituting the lyrics so as to have a specified pitch. Generated by combining waveforms. The synthesized singing sound generated in this way can be heard as a synthesized sound in which the character is pronounced by a person. Conventionally, a technique for generating a synthesized singing sound by combining a musical score (sequence data or the like) prepared in advance with a character has been used. However, as in Patent Documents 1 and 2, it is used for a performance operation on an electronic keyboard device. In response to this, a technique for generating a synthesized singing sound in real time has been developed.

JP 2008-170592 A JP 2014-62969 A

  By the way, karaoke is known as an entertainment for enjoying the singing by the accompaniment of the song to be sung. In recent years, as an electronic musical instrument disclosed in Patent Documents 1 and 2, with the development of a device that can generate a synthesized singing sound in real time according to a performance operation by a performer, accompanied by a desired music piece, There is a desire to enjoy singing using synthetic singing sound instead of own singing voice.

  However, with conventional karaoke singing with your own voice, anyone can enjoy singing as long as they know the lyrics and music, but in order to enjoy singing using the electronic musical instruments described above, Therefore, a certain level of skill is required for the performance technique of electronic musical instruments. In other words, if you don't have the skills to play along with your accompaniment, you can't produce a synthetic singing sound that you think.

  For this reason, there is a problem that a person unfamiliar with the operation of the electronic musical instrument cannot go to karaoke and enjoy the singing using the electronic musical instrument.

  One of the objects of the present invention is to support a performance operation of an electronic musical instrument that generates a synthesized singing sound in karaoke.

  According to an embodiment of the present invention, a guide sequence data acquisition unit that acquires guide sequence data in which a pitch is assigned to each character constituting lyrics data, and the sound based on the guide sequence data A display data generation unit that generates display data corresponding to an operation support image indicating a high pitch and an order in which the pitches are assigned, and a key-on signal that acquires a key-on signal based on an operation of a performance operator that specifies a plurality of pitches There is provided a display control device including an acquisition unit and an image update unit that updates the operation support image in accordance with the acquired key-on signal.

  In addition to the pitches and the order in which the pitches are assigned, the operation support image indicates characters to which the pitches indicated in the operation support image in the lyrics data are assigned. Also good. The character to which the pitch is assigned may include a symbol that continuously pronounces the immediately preceding character (for example, a tone line “-”).

  The image updating unit includes a function of updating the currently displayed operation support image to an operation support image corresponding to the next phrase in response to an instruction signal indicating an instruction to postpone the image. Also good.

  According to an embodiment of the present invention, there is provided an electronic musical instrument comprising the display control device and the performance operator. The electronic musical instrument may further include a synthetic singing sound generation unit that generates a synthetic singing sound based on information on characters acquired in response to the key-on signal among the key-on signal and the lyrics data.

  According to an embodiment of the present invention, guide sequence data in which a pitch is assigned to each character constituting lyrics data is acquired, and the pitch and the pitch are assigned based on the guide sequence data. Display data corresponding to the operation support images indicating the order in which they are displayed, obtaining a key-on signal based on an operation of a performance operator designating a plurality of pitches, and obtaining the operation support image according to the acquired key-on signal. A program for causing a computer to perform updating is provided.

  According to an embodiment of the present invention, it is possible to support a performance operation of an electronic musical instrument that generates a synthesized singing sound.

It is a block diagram which shows the structure of the karaoke system in one Embodiment of this invention. It is a block diagram which shows the structure of the karaoke apparatus in one Embodiment of this invention. It is a block diagram which shows the structure of the control terminal in one Embodiment of this invention. It is a block diagram which shows the structure of the electronic keyboard apparatus in one Embodiment of this invention. It is a flowchart explaining the process in the synthetic song sound generation part in one Embodiment of this invention. It is a figure explaining the sequence data for guides in one Embodiment of this invention. It is a block diagram which shows the structure of the display control part in one Embodiment of this invention. It is a flowchart explaining the display process of the operation assistance image by the display control part in one Embodiment of this invention. It is a figure which shows an example of the operation assistance image in one Embodiment of this invention. It is a figure which shows an example of the operation assistance image in one Embodiment of this invention.

  Hereinafter, a karaoke system 100 according to an embodiment of the present invention will be described in detail with reference to the drawings. The following embodiments are examples of the embodiments of the present invention, and the present invention is not limited to these embodiments.

<Embodiment>
[overall structure]
The karaoke system 100 according to an embodiment of the present invention performs a performance operation of an electronic musical instrument on a performer when performing karaoke using an electronic musical instrument (for example, an electronic keyboard device) that can generate a synthesized singing sound. Has a function of displaying an image for assisting the user (hereinafter referred to as “operation support image”).

  FIG. 1 is a block diagram showing a configuration of a karaoke system 100 according to an embodiment of the present invention. The karaoke system 100 includes a karaoke device 1, a control terminal 2, an electronic keyboard device 3, a karaoke server 1000, and a song synthesis server 2000. The karaoke apparatus 1, the control terminal 2, the karaoke server 1000, and the singing composition server 2000 are connected via a network NW such as a LAN (Local Area Network), a WAN (Wide Area Network), or the Internet.

  The karaoke apparatus 1 has a function of reproducing an accompaniment of a song and displaying a video and lyrics telop associated with the song on a display screen when a song is designated. The accompaniment of the song is read from the mass storage device of the karaoke apparatus 1 and reproduced. When music data corresponding to the designated song is not stored, it can be downloaded from the karaoke server 1000.

  The control terminal 2 may be a portable information terminal such as a smartphone or a notebook personal computer, or may be a stationary information terminal such as a desktop personal computer. The function executed by the control terminal 2 may be realized by installing and executing an application program on the control terminal 2. In the present embodiment, the control terminal 2 is connected to the karaoke device 1 and the electronic keyboard device 3 through short-range wireless communication without going through the network NW, but may be connected through communication through the network NW. Good. As the short-distance wireless communication, for example, a communication unit using Bluetooth (registered trademark), infrared communication, or the like can be used.

  The karaoke server 1000 includes a storage device that stores music data necessary for providing karaoke in the karaoke apparatus 1 in association with the music ID. As the karaoke server 1000, a known karaoke server can be used. The music data includes data related to the singing song provided at karaoke, such as guide melody data (also referred to as main melody data), accompaniment data, lyrics data, and the like.

  The guide melody data is data indicating the melody (main melody) of the song. Accompaniment data is data indicating the accompaniment of a song. The guide melody data and accompaniment data may be data expressed in the MIDI format. Further, timing data (time data) may be added to the guide melody data and the accompaniment data. At this time, the guide melody data and the accompaniment data are data indicating music elements such as pitches and pronunciation levels, and the time data are data indicating pronunciation timing and pronunciation time.

  The lyric data is data that defines the lyrics of the song, and includes a plurality of character data. The character data is, for example, a character code based on JIS (Japanese Standards Association) standard, and a character code is set for each character. The lyrics data also includes timing data (time data) associated with the lyrics data. This time data includes data indicating the timing for changing the displayed lyrics (characters) and data for indicating the progress position of the song. By using this time data, it is possible to associate the progress position of the singing song with the lyrics to be sung at the progress position.

  The singing voice synthesizing server 2000 includes a storage device that stores various control data necessary for singing using the electronic keyboard device 3. The control data necessary for singing using the electronic keyboard device 3 includes, for example, setting data for performing various settings that define the operation and sound generation of the electronic keyboard device 3, and the player operates the electronic keyboard device 3. Guide sequence data for providing an operation support image for the operation is included. Of course, the setting data and the guide sequence data listed here are merely examples of control data, and the control data stored in the song synthesis server 2000 is not limited to these data.

  The setting data described above may be stored in association with the song ID. For example, timbre setting data (sound range setting data) for setting the timbre of the electronic keyboard device 3 in accordance with the melody (main melody) range of the music data corresponding to the song ID, vibrato, reverb, Effect effect setting data for providing an effect such as pitch bend can be included.

  Guide sequence data is data in which a pitch is assigned to each character constituting lyrics data. The guide sequence data is used to display an operation support image when the performer plays the electronic keyboard device 3. Details of the guide sequence data will be described later.

  The electronic keyboard device 3 is an example of an electronic musical instrument that includes a keyboard as a performance operator. When the electronic keyboard device 3 presses an arbitrary key, the electronic keyboard device 3 can generate a sound (synthetic singing sound) corresponding to a preset character with a pitch corresponding to the key. Therefore, it is possible to easily sing a synthesized singing sound by operating the keyboard. In the present embodiment, an electronic keyboard device is shown as an example of an electronic musical instrument. However, the present invention can be applied to all electronic musical instruments including a performance operator.

  Next, the hardware configuration of the karaoke device 1, the control terminal 2, and the electronic keyboard device 3 will be described.

[Hardware configuration of karaoke equipment]
FIG. 2 is a block diagram showing the configuration of the karaoke apparatus in one embodiment of the present invention. The karaoke apparatus 1 includes a control unit 101, a storage unit 103, an operation unit 105, a display control unit 106, an image display unit 107, a communication unit 109, and an audio signal processing unit 111. Each of these components is connected via a bus. The audio signal processing unit 111 is connected to an input terminal 113 for receiving an audio signal and a speaker 115. In this example, the audio signal input to the input terminal 113 is supplied by the electronic keyboard device 3 or supplied from a microphone.

  The control unit 101 includes an arithmetic processing circuit such as a CPU. The control unit 101 causes the CPU to execute a control program stored in the storage unit 103 to realize various functions in the karaoke apparatus 1. The storage unit 103 includes a storage device such as a RAM (Random Access Memory), a ROM (Read Only Memory), and a hard disk. The storage unit 103 stores the control program described above. The control program may be downloaded via a network, but may be provided in a state stored in a computer-readable recording medium such as a magnetic recording medium, an optical recording medium, a magneto-optical recording medium, or a semiconductor memory. Control programs used in the control terminal 2 and the electronic keyboard device 3 described below may be provided in the same manner. The storage unit 103 may also function as a buffer that temporarily stores music data downloaded from the karaoke server 1000 and image data to be displayed on the image display unit 107.

  The operation unit 105 is a device such as an operation panel, and outputs an instruction signal corresponding to the input operation to the control unit 101. The image display unit 107 is a display device such as a liquid crystal display or an organic EL display, and displays a screen based on control by the display control unit 106. The operation unit 105 and the image display unit 107 may be integrated to form a touch panel type image display unit.

  The display control unit 106 is a DSP (Digital Signal Processor) that controls an image displayed on the image display unit 107, and automatically generates image display data in accordance with a microprogram in response to an instruction from the control unit 101. To do. As the image to be displayed, a video related to a singing song, a lyrics telop, and the like are common, but in this embodiment, an operation support image for performing operation support for a player of the electronic keyboard device 3 is displayed. To do. More detailed functions of the display control unit 106 will be described later.

  The communication unit 109 connects to the karaoke server 1000 and the singing voice synthesis server 2000 via the network NW based on the control of the control unit 101. It is also possible to connect to the control terminal 2 and the electronic keyboard device 3 by short-range wireless communication or wireless LAN.

  The audio signal processing unit 111 includes a sound source that generates an audio signal from a MIDI format signal, an A / D converter, a D / A converter, and the like. The audio signal input from the input terminal 113 is output to the audio signal processing unit 111, subjected to A / D conversion in the audio signal processing unit 111, and output to the control unit 101. The input audio signal may be buffered in the storage unit 103 and used for singing evaluation or the like. The accompaniment data is read by the control unit 101, converted into an audio signal by the audio signal processing unit 111, and output as an accompaniment sound of the song from the speaker 115. At this time, the audio signal input to the input terminal 113 may be synthesized with the accompaniment sound and output.

[Hardware configuration of control terminal]
The control terminal 2 executes control on the karaoke device 1 (for example, designation of a song, volume adjustment, transpose, etc.) and control on the electronic keyboard device 3 (for example, setting of lyrics, setting of tone, etc.). Or run. Further, when the karaoke device 1 cannot directly communicate with the singing voice synthesis server 2000, the guide sequence data stored in the singing voice synthesis server 2000 is transferred to the karaoke device 1 or various control data are transferred to the electronic keyboard device. 3 can be provided.

  FIG. 3 is a block diagram showing the configuration of the control terminal in one embodiment of the present invention. The control terminal 2 includes a control unit 201, a storage unit 203, an operation unit 205, a display control unit 206, an image display unit 207, and a communication unit 209. Each of these components is connected via a bus.

  The control unit 201 includes an arithmetic processing circuit such as a CPU. The control unit 201 executes a control program stored in the storage unit 203 by the CPU, and realizes various functions in the control terminal 2. The storage unit 203 includes a storage device such as a RAM (Random Access Memory), a ROM (Read Only Memory), and a hard disk. The storage unit 203 stores the control program described above.

  The operation unit 205 is a device such as an operation panel, and outputs a signal corresponding to the input operation to the control unit 201. The image display unit 207 is a display device such as a liquid crystal display or an organic EL display, and displays a screen based on control by the display control unit 206. Note that the operation unit 205 and the image display unit 207 may be integrated to form a touch panel type image display unit.

  The display control unit 206 is a DSP that controls an image displayed on the image display unit 207, and automatically generates image display data according to a microprogram in response to an instruction from the control unit 201.

  The communication unit 209 connects to the karaoke server 1000 and the song synthesis server 2000 via the network NW based on the control of the control unit 201. It is also possible to connect to the karaoke device 1 or the electronic keyboard device 3 by short-range wireless communication, communication via a wireless LAN, or communication using a communication cable such as a USB cable.

[Hardware configuration of electronic keyboard device]
The electronic keyboard device 3 is a device (electronic musical instrument) that generates a synthesized singing sound in accordance with preset setting data contents in response to an operation of a keyboard that is a performance operator. In the present embodiment, the generated synthesized singing sound is output from the speaker 115 of the karaoke device 1 via the input terminal 113, but may be configured to be output from the speaker provided in the electronic keyboard device 3. Good.

  FIG. 4 is a block diagram showing the configuration of the electronic keyboard device according to one embodiment of the present invention. The electronic keyboard device 3 includes a control unit 301, a storage unit 303, an operation unit 305, a display control unit 306, an image display unit 307, a synthesized singing sound generation unit 308, a communication unit 309, and a keyboard 310. Each of these components is connected via a bus. In the present embodiment, the display control unit 306 and the image display unit 307 are provided so that lyrics can be displayed on the electronic keyboard device 3, but may be omitted if the lyrics need not be displayed.

  The control unit 301 includes an arithmetic processing circuit such as a CPU. The control unit 301 causes the CPU to execute a control program stored in the storage unit 303 to realize various functions in the electronic keyboard device 3. The storage unit 303 includes a storage device such as a RAM (Random Access Memory), a ROM (Read Only Memory), and a hard disk. The storage unit 303 stores the control program described above.

  The storage unit 303 stores setting data (tone color setting data, effect effect setting data, etc.) used when generating an audio signal indicating a synthesized singing sound, lyric data, and speech segments for generating a synthesized singing sound. To do. As the lyric data, it is only necessary to store at least character data indicating the lyrics of the designated song. Moreover, the lyrics data may be supplied from the karaoke apparatus 1 or may be supplied from the control terminal 2.

  The operation unit 305 is a device such as a switch, a volume knob, or a capacitance sensor, and outputs an instruction signal corresponding to the input operation to the control unit 301.

  The image display unit 307 is a display device such as a liquid crystal display or an organic EL display, and displays a screen based on control by the display control unit 306. Note that the operation unit 305 and the image display unit 307 may be integrated to form a touch panel type image display unit.

  The synthesized singing sound generation unit 308 generates a sound (synthetic singing sound) corresponding to the lyrics to be sung based on the setting data such as the timbre setting data described above, the lyrics data, the performance signal described later, the speech segment, and the like. To do. A well-known technique can be used regarding the production | generation of a synthetic song sound. For example, the techniques described in Patent Documents 1 and 2 described above may be used.

  The communication unit 309 communicates with the karaoke apparatus 1 and the control terminal 2 based on the control of the control unit 301. The communication performed here may be short-range wireless communication, communication via a wireless LAN, or communication using a communication cable such as a USB cable.

  A plurality of keys are arranged on the keyboard 310. When each key is operated, a performance signal corresponding to the operation is output to the control unit 301. The performance signal includes information indicating the position of the operated key (key number), information indicating that the key has been pressed (key on), information indicating that the key has been released (key off), key pressing speed (velocity), and the like. Specifically, when a key is pressed, key-on, velocity, and key number are output in association with each other, and when the key is released, key-off is output. The synthetic singing sound generation unit 308 generates a synthetic singing sound using the performance signal.

  Here, an example of the synthetic singing sound generation processing by the synthetic singing sound generation unit 308 will be described. FIG. 5 is a flowchart for explaining processing in the synthesized singing sound generation unit according to the embodiment of the present invention.

  When the process shown in FIG. 5 is started with the designation of the song song or the completion of the download of the lyrics data, first, the synthesized song sound generation unit 308 starts the first of the lyrics indicated by the lyrics data of the designated song song. A character is designated with a pointer (step S301). The lyric data may be stored in a predetermined memory area secured in the storage unit 303. In the present embodiment, the lyrics data of the currently designated song is stored in a predetermined memory area, and the contents of the memory area are updated each time a song is selected by the performer. However, when a plurality of lyrics data is stored in advance and a song is selected by the performer, the lyrics data to be sung can be selected using the identification number or the like.

  When the first character of the lyrics to be sung is designated, it is determined whether or not a key depression operation (performance operation) has been performed on the keyboard 310 (step S303), and the process waits until a key depression is performed (step S303). No). Whether or not a key pressing operation has been performed can be recognized by the control unit 301 acquiring a key-on signal output from the keyboard 310.

  When the control unit 301 acquires a key-on signal and recognizes that a key pressing operation has been performed on the keyboard 310 (step S303; Yes), the control unit 301 reads character data designated by the pointer (step S303). S305), and transfers to the synthetic singing sound generation unit 308. At this time, the key number indicating the position of the key on which the key pressing operation has been performed is also transferred to the synthesized singing sound generation unit 308.

  The synthesized singing sound generation unit 308 generates an audio signal indicating a pronunciation (synthetic singing sound) corresponding to the read character using the pitch information specified by the key number and the speech segment for uttering the read character. It is generated (step S307). For example, if the read character is “ka (ka)” and the key that has been pressed is “C4”, a synthesized singing sound that generates “ka (ka)” at the pitch of C4 is generated. Is done. The synthesized singing sound generated in this way is transmitted to the karaoke apparatus 1 via the communication unit 309 and reproduced. Of course, if the electronic keyboard device 3 includes a speaker, the sound may be reproduced from the speaker.

  When the synthesized singing sound of the character currently designated by the pointer is generated, it is determined whether or not the designated character (read character) is the last character of the lyrics (step S309). If the designated character is not the last character of the lyrics (step S309; No), the pointer is moved to the next character stored in the memory area described above and designated (step S311), and the next key press is performed. Wait until the operation (step S303; No). Thereafter, in the same procedure, characters are designated in order along the lyrics, and when it becomes the last character (step S309; Yes), the process returns to step S301 again. Of course, the processing may be terminated without returning the pointer to the beginning of the lyrics data.

  Through the above processing, lyrics adjusted to the pitch corresponding to the key pressing operation are sequentially generated as a synthesized singing sound. These synthesized singing sounds are output to the karaoke apparatus 1 as audio signals. When a predetermined operation on the operation unit 305 or the like is received, the next character may be forcibly designated.

  Also, if you have acquired information that categorizes the lyrics in phrase units, you can force the first character of one of the phrases (immediately before or after) to be specified when a given operation is accepted. Also good. For example, if there is data indicating a specific symbol such as a space or slash as character data in the lyrics data, the position where the symbol exists is determined as a phrase break, or a position indicating a phrase break in the timing data in the lyrics data Information may be provided, and the position indicated by the position information may be determined as a phrase break.

[Configuration of guide sequence data]
Next, before explaining the operation of the display control unit 106, guide sequence data that is a premise of the operation will be described. FIG. 6 is a diagram illustrating guide sequence data according to an embodiment of the present invention. Specifically, FIG. 6A shows a part of a guide melody and lyrics included in music data provided from the karaoke server 1000 in the form of a staff. FIG. 6B shows data (for example, note data) representing the main melody included in the guide sequence data provided from the song synthesis server 2000 and a part of the lyrics in the form of a staff.

  In the karaoke system 100 of the present embodiment, in order to generate an accurate synthesized singing sound from the electronic keyboard device 3, the performer generates the desired lyrics at the appropriate timing or the pitch being generated. There is a need to make changes. The timing can be determined by the performer himself in the same way as during normal singing, so no special operation support is required. Therefore, the operation support information for the performer may be information indicating which key of the electronic keyboard device 3 is pressed in accordance with the accompaniment of the designated song. As a result, the user can determine the key pressing timing from the accompaniment and press the correct key at that timing to generate a synthesized singing sound that matches the accompaniment and enjoy karaoke.

  Therefore, referring to FIG. 6A, when the guide melody and lyrics included in the music data provided from the karaoke server 1000 are converted into a notation format, the pitch and the lyrics may not always correspond to each other. That is, the lyrics (letters) of “I”, “KU”, “MO”, and “NO” correspond to one note each (ie, pitch), but “T” has a pitch. It does not correspond, and the lyric does not correspond to the eighth note “do”.

  Therefore, if the music data of the karaoke server 1000 is used as it is as the above-described operation support information, there may be a problem that the lyrics of “tsu” cannot be pronounced. In addition, when the “do” of the eighth note is pressed, the next lyrics are designated according to the output of the key-on signal as described with reference to FIG. The word “no” that should be sung will be pronounced. As a result, the lyrics that follow and the pitches are shifted, and there is a possibility that a correct synthesized singing sound cannot be produced.

  Therefore, in the present embodiment, guide sequence data is prepared for the same song as the song stored in the karaoke server 1000 and stored in the storage unit of the song synthesis server 2000. As shown in FIG. 6 (B), the guide sequence data is obtained by converting the music data, and for each of the timing when the lyrics characters change and the timing when the melody pitch changes, there is always one character and one character. The pitch is configured to correspond to the pitch. That is, one pitch is always assigned to each character (including the tone line “-”) constituting the lyrics. As a result, the key-pressing operation of the electronic keyboard device 3 and the character feed transition coincide with each other, and a correct synthesized singing sound can be generated.

  As an example of a method for converting music data to guide sequence data, the timing of notes and the timing of characters do not match by comparing the lyrics data included in the music data with the guide melody data. At the time position where only the note exists, a sound line ``-'' is added as a symbol that continuously pronounces the previous character, and the note with the same key number as the previous note is added at the time position where only the character exists. Add it. When the sound drawing line “-” is added as the lyrics, it is preferable that the vowel of the immediately preceding character is pronounced as a synthesized singing sound when a key is pressed correspondingly. Therefore, an appropriate vowel may be added to the time position where only the notes exist instead of adding the sound-drawing line “-” as the lyrics (for example, in the example of FIG. Or the like.

  In the present embodiment, an example is shown in which the guide sequence data is configured such that one pitch is always assigned to all the characters constituting the lyrics, but the present invention is not limited to this. For example, when it is natural to generate two-character lyrics with a single key-on operation for a fast-tempo music, etc., the sequence data for the guide is configured so that one pitch is assigned to the two-character lyrics. May be. In this case, information that one pitch corresponds to the corresponding two-character lyric data is included, and the first character is pronounced by the key-on signal and the second character is pronounced by the key-off signal. It is possible to respond.

  As described above, in the karaoke system 100 of the present embodiment, the music sequence data provided from the karaoke server 1000 is not used as it is, but guide sequence data corresponding to the music data is used. Thereby, when enjoying a karaoke using the synthesized singing sound of the electronic keyboard device 3, data suitable for the problem of displaying an operation support image that supports the operation of the electronic keyboard device 3 is provided to the display control unit 106. can do.

  In this embodiment, the guide sequence data is stored in advance in the song synthesis server 2000. However, the guide sequence data is stored in any of the devices constituting the karaoke system 100. Also good. In addition to storing in advance, guide sequence data may be generated each time a song is designated based on music data provided from the karaoke server 1000. In this case, the generation of the guide sequence data may be performed by any of the devices constituting the karaoke system 100. For example, the karaoke apparatus 1 may generate guide sequence data based on music data downloaded from the karaoke server 1000. Depending on the music, the music data downloaded from the karaoke server may be used as it is as guide sequence data.

[Configuration of display control unit in karaoke equipment]
A more detailed configuration of the display control unit 106 provided in the karaoke apparatus 1 will be described. Part or all of each configuration described below may be realized by hardware or may be realized by executing software. When implemented by executing software, the control unit 101 of the karaoke apparatus 1 reads out and executes the control program from the storage unit 103, thereby realizing part or all of the functions of the display control unit 106.

  FIG. 7 is a block diagram illustrating a configuration of the display control unit 106 according to an embodiment of the present invention. The display control unit 106 in the karaoke apparatus 1 of the present embodiment includes a guide sequence data acquisition unit 11, a display data generation unit 13, a key-on signal acquisition unit 15, and an image update unit 17. However, the guide sequence data acquisition unit 11 and the key-on signal acquisition unit 15 can also be configured as a common signal receiving unit, and the functions of both the display data generation unit 13 and the image update unit 17 are DSP (Digital Signal Processor). Or the like. The control unit 101 may also function as the display control unit 106 by executing software.

  The guide sequence data acquisition unit 11 acquires guide sequence data provided from the song synthesis server 2000. Since the karaoke apparatus 1 and the singing voice synthesizing server 2000 are communicably connected via the network NW, for example, the guide sequence data acquisition unit 11 can be constructed as a network communication processing unit using a known protocol. it can. The guide sequence data acquisition unit 11 may acquire the guide sequence data in units of one song or in units of phrases. Moreover, you may acquire in real time according to the accompaniment of the designated song. The acquired guide sequence data may be stored in the storage unit 103.

  The display data generation unit 13 displays display data corresponding to the operation support image indicating the pitches assigned to the characters constituting the lyrics data and the order in which the pitches are assigned based on the guide sequence data. Is generated. The operation support image provides support information indicating to the performer which key of which pitch should be operated in what order. Specific contents of the image will be described later. The display data generation unit 13 can read the guide sequence data from the storage unit 103 and generate an operation support image based on data (for example, note data) representing the main melody and lyrics data included in the guide sequence data. . At that time, the order of pressing keys on the keyboard can be determined according to the order of the notes or lyrics.

  The display data generation unit 13 may generate an operation support image for one song in advance before the accompaniment of the designated song is started, or may generate an operation support image for each phrase. Good. Further, when the guide sequence data is acquired in real time as described above, the display data can be generated in real time. In any case, a predetermined memory area of the storage unit 103 can function as a buffer.

  The display data generation unit 13 also has a function of generating display data for displaying a video (such as a video related to a song) displayed on the image display unit and a lyrics telop on a normal karaoke apparatus. Furthermore, the display data generation unit 13 also performs a process of synthesizing the above-described operation support image with these videos. A known technique can be used for the composition process, and for example, a two-screen composition process or an α blending process may be used. In addition, when performing these synthesis processes, the RAM of the storage unit 103 may be used as a work area.

  The key-on signal acquisition unit 15 acquires a key-on signal indicating the key depression of the keyboard output from the electronic keyboard device 3. Since the karaoke device 1 and the electronic keyboard device 3 are connected via short-range wireless communication, for example, the key-on signal acquisition unit 15 can be constructed as a communication processing unit using Bluetooth (registered trademark).

  The image update unit 17 updates the above-described operation support image according to the key-on signal acquired from the key-on signal acquisition unit 15. As will be described later, every time the performer presses the keyboard, the contents of the operation support image are changed so as to notify the key to be operated next. That is, the image update unit 17 updates the operation support image and changes the content, triggered by a key-on signal indicating a key pressing operation. Note that if there is a key-on operation indicating a key pressing operation, the image is updated regardless of whether or not the key is pressed as instructed.

  In the present embodiment, the display control unit 106 of the karaoke apparatus 1 has been described as an example, but each configuration may be provided in the control terminal 2 or the display control unit of the electronic keyboard device 3. Furthermore, each structure mentioned above is not restricted only to what is comprised only by a display control part, You may implement | achieve by distributed processing within the karaoke system 100 of this embodiment. For example, the control terminal 2 includes a guide sequence data acquisition unit and a display data generation unit, and the karaoke device 1 includes an image display unit, a key-on signal acquisition unit, and an image update unit. May exist.

  Here, an example of display processing of an operation support image by the display control unit 106 will be described. FIG. 8 is a flowchart for explaining the operation support image display processing by the display control unit according to the embodiment of the present invention.

  When the guide sequence data is downloaded to the storage unit 103 of the karaoke apparatus 1, the image display process of the display control unit 106 is started by an instruction from the control unit 101. First, note information (for example, note data) and character information (part of lyrics) for one phrase are acquired from the guide sequence data stored in the storage unit 103 (step S101).

  Next, display data representing an operation support image for one phrase is generated and displayed based on the acquired note information and character information and their arrangement order (step S102). Although details of the operation support image will be described later, for example, a keyboard image, a number indicating the order of pressing each key constituting the keyboard, a part of the lyrics, and the like are displayed.

  When the operation support image is displayed, the process waits until a key pressing operation is performed (step S103). When the key pressing operation is performed, the note and the character are updated to the operation support image in which the character is lowered by one character and displayed (step S104). To lower the note by one character means to change the display indicating the position of the key to be pressed to the next key position to be pressed. Moreover, moving down a character by one character means changing the display showing the character to be sung to the display showing the character to be sung next.

  When the update of the operation support image is completed, it is determined whether or not the processing for one phrase has been completed (step S105). If the processing for one phrase has not been completed (step S105; No), the process returns to step S103 again and waits for a key pressing operation. As described above, by sequentially repeating steps S103 to S105, the operation support images for one phrase are sequentially updated and displayed.

  When it is determined that the processing for one phrase has been completed (step S105; Yes), it is determined whether or not the processed phrase is the last phrase (step S106). At this time, if it is not the last phrase (step S106; No), the process returns to step S101 to obtain the note information and character information of the next phrase from the guide sequence data. If it is the last phrase (step S106; Yes), the image display process is terminated.

  Through the processing as described above, display processing of the operation support image is performed. In addition, although the example which starts the process of the next phrase after the process for 1 phrase was completed was shown, when the process of the 2nd-3rd character from the end of 1 phrase is complete | finished, the operation assistance image corresponding to the present phrase In addition, an operation support image corresponding to the next phrase may be displayed. For example, various display modes can be employed, such as displaying the next operation support image superimposed on the current operation support image, or displaying the next operation support image small on the screen edge or the like.

  Specific contents of the operation support image displayed by the operation of the display control unit 106 described above will be described. In the karaoke system 100 of the present embodiment, the operation support image is displayed on the image display unit 107 of the karaoke apparatus 1. Although the description will focus on the contents of the operation support image here, in actuality, the image and the lyrics telop are combined and displayed at the bottom of the screen of the image display unit 107. However, the position of the operation support image is not limited to the lower part of the screen, and may be arranged at the upper part or at either the left or right end part. In this case, the moving direction of the characters and the direction of the keyboard, which will be described later, can be appropriately changed according to the position where the operation support image is arranged.

[Configuration Example 1 of Operation Support Image]
FIG. 9 is a diagram illustrating an example of an operation support image according to the embodiment of the present invention. FIG. 9A shows an operation support image 20a that is displayed immediately after the start of a phrase with the designated song, and FIG. 9B shows a single key press from the state of FIG. 9A. The operation support image 30a displayed when the operation is performed is shown. In addition, although the example which also displays a lyric (character) is shown in FIG. 9, a lyric is not indispensable, What is necessary is just what can understand the order to press a key at least.

  In the example shown in FIG. 9A, an image 21 representing a piano keyboard is displayed, and a plurality of numbers 23 are displayed in the order of pressing keys on each key 22 constituting the keyboard. In the lyrics display area 24, characters 25 representing a part of the lyrics to be sung are displayed. Of the lyric display area 24, the first area 24a displays a character to be sung first (here, “I”), and the second area 24b displays a subsequent character to be sung (here, “ ”,“ Tsu ”,“ mo ”,“-”and“ no ”) are displayed. Note that the characters displayed in the first area 24a may be displayed so as to gradually descend downward as indicated by arrows.

  At this time, the display position of each number 23 and the display position of each character 25 are in a one-to-one correspondence. That is, the numbers 23 indicate the order in which the pitches are assigned to the characters 25, and the order in which the performer presses the keys. In addition, the number of the numbers 23 and the characters 25 displayed at a time may be a number included in one phrase or an arbitrary number.

  The player who has confirmed the operation support image 20a shown in FIG. 9A on the screen has a key corresponding to the letter “I” to be sung first in accordance with the accompaniment of the designated song (ie, the number “1”). Press the key to which "" is assigned. In response to the key-on signal output by the key depression, the synthetic singing sound generation unit 308 of the electronic keyboard device 3 generates the synthetic singing sound of “I” with the pitch of “Mi” according to the flow of FIG. 5 described above. An audio signal corresponding to the synthesized singing sound is sent to the karaoke apparatus 1 and is produced from the speaker 115. When the performer presses a key different from the instruction, a synthesized singing sound of the character “I” based on the pitch of the pressed key is generated.

  FIG. 9B shows the operation support image 30a immediately after the pronunciation of “I” in the above procedure. In the operation support image 30a shown in FIG. 9B, the pronounced character “I” disappears from the screen, and the next character to be sung (ie, the number “2”) assigned to the second area 24b is assigned. ), “Ku”, has moved from the second area 24b to the first area 24a. By confirming the operation support image 30a shown in FIG. 9B, the performer confirms that the key to be pressed next is “mi”, which is one octave higher than the previous key, and that the lyrics to be sung are You can recognize that it is “ku”.

  As described above, the performer can know the position of the keyboard to be operated next by confirming on the screen the operation support image that is changed each time the key is pressed. Further, since the order of a plurality of key pressing operations can be predicted in units (for example, phrase units) displayed on the screen, the position of the hand can be prepared in advance, and the performance can be performed smoothly. Furthermore, by confirming the lyrics (characters), it is possible to determine the pronunciation timing of the synthesized singing sound while anticipating the future in units of phrases. Therefore, even if you are unfamiliar with the performance operation, it is possible to easily generate the correct synthesized singing sound. You will be able to enjoy it.

  In the example shown in FIG. 9A, the character “NO” is highlighted (specifically, it is surrounded by a double circle). This highlighting indicates an instruction to effect effects such as vibrato, reverb, and pitch bend at the character “no”. As the highlighting mode, various modes such as color coding or shading with other characters can be applied. Furthermore, it is possible to distinguish what kind of effect instruction is given by attaching a specific mark.

  In the example shown in FIGS. 9A and 9B, the number display is not changed even if the keyboard operation is advanced. For example, when the key with the number “1” is pressed, The number “1” disappears, and the display after the number “2” is incremented by one (that is, the number “2” becomes the number “1” and the number “3” becomes the number “2”). It may be changed.

  In the example shown in FIGS. 9A and 9B, the sounded character disappears from the first region 24a, and the character to be sung next moves from the second region 24b to the first region 24a. It is also possible to change the display so that the pronounced characters disappear on the spot. Of course, it is also possible to change the display in another manner so that the player can know that the sound has already been pronounced, such as changing the color without erasing the display.

[Configuration Example 2 of Operation Support Image]
FIG. 10 is a diagram showing an example of the operation support image in the embodiment of the present invention. FIG. 10A shows an operation support image 20b that is displayed immediately after the start of a phrase having a designated song, and FIG. 10B shows a single key press from the state of FIG. The operation support image 30b displayed when the operation is performed is shown. In addition, although the example which also displays a lyric (character) is shown in FIG. 10, a lyric is not indispensable, What is necessary is just what can understand the order to press a key at least.

  In the example shown in FIG. 10 (A), an image 31 representing a piano keyboard is displayed, and each key 32 constituting the keyboard is given a first highlight on the key 32a to be pressed at the present time, A second highlight is given to the key 32b to be pressed second, and a third highlight is given to the key 32c to be pressed third. That is, the performer can recognize the key with the first highlighting as the key to be pressed by checking the operation support image 20b.

  In the example of FIG. 10A, highlighting is performed so that the order in which the keys 32b and 32c should be pressed can be distinguished. However, the same order is used without distinguishing the order of the keys 32b and 32c. It is also possible to give a highlight.

  In the lyrics display area 33, characters 34 representing a part of the lyrics to be sung are displayed. Each character 34 is displayed at a position closer to the key as the order of key depression becomes earlier. Thereby, in addition to the key 32a to be pressed next, the performer can confirm the lyrics (characters) to be pronounced at that time. The position of each character may be continuously changed from the bottom so as to gradually approach the key 32 as indicated by an arrow, or stepwise so as to approach the key 32 each time the key is pressed. It may be a mode in which the position changes. Note that the number of characters 34 displayed in the lyrics display area 33 at a time may be a number included in one phrase or an arbitrary number.

  The player who has confirmed the operation support image 20b shown in FIG. 10A on the screen displays a key (first highlighting) corresponding to the letter “T” to be sung at the present time in accordance with the accompaniment of the designated song. The key 32a) marked with is depressed. In response to the key-on signal output by the key depression, the synthetic singing sound generation unit 308 of the electronic keyboard device 3 generates the “single” synthetic singing sound with the pitch of “mi” according to the flow of FIG. 5 described above. An audio signal corresponding to the synthesized singing sound is sent to the karaoke apparatus 1 and is produced from the speaker 115.

  FIG. 10B shows the operation support image 30b immediately after the pronunciation of “tsu” in the above procedure. In the operation support image 30 b shown in FIG. 10B, the pronounced character “T” disappears from the screen, and in the lyrics display area 33, “MO”, which is the next character to be sung, is closest to the key 32. Specifically, the key 32b that has been second highlighted as the key to be pressed next in FIG. 10A is changed to the first highlighted key in FIG. 10B. The character “mo” is closest to 32b. Similarly, the characters “-” and “no” approach the key 32c, respectively, and the highlighting of the key 32c is changed to the second highlighting.

  By confirming the operation support image 30b shown in FIG. 10B, the performer recognizes that the key to be pressed next is “L” and that the lyrics to be sung is “M”. can do.

  As described above, the performer can know the position of the keyboard to be operated next by checking on the screen the operation support image that is changed every time the key is pressed, and is not accustomed to the performance operation. It is also possible to easily generate the correct synthesized singing sound. Thereby, even a person unfamiliar with the operation of the electronic musical instrument can go to karaoke and enjoy the singing using the electronic musical instrument.

  In the example shown in FIG. 10A, the character “NO” is highlighted with a double line, but the meaning is the same as described in FIG. To do. Of course, as in FIG. 9, various modes such as color coding or shading with other characters can be applied as highlighting modes. Furthermore, it is possible to distinguish what kind of effect instruction is given by attaching a specific mark.

[Configuration Example 3 of Operation Support Image]
The operation support image in the present embodiment may be displayed in the form of a staff notation as shown in FIG. That is, from the time information included in the guide sequence data, as shown in FIG. 6B, a note indicating the pitch of the phrase to be sung and the lyrics (characters) corresponding to the note are added in association with each other. The obtained image can be displayed as an operation support image.

  In the present embodiment, display data is created based on guide sequence data for performing singing by playing on the electronic keyboard device 3 instead of music data provided from a normal karaoke server. A pitch is always assigned to each character that constitutes. Therefore, if an image obtained by associating and adding a note indicating the pitch of a phrase to be sung and a lyrics (character) corresponding to the note as an operation support image is referred to, the performer can use the correct pitch and lyrics. Singing with synthetic singing sound can be performed.

<Other embodiments>
In the embodiment described above, the operation support image is updated in response to the key-on signal acquired from the electronic keyboard device 3. In other words, since the operation support image is not updated unless the key-on signal is acquired, there may be a time lag between the accompaniment and the operation support image if the performer cannot keep up with the accompaniment speed. Also, if the performer mistakenly presses two keys at a time, the lyrics will be advanced one more, and the correspondence between the pitch and the lyrics after that may be shifted.

  In preparation for such a case, the operation unit 305 of the electronic keyboard device 3 in another embodiment includes an operator that outputs an instruction signal (hereinafter referred to as a “skip instruction signal”) to postpone the operation support image in units of phrases. May be included. When the image update unit 17 included in the display control unit 106 of the karaoke apparatus 1 acquires the skip instruction signal from the electronic keyboard device 3, the image update unit 17 performs an operation support image update process so that the operation support image of the next phrase is displayed. . As a result, if the performance operation is delayed or an erroneous operation is performed, the performer can forcibly postpone the operation support image in phrase units and reset the performance delay or deviation. it can.

  Further, instead of the configuration in which the player postpones the phrase by his / her own operation, the control unit 101 of the karaoke apparatus 1 automatically detects a deviation (phase difference) between the accompaniment and the performance operation, and the deviation is a predetermined threshold value. It is also possible to adopt a configuration in which the image update processing unit 17 is controlled so as to become an operation support image of the next phrase when the value exceeds. In this case, for example, the control unit 101 compares the time data included in the accompaniment data with the position in the lyrics data of the lyrics that should be sung at the present time indicated by the operation support image, so that the deviation between the accompaniment and the performance operation is detected. It is only necessary that the above-described skip instruction signal is transmitted to the image update unit 17 when the detection and the deviation exceeds a predetermined threshold. Alternatively, the skip signal described above may be transmitted when a user's performance mistake is detected, such as continuously operating a key different from the instructed pitch. Note that the process of detecting the difference between the accompaniment and the performance operation may be executed while waiting for the key pressing operation in step S103 in the operation support image display processing flow shown in FIG. 8, for example. At this time, if a shift between the accompaniment and the performance operation is detected, it is preferable to return to step S101 and display the operation support image of the next phrase.

  In the above-described embodiment, the case where the display control unit 106 of the karaoke apparatus 1 generates and displays the operation support image has been described as an example. However, the display control unit of the control terminal 2 or the electronic keyboard device 3 performs the operation support. Image generation and display processing may be performed. That is, an existing device is used as the karaoke server 1000 and the karaoke device 1, and operation support image generation processing and display processing may be performed using the display control unit or the image display unit of the control terminal 2 or the electronic keyboard device 3. Is possible. This is because, for example, the performance of the electronic keyboard device 3 is performed while viewing the operation support image displayed on the image display unit 207 of the control terminal 2, or the operation displayed on the image display unit 307 provided in the electronic keyboard device 3 itself. It means that you can go while watching the support image.

  Moreover, since the function as a display control part may be implement | achieved by distributed processing within the karaoke system 100 as mentioned above, it is also possible to share the function as a display control part with the karaoke apparatus 1 and the control terminal 2. is there. For example, the display control unit 206 of the control terminal 2 downloads guide sequence data from the song synthesis server 2000 and generates display data, and triggered by acquisition of a key-on signal and acquisition of a key-on signal from the electronic keyboard device 3. The display control unit 106 of the karaoke apparatus 1 may perform update processing of the operation support image to be performed.

  As described above, in the karaoke system 100, the functions of the karaoke device 1, the control terminal 2, and the electronic keyboard device 3 are realized in the respective devices. May be realized by other devices. For example, some or all of the functions in the control terminal 2 may be realized in the electronic keyboard device 3. When all the functions of the control terminal 2 are realized in the electronic keyboard device 3, it is equivalent to the control terminal 2 and the electronic keyboard device 3 being an integrated device. On the contrary, each function may be realized by using more devices. For example, among the electronic keyboard device 3, a device (synthetic singing sound generation device) provided with a synthesized singing sound generation unit 308 and a device (for example, a MIDI keyboard) provided with a keyboard 310 portion are configured separately. Also good. At this time, the synthetic singing sound generation device may be included in the control terminal 2.

  The synthesized singing sound is generated when the performer presses a key on the keyboard 310 which is an example of a performance operator. However, the performance operator may be other than a key, for example, a string of a guitar synthesizer (guitar type controller) or a key of a wind synthesizer (wind instrument type controller).

DESCRIPTION OF SYMBOLS 1 ... Karaoke apparatus, 2 ... Control terminal, 3 ... Electronic keyboard apparatus, 11 ... Sequence data acquisition part for guides, 13 ... Display data generation part, 15 ... Key-on signal acquisition part, 17 ... Image update part, 20a, 20b, 30a , 30b ... operation support image, 21, 31 ... keyboard, 22, 32 ... key, 24, 33 ... lyrics display area, 25, 34 ... text, 100 ... karaoke system, 101, 201, 301 ... control unit, 103, 203 , 303: Storage unit, 105, 205, 305 ... Operation unit, 106, 206, 306 ... Display control unit, 107, 207, 307 ... Image display unit, 109, 209, 309 ... Communication unit, 111 ... Audio signal processing unit , 113 ... input terminal, 115 ... speaker, 308 ... synthetic singing sound generation unit, 310 ... keyboard, 1000 ... server for karaoke, 2000 ... server for singing synthesis

Claims (8)

A guide sequence data acquisition unit for acquiring guide sequence data in which pitches are assigned to each character constituting the lyrics data;
A display data generation unit that generates display data corresponding to the operation support image indicating the pitch and the order in which the pitches are assigned based on the guide sequence data;
A key-on signal acquisition unit that acquires a key-on signal based on an operation of a performance operator that specifies a plurality of pitches; and
An image update unit that updates the operation support image according to the acquired key-on signal;
Bei to give a,
The operation support image shows the position of the performance operator to be operated next,
The update of the operation support image includes updating the position of the performance operator to be operated next each time the key-on signal is acquired . The update of the operation support image displays the position of the performance operator to be operated next every time the key-on signal is acquired, from the area where the character to be sung is displayed, and the character to be sung first. The display control device according to claim 1, comprising updating to a region to be processed.   The display according to claim 1, wherein the operation support image indicates a character to which the pitch indicated in the image is assigned in the lyrics data in addition to the pitch and the order in which the pitch is assigned. Control device. The display control apparatus according to claim 3 , wherein the character to which the pitch is assigned includes a symbol that continuously pronounces the immediately preceding character.   The image updating unit includes a function of updating an operation support image currently displayed to an operation support image corresponding to a next phrase in response to an instruction signal indicating an instruction to postpone an image. Display controller. A display control device according to any one of claims 1 to 5 ,
The performance operator;
An electronic musical instrument. Furthermore, the electronic musical instrument of Claim 6 provided with the synthetic | combination song sound production | generation part which produces | generates a synthetic song sound based on the information regarding the character acquired according to the said key-on signal among the said key-on signal and the said lyric data. Obtain sequence data for guides with pitches assigned to each character constituting the lyrics data,
Based on the guide sequence data, generate display data corresponding to the operation support image indicating the pitch and the order in which the pitch is assigned,
Acquires a key-on signal based on the operation of a performance controller that specifies multiple pitches,
A program for causing a computer to update the operation support image according to the acquired key-on signal,
The operation support image shows the position of the performance operator to be operated next,
The update of the operation support image includes updating the position of the performance operator to be operated next each time the key-on signal is acquired .

Images