JP5598516B2 - Voice synthesis system for karaoke and parameter extraction device - Google Patents

Description

  The present invention relates to a speech synthesis system that performs speech synthesis in karaoke and a parameter extraction device that extracts speech parameters necessary for speech synthesis from speech.

2. Description of the Related Art Conventionally, there is known a karaoke apparatus that corrects and outputs a pitch pitch of a singing voice input by a user singing (see Patent Document 1).
In the karaoke apparatus described in Patent Document 1, the pitch pitch of the singing voice input by the user singing is matched with the closest pitch among the pitches used for the musical tone in the music. It is corrected as follows.

JP 2003-167487 A

  In the karaoke apparatus described in Patent Document 1, the singing voice with the corrected pitch is output during a period in which the user utters and inputs the voice to the karaoke apparatus, but the voice is input. It is not output during no period.

  In general, while the user is singing, the uttered sound before correction is also heard by the user's own ear. For this reason, it is difficult to listen to two types of singing voices at the same time according to the performance sound and practice singing. It is done listening.

  However, in the karaoke device described in Patent Document 1, the user's own singing voice is listened to at an appropriate pitch of the music because the user himself is singing the music. After singing, and even for other songs, there was a problem that it was difficult to practice singing without singing again.

  Therefore, an object of the present invention is to make it possible to listen to the user's own singing voice at an appropriate pitch of the music other than the period during which the user sings the music.

The present invention made to achieve the above object relates to a speech synthesis system including a parameter extraction device and a synthesized sound output device.
The parameter extraction apparatus constituting the speech synthesis system of the present invention includes utterance information acquisition means, waveform acquisition means, parameter derivation means, parameter registration means, and model storage means.

  Among these, the utterance information acquisition means includes a song ID for identifying a song, performance information representing a musical sound constituting the song identified by the song ID, lyrics information representing the lyrics of the song identified by the song ID, and the lyrics A song ID, performance information, lyrics information, and utterance timing information are acquired from song data including utterance timing information indicating the utterance start timing of the lyrics represented by the information, and a song based on the song data is reproduced.

  Then, the waveform acquisition means acquires the input voice waveform at the utterance start timing of the lyrics during the reproduction of the music based on the music data, and the parameter derivation means uses the voice for each syllable forming the lyrics from the voice waveform. A syllable waveform that is a waveform is extracted, and a speech parameter that is at least one feature amount defined in advance is derived from each extracted syllable waveform.

The parameter registering unit stores the voice parameter for each syllable derived by the parameter deriving unit in the first storage device in association with the user ID for identifying the user who sang the song. Further, the model storage means stores model voice data including at least one voice parameter based on an ideal singing voice for each song in the second storage device in association with the song ID.

On the other hand, the synthesized sound output device constituting the speech synthesis system includes identification information acquisition means, parameter acquisition means, model acquisition means, speech synthesis means, and output means.
In the synthesized speech output device, the identification information acquiring means, the specified music ID, and is acquired user ID. The model voice data associated with acquired music ID of that, the model obtaining unit obtains from the second storage device.
Further, the parameter acquisition unit is a voice parameter associated with the user ID acquired by the identification information acquisition unit from the voice parameters stored in the first storage device, and the model acquired by the model acquisition unit The sound parameter most similar to the sound parameter associated with the music ID included in the sound data is acquired.

The voice synthesis unit adjusts the voice parameter acquired by the parameter acquisition unit to match the voice parameter associated with the music ID, and performs voice synthesis based on the adjusted voice parameter . Then, the output unit outputs the generated synthesized sound.
Furthermore, the utterance information acquisition means represents the score of the target song that is one of the songs, and for each output sound output from the sound module, at least the musical score data in which the pitch and the performance start timing are defined is the performance information. Get as. If the musical score data is transposed in the music of the target music, a modulation flag that indicates the time at which the target music is transposed along the time axis is included .
The parameter registration means of the present invention comprises section specifying means, main sound specifying means, pitch name frequency deriving means, and key estimating means.
Among these, the section specifying means specifies the same key section, which is each section in which the same key is continued in the target music, based on the acquired musical score data. The main sound specifying means is included in each key-same section specified by the section specifying means, and specifies the last output sound along the time axis in each key-same section as the main sound. Then, the pitch name frequency deriving means starts from the pitch name of the main tone specified by the main tone specifying means, with the appearance pitch name frequency representing the frequency of the output sound of the same pitch name included in the same key interval specified by the interval specifying means. Is derived for each of the same key intervals. Furthermore, the key estimation means is a result of matching each appearance name frequency derived by the pitch name frequency deriving means to a key template prepared in advance for each key as a template representing the distribution of pitch names that can be used in each key, Each key having the highest correlation is estimated as metadata.
The parameter registration means of the present invention stores the metadata estimated by the key estimation means in the first storage device in association with the speech parameter.

According to such a voice synthesis system, after the user sings one piece of music, the user voice in the model singing state can be heard.
That is, using the speech parameters derived from the user's singing voice, speech synthesis using the user's own voice can be performed so that the speech is output at the utterance start timing and the utterance pitch of the music.

  In addition, in the speech synthesis system of the present invention, after generating the speech parameter and model singing voice data for one piece of music, it is not necessary to generate the voice parameter and the model singing voice data for another piece of music again.

From the above, according to the speech synthesis system of the present invention, in addition to the period in which the user sings the song, the user's own singing voice can be listened to various songs including appropriate songs at appropriate pitches. Can be possible.
In the speech synthesis system of the present invention, metadata can be automatically estimated based on specific content information and utterance timing information. For this reason, according to the speech synthesis system of the present invention, when the content of the character string represented by the utterance content information is uttered, it is not necessary for the users to input the property of the speech as metadata. it can.

  Note that the feature amount as a speech parameter in the present invention is a feature amount required when executing speech synthesis by formant synthesis. This feature amount includes, for example, a fundamental frequency, a mel frequency cepstrum (MFCC), power, and their respective time differences.

  The utterance information acquisition means in the speech synthesis system of the present invention represents the score of the target song that is one of the songs, and for each output sound output from the sound module, at least the pitch and the performance start timing are defined. Data may be acquired as performance information. In addition, the utterance information acquisition unit may acquire a character string of the lyric constituent characters constituting the lyrics of the target music as lyric information. Further, the utterance information acquisition means acquires, as the utterance timing information, the lyric output timing in which the output timing for at least one of the lyrics constituent characters is associated with the performance start timing of the output sound corresponding to the lyrics constituent characters. Also good.

  In these cases, the waveform acquisition means in the speech synthesis system of the present invention acquires, as a speech waveform, a waveform in which the voice input during the performance of the target music based on the score data has shifted along the time axis, and parameter derivation means However, in the speech waveform, the speech waveform in the section corresponding to each output sound may be extracted as a syllable waveform.

According to such a speech synthesis system, Ru can collect speech waveform input to the period in which playing object song based on musical score data.

  In addition, the nature of speech in the present invention includes at least the emotion of the speaker when the speech is uttered, and is a concept that includes, for example, emotion and atmosphere. Furthermore, information necessary for estimating emotions may be included in the nature of speech.

  Further, in the parameter registration means in the present invention, the word dividing means divides the character string represented by the lyric information acquired by the utterance information acquiring means for each word character that is a character string constituting the word, and extracts metadata. The means prepares in advance a word property table in which the property information representing the property of each word is associated with the identification information of the word, and the property information corresponding to the word represented by each word character divided by the word dividing device May be extracted from the word property table as metadata.

At the same time, the parameter registration unit may store the metadata extracted by the metadata extraction unit in the first storage device in association with the audio parameter.
According to such a speech synthesis system, the property of each word can be used as metadata. The nature of the word mentioned here includes the meaning of the word and the emotion represented by the word.

In addition, this invention may be made | formed as a parameter extraction apparatus. In this case, it is preferable that the parameter extraction device of the present invention includes the utterance information acquisition unit, the waveform acquisition unit, the parameter derivation unit, the parameter registration unit, and the model storage unit.
Further, when the present invention is configured as a parameter extraction device, the parameter registration means includes a section specifying means, a main tone specifying means, a pitch name frequency deriving means, and a key estimating means, and the key estimating means The estimated metadata may be stored in the first storage device in association with the audio parameter.
In the case where the present invention is implemented as a parameter extracting device, the parameter registering unit includes a word dividing unit and a metadata extracting unit, and the metadata extracted by the metadata extracting unit is used as an audio parameter. You may match and memorize | store in a 1st memory | storage device.

According to such a parameter extraction device, when a user sings a piece of music for the first time, voice parameters can be derived and model singing voice data can be generated.
Then, according to the parameter extraction device of the present invention, based on the voice parameters stored in the first storage device and the model singing voice data stored in the second storage device, the voice start timing and the voice pitch in the music The speech synthesizer can perform speech synthesis so that the speech is output.

1 is a block diagram showing an overall configuration of a speech synthesis system to which the present invention is applied. It is a flowchart which shows the process sequence of a parameter registration process. It is a flowchart which shows the process sequence of a parameter data registration process. It is a flowchart which shows the process sequence of the metadata estimation process in 1st embodiment. It is a figure which shows the processing content of a metadata estimation process. It is a figure which shows the processing content of a metadata estimation process. It is a figure which shows the processing content of a metadata estimation process. It is a figure which shows the outline | summary of the parameter data registration process in 1st embodiment. It is a figure which shows the outline | summary of parameter data and model singing voice data. It is a flowchart which shows the process sequence of an exemplary singing voice data registration process. It is a flowchart which shows the process sequence of a speech synthesis process. It is a flowchart which shows the process sequence of the metadata estimation process in 2nd embodiment. It is a figure which shows the outline | summary of the parameter data registration process in 2nd embodiment.

Embodiments of the present invention will be described below with reference to the drawings.
[First embodiment]
<Speech synthesis system>
As shown in FIG. 1, the speech synthesis system 1 outputs an ideal singing voice in a music designated by a user (singer) (hereinafter referred to as a designated music) as the user's own voice. This is a system for performing speech synthesis.

  In order to realize this, the speech synthesis system 1 includes a speech input device 10, an information storage server 25, an information processing device 30, a data storage server 50, and a speech output terminal 60.

  The voice input device 10 plays a song based on the music data MD used for karaoke, and accepts voice input during the performance of the song. The information storage server 25 stores each of the music data MD prepared for each music piece, and stores voice waveform data SV that is data of each voice input via the voice input device 10.

  Furthermore, the information processing apparatus 30 generates parameter data PM and model singing voice data ED based on the voice waveform data SV and the music data MD stored in the information storage server 25.

  The parameter data PM referred to here is data including at least one voice parameter used for so-called formant synthesis, which will be described later in detail. The model singing voice data ED is data including at least one voice parameter based on an ideal singing voice for each music piece. The at least one speech parameter includes, for example, a fundamental frequency at each syllable in the uttered speech, a mel frequency cepstrum (MFCC), power, and a time difference therebetween.

The data storage server 50 stores parameter data PM and model singing voice data ED generated by the information processing apparatus 30. Also, the voice output terminal 60 is based on the parameter data PM and the model singing voice data ED stored in the data storage server 50, so that the voice of the user himself / herself becomes an ideal singing voice when singing the designated music piece. The synthesized sound synthesized by voice is output. Note that the speech synthesis system 1 in this embodiment may include a plurality of speech output terminals 60.
<Information storage server>
The information storage server 25 is a device that is mainly configured of a storage device configured to be able to read and write stored contents, and is connected to the voice input device 10 via a communication network.

  The music data MD stored in the information storage server 25 includes music MIDI data DM, lyrics data group DL, and guide vocal data GD, and is associated with each corresponding music. Furthermore, when the music is a duet music, the music data MD may include information for distinguishing the male and female parts of the lyrics.

  The music MIDI data DM is data representing the score of one music according to the well-known MIDI (Musical Instrument Digital Interface) standard. Each of the music MIDI data DM includes identification data that is data for discriminating music, a score track that represents a score for each musical instrument used in the music, and a modulation flag that represents the time at which the key changes in the music. Have at least.

  The musical score track defines at least a pitch (so-called note number) and a period during which the sound module outputs the output sound (hereinafter, note length) for each output sound output from the MIDI sound source. Yes. However, the note length of the musical score track is the performance start timing (so-called note-on timing) indicating the time from the start of the performance of the music until the output of the output sound is started, and the output of the output sound is ended. It is defined by the performance end timing (so-called note-off timing) that represents the time from the start of performance of the music.

Note that a score track is prepared for each instrument type.
On the other hand, the lyrics data group DL is data relating to lyrics displayed on a display device that constitutes a well-known karaoke device, and lyrics telop data DT representing characters constituting the lyrics of the music (hereinafter referred to as lyrics constituent characters); And lyrics output data DO in which a timing correspondence relationship for associating the lyrics output timing, which is the output timing of the lyrics constituent characters, with the performance of the music MIDI data DM is provided.

  Specifically, the timing correspondence relationship in the present embodiment is based on the timing of starting the output of the lyrics telop data DT to the timing of starting the performance of the music MIDI data DM, and along the time axis of the specific music. In addition, the lyrics output timing of each lyrics constituent character is defined by the elapsed time from the start of the performance of the music MIDI data DM. Note that the elapsed time referred to here is, for example, a time indicating the timing for executing color change of displayed lyrics constituent characters, and is defined by the speed of color change. Further, the lyric constituent characters referred to here may be each of the characters constituting the lyric, or may be a phrase or a phrase grouped according to a specific rule along the time axis.

  The guide vocal data GD is voice data prepared in advance as an ideal singing voice for the music. The ideal singing voice referred to here is a voice that is considered to have been sung according to the musical sound that constitutes the music, and is a voice that becomes close to a perfect score when the karaoke apparatus is scored by a well-known scoring function.

In a general karaoke apparatus, an ideal singing voice based on the guide vocal data GD is reproduced together with a musical instrument performance based on the music MIDI data DM.
<Voice input device>
Next, the voice input device 10 will be described.

  The voice input device 10 includes a communication unit 11, an input receiving unit 12, a display unit 13, a voice input unit 14, a voice output unit 15, a sound source module 16, a storage unit 17, and a control unit 20. ing. That is, the voice input device 10 in the present embodiment is configured as a so-called well-known karaoke device.

  Among these, the communication part 11 communicates between the audio | voice input apparatuses 10 outside via a communication network. The communication network in this embodiment is, for example, a public wireless communication network or a network line.

  The input receiving unit 12 is an input device that receives input of information and commands in accordance with external operations. The input device in the present embodiment is, for example, a key, a switch, a remote control receiving unit, or the like.

  The display unit 13 is a display device that displays at least an image including information indicated by a character code. The display device in the present embodiment is, for example, a liquid crystal display or a CRT. The voice input unit 14 is a device (so-called microphone) that converts sound into an electrical signal and inputs the signal to the control unit 20. The audio output unit 15 is a device (so-called speaker) that converts an electrical signal from the control unit 20 into sound and outputs the sound.

Furthermore, the sound module 16 is a device that outputs a sound (that is, an output sound) that simulates a sound from the sound source based on the music MIDI data DM, for example, a MIDI sound source.
The storage unit 17 is a non-volatile storage device configured to be able to read and write stored contents. The storage device in the present embodiment is, for example, a hard disk device or a flash memory.

  The control unit 20 is stored in the ROM 21 that stores processing programs and data that need to retain stored contents even when the power is turned off, the RAM 22 that temporarily stores processing programs and data, and the ROM 21 and RAM 22. It is mainly configured by a known computer having at least a CPU 23 that executes each process (various operations) according to the processing program.

  The ROM 21 stores a processing program in which the control unit 20 executes a karaoke performance process for playing a music specified by the user (hereinafter referred to as a target music). In the karaoke performance processing according to the present embodiment, the music ID and the target music for identifying the target music are obtained by using the voice input via the voice input unit 14 as the voice waveform data SV during the period in which the target music is being played. The information is stored in the information storage server 25 in association with the user ID for identifying the sung user voice.

  That is, in the voice input device 10, according to the karaoke performance process, the music is played based on the music MIDI data DM corresponding to the target music, and the lyrics are displayed on the display unit 13 based on the lyrics data group DL corresponding to the target music. indicate. During execution of the karaoke performance process, the voice input via the voice input unit 14 is stored in the information storage server 25 as voice waveform data SV in association with the music ID and user ID.

  The speech waveform data SV stored in the information storage server 25 is also associated with speaker feature information representing the features of the speaker. This speaker feature information includes, for example, at least the gender of the user.

The data storage server 50 is a device mainly configured of a storage device configured to be able to read and write stored contents, and is connected to the information processing device 30 via a communication network.
<Information processing device>
The information processing apparatus 30 includes a communication unit 31, an input reception unit 32, a display unit 33, a storage unit 34, and a control unit 40.

  Among these, the communication part 31 communicates with the exterior via a communication network. The input receiving unit 32 is an input device that receives input of information and commands in accordance with external operations. The display unit 33 is a display device that displays an image.

  The storage unit 34 is a non-volatile storage device configured to be able to read and write stored contents. The control unit 40 is configured around a known computer having at least a ROM 41, a RAM 42, and a CPU 43.

  Furthermore, in the ROM 41 of the information processing apparatus 30, the audio waveform data SV in which the designated music ID and user ID are associated among the audio waveform data SV stored in the information storage server 25 is acquired, and parameters are acquired. A processing program in which the control unit 40 executes parameter registration processing for generating the data PM and the model singing voice data ED and storing it in the data storage server 50 is stored.

The parameter data PM and the model singing voice data ED stored in the data storage server 50 by this parameter registration process include metadata. The metadata in the present embodiment represents the nature of sound including emotion when the sound is uttered. In addition, the emotion referred to here includes, for example, emotion and atmosphere. Furthermore, information necessary for estimating emotions may be included in the nature of speech.
<Parameter registration process>
When the parameter registration process executed by the information processing apparatus 30 is started, first, as shown in FIG. 2, a user ID is acquired via the input receiving unit 32 (S110). That is, in S110, the user logs in to the information processing apparatus 30.

Subsequently, the music ID is acquired through the input receiving unit 32 (S120). That is, in S120, the user designates music.
Subsequently, parameter data registration processing for generating and registering the parameter data PM is executed based on the user ID acquired in S110 and the voice waveform data SV corresponding to the music ID acquired in S120 (S130). Furthermore, the model singing voice data registration process which produces | generates and registers the model singing voice data ED of the music corresponding to music ID acquired in S120 is performed (S140). Details of the parameter data registration process and the model singing voice data registration process will be described later.

Thereafter, the parameter registration process is terminated.
<Parameter data registration process>
As shown in FIG. 3, when the parameter data registration process is started, the music MIDI data DM corresponding to the music ID acquired in the previous S120 is acquired (S210). Subsequently, the lyrics data group DL corresponding to the song ID acquired in the previous S120 is acquired (S220), and one voice corresponding to the song ID and corresponding to the user ID specified in the previous S110 is obtained. Waveform data SV is acquired (S230).

  Further, in the speech waveform data SV acquired in S230, a speech waveform (hereinafter referred to as a syllable waveform) in a section corresponding to each syllable included in the utterance content of the speech waveform data SV is specified (S240).

  Specifically, in S240 of the present embodiment, the performance start timing and performance end of each output sound defined in a musical score track (hereinafter referred to as a melody track) representing the singing melody in the music MIDI data DM acquired in S210. The timing is extracted, and the syllables of the lyrics constituent characters associated with each output sound are specified. Then, in the speech waveform data SV, the speech waveform in the section corresponding to each section from the performance start timing to the performance end timing of each output sound is specified as a syllable waveform. Note that each syllable waveform specified in S240 of the present embodiment is associated with the content of the syllable uttered by the syllable waveform.

  Further, a speech parameter is derived from each syllable waveform (S250). The audio parameters derived in S250 of this embodiment include at least the fundamental frequency, the mel frequency cepstrum (MFCC), the power, and their time differences. Since these fundamental frequency, MFCC, and power derivation methods are well known, detailed description thereof is omitted here. For example, if the fundamental frequency is used, autocorrelation along the time axis of the syllable waveform, syllable waveform What is necessary is just to derive | lead-out using methods, such as an autocorrelation of a frequency spectrum, or a cepstrum method. In addition, in the case of MFCC, a result obtained by applying a time analysis window to a syllable waveform and performing frequency analysis (for example, FFT) for each time analysis window is further obtained by logarithmizing the size for each frequency. It can be derived by frequency analysis. The power may be derived by applying a time analysis window to the syllable waveform and integrating the square of the amplitude in the time direction.

Subsequently, a metadata estimation process for estimating metadata for each syllable waveform specified in S240 is executed (S260).
<Metadata estimation processing>
As shown in FIG. 4, when the metadata estimation process in the present embodiment is started, first, each section in which the same key is continued in the music based on the music MIDI data acquired in the previous S210. The same key interval is specified (S310). Specifically, in S310 of the present embodiment, as shown in FIG. 5, based on the modulation flag included in the music MIDI data, the section between the modulation flags adjacent to each other along the time axis is identified as the same key section. To do.

  Subsequently, the main sound in the same key interval specified in S310 is specified (S320). Specifically, in S320 of the present embodiment, as shown in FIG. 6, in the same key interval, the last output sound along the time axis is specified as the main sound in the same key interval. In the present embodiment, the main sound is specified for each of the same key intervals specified in S310.

  Then, a histogram (hereinafter referred to as a pitch) in which the pitch name of the main tone specified in S320 is a starting point, the pitch names of the output sounds included in the same key interval in which the main tone is specified is a rank, and the frequency of appearance of each pitch name is a frequency. , Referred to as frequency of appearance sound name) (S330). Specifically, as shown in FIG. 7A, the frequency of appearance sound names derived in S330 of the present embodiment is the total number of appearances (appearance frequencies) of output sounds of the same sound name included in the same key interval. It is a thing. And in this embodiment, even if it is an output sound from which an octave differs, if the pitch name is the same, it totals as an output sound of the same pitch name. In the present embodiment, the appearance name frequency is derived for the same section.

  Then, based on the result of collating the appearance note name frequency derived in S330 with a key template prepared in advance for each key as a template representing the distribution of pitch names that can be used in each key, A key is specified (S340). Specifically, in S340 of the present embodiment, a major template (see FIG. 7B) that represents the distribution of pitch names that can be used in major music, and the distribution of pitch names that can be used in minor music. A minor template to be expressed (see FIG. 7C) is prepared in advance, and the frequency of appearance note names derived in S330 is collated with each key template. As a result, the key corresponding to the key template showing the highest correlation is specified as the key in the key same section. In S340 of the present embodiment, the key for each key-same section is specified.

  Furthermore, the sound property corresponding to the key of the music in the same key section identified in S340 is specified as metadata (S350). Specifically, in S350 of the present embodiment, if the key in the same key section is a major key, the property of the voice representing the emotion that the lyrics (that is, the utterance content) in the same key section is “bright” is used as metadata. Identify. If the key in the same key section is a minor key, the property of the voice representing the feeling that the lyrics in the same key section are “dark” is specified as metadata. In the present embodiment, metadata corresponding to the same key interval is assigned to all syllables included in the same key interval.

  Thereafter, the process returns to S270 of the parameter data registration process, and the parameter data PM is generated and associated with the data storage server 50 by associating the speech parameter derived in S250 with the metadata estimated in S260 for each corresponding syllable. The parameter data to be stored is registered (S270). Note that the parameter data PM stored in the data storage server 50 in S270 of the present embodiment includes the content (type) of the uttered syllable, the user ID, and speaker characteristic information in addition to the audio parameters and metadata. Including.

Thereafter, the parameter data registration process is terminated.
As described above, as shown in FIG. 8, in the parameter data registration process of the present embodiment, the speech waveform input during the performance period of the music is processed. Then, the speech waveform data SV based on the speech waveform is divided into sections corresponding to the performance periods of the output sounds constituting the melody line of the music (that is, each syllable included in the utterance content), and the syllable waveform is divided. At the same time, a speech parameter is derived from each syllable waveform.

  At the same time, in the parameter data registration process, each period (that is, the same key section) in which the same key continues in the music is specified, and the key (tonality) in each key same section is specified. And the property of the voice previously defined as the emotion imaged from the specified key is specified as metadata.

  In addition, in the parameter data registration process, for each corresponding syllable, the parameter data PM is generated by associating the voice parameter with the metadata and stored in the data storage server 50. That is, as shown in FIG. 9A, the parameter data PM stored in the data storage server 50 is the voice associated with the metadata in the order of appearance of syllables along the time axis of the lyrics constituent characters in the music. Includes parameters.

In addition, the parameter data registration process in this embodiment is implemented for every user and every music. Therefore, the parameter data PM is stored in the data storage server as different parameter data PM every time the song is sung even for the same user and the same song.
<Model singing voice data registration process>
As shown in FIG. 10, the model singing voice data registration process, when activated, acquires music MIDI data DM corresponding to the music ID acquired in the previous S120 (S810). Subsequently, the lyrics data group DL corresponding to the music ID acquired in the previous S120 is acquired (S820). Further, guide vocal data GD corresponding to the music ID is acquired (S830).

  In the acquired guide vocal data GD, the speech waveform in the section corresponding to each syllable included in the utterance content of the guide vocal data GD is specified (S840). S840 in the present embodiment may be performed in the same manner as S240 in the parameter data registration process.

Further, speech parameters are derived from each of the speech waveforms identified in S840 (S850). Subsequently, the metadata estimation process described above is executed (S860).
Thereafter, the data associated with the audio parameter derived in S850 and the metadata estimated in S860 is generated as model singing voice data ED, and model singing voice data registration to be stored in the data storage server 50 is executed (S870). The model singing voice data ED generated in S870 of the present embodiment includes the content (type) of the uttered syllable and the song ID in addition to the voice parameters and the metadata.

Then, this model singing voice data registration process is complete | finished.
That is, the exemplary singing voice data ED stored in the data storage server 50 is associated with the metadata in the order of appearance of syllables along the time axis of the lyrics constituent characters in the music, as shown in FIG. 9B. Includes voice parameters.

Normally, the metadata of the model singing voice data ED (FIG. 9 ( B )) and the singer's metadata (FIG. 9 ( A )) are the same song, and if the tune matches, the type of metadata ( (Bright and dark) often coincide.

If a singer sings a major song, the metadata is likely to be “bright”, and if a singer sings a minor song, the metadata is likely to be “dark”.
However, the tone of the song sung by the user may be different from the tone of the song that the user wants to listen to as a model voice.

  For this reason, in order to listen at an appropriate pitch, prepare two types of guide vocals, major (bright) and minor (dark), and create two types of metadata and audio parameters. Good.

  The CPU 23 specifies the guide vocal type associated with the music ID, and performs parameter registration processing for each type of guide vocal, thereby registering parameter data according to the number of types of guide vocals.

  The exemplary singing voice data registration process in the present embodiment is performed for each piece of music. Therefore, one model singing voice data ED is generated for one piece of music and stored in the data storage server 50.

  In the present embodiment, the metadata and sound parameters of the guide vocal data GD are created by playing the music data MD at least once (two times if the guide vocal is of the major and minor types) on the karaoke device side. doing.

If the voice data of the lyrics (guide vocal) and the guide voice data of the guide vocal are also incorporated in the music data MD in advance in association with the output timing of the lyrics synchronized with the progress of the performance, the guide vocal example The voice data registration process may not be performed.
<Audio output terminal>
As shown in FIG. 1, the audio output terminal 60 includes an information receiving unit 61, a display unit 62, a sound output unit 63, a communication unit 64, a storage unit 65, and a control unit 67. As the audio output terminal 60 in the present embodiment, for example, a known portable terminal such as a mobile phone or a portable information terminal, or a known information processing apparatus such as a so-called personal computer may be assumed.

  Among these, the information reception part 61 receives the information input via the input device (not shown). The display unit 62 displays an image based on a command from the control unit 67. The sound output unit 63 is a known device that outputs sound, and includes, for example, a PCM sound source and a speaker.

  The communication unit 64 is for the voice output terminal 60 to perform information communication with the outside via a communication network. The storage unit 65 is a non-volatile storage device configured to be able to read and write stored contents. The storage unit 65 stores various processing programs and various data.

The control unit 67 is mainly configured by a known computer having at least a ROM, a RAM, and a CPU.
Based on the parameter data PM and the model singing voice data ED, the ROM of the control unit 67 has a voice synthesis process for outputting a synthesized voice that is voice-synthesized so as to realize an ideal singing voice for the music by the user's own voice. A processing program to be executed by the control unit 67 is stored.
<About voice synthesis processing>
The voice synthesis process is started when a start command is input via the information receiving unit 61 of the voice output terminal 60.

  In the speech synthesis process, as shown in FIG. 11, the user ID is acquired via the information receiving unit 61 (S510). Subsequently, the music ID is acquired through the information receiving unit 61 (S520).

  And the model singing voice data ED corresponding to the music ID acquired in S520 is acquired (S530). Further, among the parameter data PM associated with the user ID acquired in S510, the parameter data PM most similar to the voice parameter included in the exemplary singing voice data ED acquired in S530 is acquired (S540). In S540, for example, the condition for determining “most similar” is that the calculation result of the correlation value derived for each feature amount as a speech parameter is derived for each parameter data PM, and the derived calculation is performed. The parameter data PM including the speech parameter corresponding to the largest calculation result among the results may be set as “most similar”.

The metadata plural kinds (bright, dark) exists, i.e., if the model voice data ED corresponding to the music ID there are a plurality may be selected model voice data ED metadata match. By adopting the matching audio parameter on the metadata side, further optimal audio synthesis is performed.

  Subsequently, the voice parameter included in the parameter data PM acquired in S540 is adjusted to match the voice parameter included in the exemplary singing voice data ED acquired in S530 (S550). Based on the adjusted speech parameter, speech synthesis is performed (S560). For the speech synthesis in S560, a known speech synthesis technique using formant synthesis may be used. If the song is a duet song, the gender of the user may be specified from the speaker characteristic information, and the lyrics portion that matches the gender part of the duet song may be synthesized.

Furthermore, the synthesized sound generated by the voice synthesis in S560 is output from the sound output unit 63 (S570).
In the present embodiment, the steps from S530 to S570 are sequentially executed along the time axis of the music. Specifically, based on the music MIDI data DM of the music, voice synthesis is executed so that a synthesized sound for each syllable is output at the output timing of each musical sound.

In addition, S530 to S570 are repeated until the performance of the music corresponding to the music ID acquired in S520 is completed (S580: NO). Then, when the performance of the music is finished (S580: YES), the speech synthesis process is finished.
[Effect of the first embodiment]
As described above, according to the speech synthesis system 1, after the user sings one piece of music, the user voice in the model singing state can be heard. That is, according to the speech synthesis system 1, speech synthesis using the user's own voice is performed using speech parameters derived from the user's singing voice so that speech is output at the utterance start timing and utterance pitch of the music. it can.

  Moreover, in the speech synthesis system 1, after generating the model singing voice data ED for one piece of music, the user side only sings the corresponding piece of music data MD, and the user sings the model singing voice data every time. There is no need to generate an ED.

  From the above, according to the speech synthesis system 1, in addition to the period in which the user himself sings the music, the user's own voice is also included in the user's own voice and various music including the music is appropriate. It is possible to listen at a high pitch.

In addition, in the metadata estimation process of this embodiment, the emotion of a singer who is highly likely to be represented by the key of each key in the target music is used as metadata. That is, according to the metadata estimation process of the present embodiment, it is possible to use the emotion of the speaker when the lyrics corresponding to the same key interval are uttered as metadata, and to ensure the key in the same key interval. Can be specified.
[Second Embodiment]
The speech synthesis system of the second embodiment mainly differs from the speech synthesis system 1 of the first embodiment in the processing content of the metadata estimation process. For this reason, in the present embodiment, the same configurations and processes as those in the first embodiment are denoted by the same reference numerals, description thereof will be omitted, and description will be made focusing on metadata estimation processing different from that in the first embodiment. .
<Metadata estimation processing>
As shown in FIG. 12, when the metadata estimation process of the present embodiment is started in S260 of the parameter data registration process, the lyrics telop data DT included in the lyrics data group DL acquired in the previous S210. The morphological analysis is performed on the lyrics represented by (S710). That is, in S710 of this embodiment, the character string which comprises a lyrics is divided | segmented for every word character which is the character string which comprises the word in the said lyrics by performing a morphological analysis. Note that the morphological analysis performed in S710 is a well-known process, and a detailed description thereof is omitted here.

  Subsequently, word property information is acquired for each word that is the result of the morphological analysis of S710 from the word metadata database (DB in the figure) 100 in which a word property table prepared in advance is stored (S720). However, the word property table referred to here is a table in which word property information representing the property of each word is associated with identification information of the word. Further, the word property mentioned here includes the meaning of the word and the emotion represented by the word.

Then, the word property information acquired in S720 is assigned as metadata to the section where the word is uttered (S730).
Thereafter, the metadata estimation process is terminated, and the process returns to the parameter data registration process.

As described above, in the metadata estimation process of the present embodiment, as shown in FIG. 13, morphological analysis is performed on the lyrics of the target music, and the lyrics of the target music are character strings constituting words. Divide by word character. In addition, word property information corresponding to each word is acquired from the word property information included in the word property table stored in the word metadata database 100 prepared in advance, and each of the word property information is obtained. , Metadata for the speech parameters of the corresponding syllable.
[Effects of Second Embodiment]
As described above, according to the metadata estimation process of the present embodiment, the meaning of the word uttered by the speaker and the emotion represented by the word can be used as metadata.
[Other Embodiments]
As mentioned above, although embodiment of this invention was described, this invention is not limited to the said embodiment, In the range which does not deviate from the summary of this invention, it is possible to implement in various aspects.

  For example, in the speech synthesis system 1, the information storage server 25 is provided. However, in the speech synthesis system of the present invention, the information storage server 25 may not be provided. In this case, the music data MD and the voice waveform data SV may be stored in the storage unit 17 of the voice input device 10, may be stored in the data storage server 50, and further stored in the information processing device 30. It may be stored in the unit 34.

  Similarly, although the data storage server 50 is provided in the speech synthesis system 1, the data storage server 50 may not be provided in the speech synthesis system 1. In this case, the parameter data PM and the model singing voice data ED may be stored in the storage unit 34 of the information processing device 30, may be stored in the storage unit 17 of the voice input device 10, and further store information. It may be stored in the server 25.

In addition, although the model singing voice data ED in the said embodiment was produced | generated by performing a model singing voice data registration process, the model singing voice data ED may be previously integrated in the music data MD. In this case, ideal voice waveform data SVr corresponding to the music ID and data including voice parameters based on the voice waveform data SVr may be incorporated as the model singing voice data ED. When incorporated into the music data MD, of course, it includes a model voice data E D, may be stored to one of the music data MD.

The speech synthesis system 1 includes the speech input device 10, the information processing device 30, and the speech output terminal 60, but these speech input device 10, information processing device 30, and speech output terminal are provided separately. 60 may be a common device. That is, the voice input device 10 (that is, the karaoke device) may also serve as at least one of the information processing device 30 and the voice output terminal 60.
[Correspondence between Embodiment and Claims]
Finally, the relationship between the description of the above embodiment and the description of the scope of claims will be described.

  The karaoke performance process in the above embodiment corresponds to the utterance information acquisition means in the present invention, S230 in the parameter data registration process corresponds to the waveform acquisition means in the present invention, and S250 corresponds to the parameter derivation means in the present invention. , S270 corresponds to the parameter registration means in the present invention, and S140 in the parameter registration processing corresponds to the model storage means in the present invention.

  Further, S510 and S520 in the speech synthesis process in the above embodiment correspond to the identification information acquisition means in the present invention, S540 corresponds to the parameter acquisition means in the present invention, and S530 corresponds to the model acquisition means in the present invention. To do. S550 and S560 in the speech synthesis process correspond to speech synthesis means in the present invention, and S570 corresponds to output means in the present invention.

  Further, S310 in the metadata estimation process of the first embodiment corresponds to the section specifying means in the description of the claims, S320 corresponds to the main sound specifying means in the description of the claims, and S330, It corresponds to the pitch name deriving means in the description of the claims, and S340 and S350 correspond to the key estimating means. Further, S710 in the metadata estimation process of the second embodiment corresponds to the word dividing means in the claims, and S720 and S720 correspond to the metadata extracting means.

  DESCRIPTION OF SYMBOLS 1 ... Voice synthesis system 10 ... Voice input device 11 ... Communication part 12 ... Input reception part 13 ... Display part 14 ... Voice input part 15 ... Voice output part 16 ... Sound source module 17 ... Memory | storage part 20 ... Control part 21,41 ... ROM 22, 42 ... RAM 23, 43 ... CPU 25 ... MIDI storage server 30 ... Information processing device 31 ... Communication unit 32 ... Input reception unit 33 ... Display unit 34 ... Storage unit 40 ... Control unit 50 ... Data storage server 60 ... Audio output Terminal 61 ... Information receiving unit 62 ... Display unit 63 ... Sound output unit 64 ... Communication unit 65 ... Storage unit 67 ... Control unit 100 ... Word metadata database

Claims (5)

A song ID for identifying a song, performance information representing a musical sound constituting the song identified by the song ID, lyric information representing the lyrics of the song identified by the song ID, and utterance start of the lyrics represented by the lyrics information Utterance information acquisition means for acquiring the tune ID, the performance information, the lyric information, and the utterance timing information from tune data including utterance timing information indicating timing, and reproducing the tune based on the tune data;
Waveform acquisition means for acquiring an input speech waveform at the utterance start timing of the lyrics during playback of the music based on the music data;
A parameter derivation that extracts a syllable waveform that is a speech waveform for each syllable forming the lyrics from the speech waveform and derives a speech parameter that is at least one feature amount defined in advance from the extracted syllable waveform. Means,
Parameter registration means for storing the speech parameters for each syllable derived by the parameter derivation means in association with a user ID for identifying a user who sang the music in a first storage device;
Model storage means for storing model voice data including at least one voice parameter based on an ideal singing voice for each song in association with the song ID in the second storage device ;
A parameter extracting device having
Identification information acquisition means for acquiring the designated music ID and user ID ;
A model acquiring means for acquiring a model voice data associated with the music ID obtained in the previous SL identification information acquiring means from said second storage device,
Among the voice parameters stored in the first storage device, the voice parameters are associated with the user ID acquired by the identification information acquisition unit, and are included in the model voice data acquired by the model acquisition unit A parameter acquisition means for acquiring an audio parameter most similar to the audio parameter associated with the music ID to be recorded;
A speech synthesizer that adjusts the speech parameter acquired by the parameter acquisition unit to match the speech parameter associated with the music ID, and synthesizes speech based on the adjusted speech parameter ;
Output means for outputting synthesized sound generated by voice synthesis by the voice synthesis means ;
And a synthesized sound output device having
The utterance information acquisition means includes
Representing the score of the target music that is one of the music, for each output sound that is output from the sound source module, to obtain at least the musical score data that defines the pitch and performance start timing as the performance information,
The musical score data is
If it is in the song of the target song, including a modulation flag indicating the time at which the target song was modulated along the time axis,
The parameter registration means includes
Based on the acquired musical score data, section specifying means for specifying the same key section, which is each section in which the same key is continued in the target music,
A main sound specifying means for specifying the last output sound along the time axis in each key same section as the main sound included in each key same section specified by the section specifying means;
An appearance pitch name frequency representing a frequency of output sounds of the same pitch name included in the same pitch section specified by the section specifying means is set for each key same section starting from the pitch name of the main sound specified by the main sound specifying means. A pitch name deriving means for deriving;
As a result of comparing each appearance name name frequency derived by the above-mentioned sound name frequency deriving means against a key template prepared in advance for each key as a template representing the distribution of pitch names that can be used in each key, Key estimation means for estimating as metadata,
With
The speech synthesis system , wherein the metadata estimated by the key estimation means is stored in the first storage device in association with the speech parameter . The utterance information acquisition means includes
A string of lyric constituent characters constituting the lyrics of the previous SL object music pieces, and obtained as the lyric information,
The output timing for at least one of the lyrics constituent characters is acquired as the utterance timing information, the lyrics output timing associated with the performance start timing of the output sound corresponding to the lyrics constituent characters,
The waveform acquisition means includes
A waveform in which the voice input during the performance of the target music based on the score data has shifted along the time axis is acquired as the voice waveform,
The parameter derivation means includes
The speech synthesis system according to claim 1, wherein in the speech waveform, a speech waveform in a section corresponding to each output sound is extracted as the syllable waveform.   A song ID for identifying a song, performance information representing a musical sound constituting the song identified by the song ID, lyric information representing the lyrics of the song identified by the song ID, and utterance start of the lyrics represented by the lyrics information Utterance information acquisition means for acquiring the tune ID, the performance information, the lyric information, and the utterance timing information from tune data including utterance timing information indicating timing, and reproducing the tune based on the tune data;
  Waveform acquisition means for acquiring an input speech waveform at the utterance start timing of the lyrics during playback of the music based on the music data;
  A parameter derivation that extracts a syllable waveform that is a speech waveform for each syllable forming the lyrics from the speech waveform and derives a speech parameter that is at least one feature amount defined in advance from the extracted syllable waveform. Means,
  A parameter registration means for storing the speech parameter for each syllable derived by the parameter derivation means in association with a user ID for identifying a user who sang the song in a first storage device;
  Model storage means for storing model voice data including at least one voice parameter based on an ideal singing voice for each song in association with the song ID in the second storage device;
  A parameter extracting device having
  Identification information acquisition means for acquiring the designated music ID and user ID;
  Model acquisition means for acquiring model singing voice data associated with the music ID acquired by the identification information acquisition means from the second storage device;
  Among the voice parameters stored in the first storage device, the voice parameters are associated with the user ID acquired by the identification information acquisition unit, and are included in the model voice data acquired by the model acquisition unit A parameter acquisition means for acquiring an audio parameter most similar to the audio parameter associated with the music ID to be recorded;
  A speech synthesizer that adjusts the speech parameter acquired by the parameter acquisition unit to match the speech parameter associated with the music ID, and synthesizes speech based on the adjusted speech parameter;
  Output means for outputting synthesized sound generated by voice synthesis by the voice synthesis means;
  A synthesized sound output device having
  With
  The parameter registration means includes
  A word dividing unit that divides the character string represented by the lyrics information acquired by the utterance information acquiring unit for each word character that is a character string constituting the word;
  A word property table in which property information representing the property of each word is associated with identification information of the word is prepared in advance, and the property information corresponding to the word represented by each word character divided by the word dividing unit is Metadata extraction means for extracting from the word property table as metadata
  With
  A speech synthesis system, wherein the metadata extracted by the metadata extraction means is stored in the first storage device in association with the speech parameter.   A song ID for identifying a song, performance information representing a musical sound constituting the song identified by the song ID, lyric information representing the lyrics of the song identified by the song ID, and utterance start of the lyrics represented by the lyrics information Utterance information acquisition means for acquiring the tune ID, the performance information, the lyric information, and the utterance timing information from tune data including utterance timing information indicating timing, and reproducing the tune based on the tune data;
  Waveform acquisition means for acquiring an input speech waveform at the utterance start timing of the lyrics during playback of the music based on the music data;
  A parameter derivation that extracts a syllable waveform that is a speech waveform for each syllable forming the lyrics from the speech waveform and derives a speech parameter that is at least one feature amount defined in advance from the extracted syllable waveform. Means,
  A parameter registration means for storing the speech parameter for each syllable derived by the parameter derivation means in association with a user ID for identifying a user who sang the song in a first storage device;
  Model storage means for storing model voice data including at least one voice parameter based on an ideal singing voice for each song in association with the song ID in the second storage device;
  With
  The utterance information acquisition means includes
  Representing the score of the target music that is one of the music, for each output sound that is output from the sound source module, to obtain at least the musical score data that defines the pitch and performance start timing as the performance information,
  The musical score data is
  If it is in the song of the target song, including a modulation flag indicating the time at which the target song was modulated along the time axis,
  The parameter registration means includes
  Based on the acquired musical score data, section specifying means for specifying the same key section, which is each section in which the same key is continued in the target music,
  A main sound specifying means for specifying the last output sound along the time axis in each key same section as the main sound included in each key same section specified by the section specifying means;
  An appearance pitch name frequency representing a frequency of output sounds of the same pitch name included in the same pitch section specified by the section specifying means is set for each key same section starting from the pitch name of the main sound specified by the main sound specifying means. A pitch name deriving means for deriving;
  As a result of comparing each appearance name name frequency derived by the above-mentioned sound name frequency deriving means against a key template prepared in advance for each key as a template representing the distribution of pitch names that can be used in each key, Key estimation means for estimating as metadata,
  With
  The parameter extraction device characterized in that the metadata estimated by the key estimation means is stored in the first storage device in association with the voice parameter. A song ID for identifying a song, performance information representing a musical sound constituting the song identified by the song ID, lyric information representing the lyrics of the song identified by the song ID, and utterance start of the lyrics represented by the lyrics information Utterance information acquisition means for acquiring the tune ID, the performance information, the lyric information, and the utterance timing information from tune data including utterance timing information indicating timing, and reproducing the tune based on the tune data;
Waveform acquisition means for acquiring an input speech waveform at the utterance start timing of the lyrics during playback of the music based on the music data;
A parameter derivation that extracts a syllable waveform that is a speech waveform for each syllable forming the lyrics from the speech waveform and derives a speech parameter that is at least one feature amount defined in advance from the extracted syllable waveform. Means,
A parameter registration means for storing the speech parameter for each syllable derived by the parameter derivation means in association with a user ID for identifying a user who sang the song in a first storage device;
Model storage means for storing model voice data including at least one voice parameter based on an ideal singing voice for each song in association with the song ID in the second storage device;
With
The parameter registration means includes
A word dividing unit that divides the character string represented by the lyrics information acquired by the utterance information acquiring unit for each word character that is a character string constituting the word;
A word property table in which property information representing the property of each word is associated with identification information of the word is prepared in advance, and the property information corresponding to the word represented by each word character divided by the word dividing unit is Metadata extraction means for extracting from the word property table as metadata
With
The parameter extraction device characterized in that the metadata extracted by the metadata extraction means is stored in the first storage device in association with the voice parameter .