JP4487909B2 - Voice control device and voice control method - Google Patents

Description

  The present invention relates to a sound control device and a sound control method, and more particularly to a sound control device and a sound control method for performing effect processing on a sound signal output from a microphone.

In recent years, systems that add an echo or reverb effect to musical sounds generated from electronic musical instruments by combining electronic musical instruments and devices such as effectors have become widespread. In addition, there have been proposals for combining electronic musical instruments and video cameras to give various variations to musical sounds generated from electronic musical instruments. For example, there is a proposal of a music performance device that gives an expression effect to music to be played based on a player's facial expression. In this proposal, based on the image data from the photographing means for photographing the performer's face image, the sound is determined according to the shape parameter extracted from the attention portion of the face image, for example, the shape of the performer's mouth. A control signal for controlling the expression effect on the data is generated. Specifically, a control signal is generated according to the degree of vertical opening of the opening of the performer's mouth, and the cut-off frequency of the low-pass filter through which the sound passes is changed according to the control signal. For example, when the mouth is opened wide, the cutoff frequency of the low-pass filter is increased, and as the mouth is closed, the cutoff frequency of the low-pass filter is decreased. Alternatively, a control signal is generated according to the width of the opening of the performer's mouth, and the nonlinearity of the amplifier through which the sound passes is controlled according to the control signal. For example, if the width of the mouth widens, distortion is applied by changing the nonlinearity of the amplifier and clipping the sound amplitude. As a musical instrument to be played, in addition to the electric guitar described in the embodiment, a piano or a synthesizer is assumed. Furthermore, it is described that the expression effect on the sound data is controlled not only by the performer but also by the facial expression of a DJ (disc jocker). (See Patent Document 1)
Japanese Patent Laid-Open No. 2002-140066

  However, as in the above-mentioned patent document, controlling the expression effect on the music to be played based on the player's facial expression may cause loss or destruction of musicality. This is because it cannot be considered that there is a correlation between the performer's facial expression and the generated musical sound. For example, even if the performer is playing with a bright tone while laughing, if the sound is distorted and the tone is rough due to an image with a wide mouth, the effect will be the opposite of the intention of the performer. End up. Also, some performers may open their mouths with a melody, even if they are playing a quiet, melancholic song. In such a case, if the sound is distorted according to the movement of the mouth to create a rough tone, the performance will be spoiled.

There is a correlation between the expression of the performer's mouth and the generated musical sound only when singing. No one sings quietly, saying that “... is wet with night dew ...” and singing loudly, “... Came Ze Hakodate ...” without opening his mouth. There are no people. That is, there is a high correlation between the movement of the singer's mouth and the timbre of the singing voice, and if this correlation is used, it can be expected to improve the singing ability.
For example, karaoke, which is widely used, is used as an event for relatives, banquets, entertainment, etc. in organizations such as companies rather than competing for singing ability. However, it is embarrassing and painful to sing in front of everyone for those who have no singing ability. Therefore, if you can sing with higher singing ability than you can, you can relieve shame and pain, and the atmosphere will be better for those around you.
The present invention is for solving such a conventional problem, and by utilizing the correlation between the movement of the singer's mouth and the timbre of the singing voice, the singing ability can be enhanced more than actual. The purpose is to do.

  The voice control device according to the first aspect is a signal generating means that is accommodated in a microphone and outputs a voice signal in accordance with an inputted voice (in the embodiment, it corresponds to the microphone unit 5 in FIGS. 2 and 3). And an image pickup means (corresponding to the camera unit 6 in FIGS. 2 and 3 in the embodiment) which is housed in the microphone and picks up an image of the mouth of the singer and outputs an image signal, and is output from the signal generation means. Correlation detection means (corresponding to the CPU 1 in FIG. 3 in the embodiment) for detecting the correlation between the sound signal thus generated and the image signal output by the imaging means, and the correlation detected by the correlation detection means Signal processing means for effecting the audio signal output from the signal generating means in accordance with the sex data (corresponding to the CPU 10 and the DSP control section 18 in FIG. 3 in the embodiment). It has a configuration which includes a, and.

  2. The audio control apparatus according to claim 1, wherein the correlation detecting unit detects a difference between the image signal output from the imaging unit and the audio signal output from the signal generating unit. The signal processing means includes a detection means (corresponding to the CPU 10 in FIG. 3 in the embodiment), and the signal processing means outputs an audio signal output from the signal generation means based on the difference relation data detected by the difference detection means. You may make it a structure which performs an effect process.

In the voice control device according to claim 1 or 2, as described in claim 3, the signal processing means may be accommodated in a microphone.
In the audio control apparatus according to claim 1 or 2, as described in claim 4, the signal processing means may be configured to control a feedback component of the reverberation process.
In the voice control device according to claim 1 or 2, as described in claim 5, the signal processing means may be configured to control a reverb time of the reverb process.
In the voice control device according to claim 1 or 2, as described in claim 6, the signal processing means may be configured to control the feedback component of the delay processing.
In the voice control device according to claim 1 or 2, as described in claim 7, the signal processing means may be configured to control the delay time of the delay processing.

The voice control method according to claim 8 includes a step A for detecting a voice signal output from a microphone in accordance with an input voice, and an imaging means for picking up an image of a singer's mouth contained in the microphone. In the embodiment, step B for detecting an image signal output from the camera unit 6 in FIGS. 2 and 3), the audio signal detected in step A, and the image signal detected in step B Step C for detecting the correlation and Step D for performing the effect processing on the audio signal output from the microphone in accordance with the correlation data detected in Step C are executed.
Step A to step D correspond to the processing of the CPU 10 in FIG. 3 in the embodiment.

  9. The audio control method according to claim 8, wherein, as described in claim 9, step C includes step E for detecting a difference relationship between the audio signal detected at step A and the image signal output at step B. Then, step D may be configured such that effect processing is performed on the audio signal output from the microphone based on the difference-related data detected in step E.

In the audio control method according to claim 8 or 9, as described in claim 10, step D may be configured such that the effect processing is performed by the signal processing means accommodated in the microphone.
In the voice control method according to claim 8 or claim 9, as described in claim 11, step D may be configured to control the feedback component of the reverb process.
In the voice control method according to claim 8 or 9, as described in claim 12, step D may be configured to control the reverb time of the reverb process.
In the voice control method according to claim 8 or claim 9, as described in claim 13, step D may be configured to control a feedback component of delay processing.
In the voice control method according to claim 8 or claim 9, as described in claim 14, step D may be configured to control the delay time of the delay processing.

  According to the voice control device and the voice control method of the present invention, the effect that the singing ability can be enhanced more than the actual one is obtained by utilizing the correlation between the movement of the singer's mouth and the tone of the singing voice.

Hereinafter, a first embodiment and a second embodiment of a voice control device and a voice control method according to the present invention will be described in detail with reference to the drawings.
FIG. 1 is an external view of a karaoke microphone 1 common to each embodiment, and a microphone cover 2 in which a mesh shape or a plurality of holes are formed is attached. FIG. 2 is a diagram showing an internal structure of the microphone 1. The microphone cover 2 is provided with a transparent protective cover 3 made of a resin such as polycarbonate or acrylic by bonding or other methods. A substrate 4 is attached inside the microphone 1 behind the microphone cover 2. The board 4 is equipped with a microphone section 5 that converts sound input through the microphone cover 2 into an electrical signal and outputs the sound signal, and a camera section 6 that captures an image of the singer's mouth that can be seen through the transparent protective cover 3. Has been. Although not shown in the figure, the microphone unit 5 includes an amplifier circuit for amplifying an audio signal. The camera unit 6 includes an image pickup device such as a CCD or CMOS, a drive circuit, an amplifier circuit, an A / D conversion circuit, and the like. An analog audio signal from the microphone unit 5 is output from the microphone 1 through the lead wire 7, and a digital image signal from the camera unit 6 is output from the microphone 1 through the lead wire 8.

  FIG. 3 is a block diagram showing the configuration of the karaoke system in the first embodiment using the voice control device according to the present invention. In FIG. 3, a CPU 10 is connected to a program ROM 12, a work RAM 13, an operation switch 14, a display unit 15, a sound source 16, a music data ROM 17, a DSP (Digital Signal Processor) unit 18 and a DSP (Digital Signal Processor) unit 18 via a system bus 11. It is connected to the camera unit 6. The CPU 10 controls the entire karaoke system by exchanging data and commands with the above-described units connected via the system bus 11.

  The program ROM 12 stores a voice control processing program executed by the CPU 10 and initial data in advance. Also, a general mouth shape pattern is stored. The work RAM 13 is a work area that temporarily stores data processed by the CPU 10, and is provided with various registers and flags. The switch unit 14 includes a group of switches such as a song selection key, a song start key, and a song stop key, and inputs commands and data corresponding to operations to the CPU 10. The display unit 15 displays a list of karaoke songs and lyrics. The sound source 16 has a built-in waveform ROM for storing PCM waveform data and the like, and generates a digital musical tone signal in accordance with the sound generation command of the CPU 10. The song data ROM 17 stores musical tone data and lyrics data of karaoke accompaniment.

  On the other hand, the A / D conversion circuit 20 receives the audio signal from the microphone unit 5 shown in FIG. 2, converts the audio signal from analog to digital, and inputs it to the DSP unit 18. The sound source 16 generates an accompaniment musical tone signal based on the waveform data read from the internal waveform ROM in accordance with the karaoke accompaniment read and input from the song data ROM 17 by the CPU 10 to the DSP unit 18. input. The DSP unit 18 performs signal processing on the audio signal input from the microphone unit 5 via the A / D conversion circuit 20 based on the coefficient from the CPU 10 and synthesizes the audio signal and the musical tone signal of the accompaniment. And input to the D / A conversion circuit 21. The D / A conversion circuit 21 converts the composite signal input from the DSP unit 18 from digital to analog, and inputs it to the power amplifier 22 to generate sound from the speaker 23.

  FIG. 4 is a block diagram showing an internal configuration of the DSP unit 18 in the first embodiment. In FIG. 4, the effector 181 performs signal processing on the audio signal input from the microphone unit 5 based on the coefficient input from the CPU 10, and inputs the signal processing unit 182. The signal synthesis unit 182 synthesizes the audio signal input from the effector 181 and the musical tone signal of the accompaniment input from the sound source 16 in FIG. 3 and inputs the synthesized signal to the D / A conversion circuit 21 in FIG. The effector 181 includes a delay unit 183 and a feedback unit 184. The delay unit 183 performs a delay process corresponding to the delay coefficient input from the CPU 10 on the audio signal input from the microphone unit 5 and outputs the result. The feedback unit 184 feeds back the delayed audio signal to the input side of the delay unit 183 in accordance with the feedback coefficient input from the CPU 10. The effect processing in this case is the most popular plate type reverb, and the reverberation time is 4 seconds which is normally used. The pre-delay is fixed at 10 ms for producing a natural feeling.

Next, the voice control processing method of the first embodiment will be described based on the flowchart of the CPU 10 shown in FIGS. 5 to 9 and other drawings.
FIG. 5 is a flowchart of a main routine of the CPU 10 common to the embodiments. First, after predetermined initialization (step SA1), music selection processing is performed in accordance with the music selection operation (step SA2), and it is determined whether or not the music start key is turned on (step SA3). When the music start key is turned on, a timer is started (step SA4), and a variable N indicating the number of opening / closing of the mouth is set to 0 (step SA5). Next, the selected music data is read from the music data ROM 17 (step SA6) and sent to the sound source 16 (step SA7). Next, image recognition processing is executed based on the image signal from the camera unit 6 (step SA8), and DSP control processing is executed (step SA9). After this, it is determined whether or not the music is finished or the stop key of the switch unit 14 is turned on (step SA10). If the music is not finished and the stop key is not on, the process proceeds to step SA6. And repeat the reading of the song data. If the song end or stop key is on in step SA10, the process proceeds to step SA2 to select the next song in accordance with the operation of the switch unit 14.

  6 and 7 are flowcharts of the image recognition process in the main routine. In FIG. 6, it is determined whether or not it is the start time of the music, that is, whether or not it is immediately after the initialization of the main routine (step SA1) (step SB1). The first mouth recognition process is executed (step SB2). FIG. 8 is a flowchart of the mouth recognition process. An image is captured from the camera unit 6 (step SC1) and converted into a monochrome image (step SC2). Next, it is matched with a mouth shape pattern stored in advance in the program ROM 12 (step SC3), and it is determined whether or not the singer's mouth is recognized (step SC4). If it cannot be recognized, an image from the camera unit 6 is captured again in step SC1. When the mouth of the singer is recognized in step SC4, the coordinates a1 to a4 of both ends of the mouth and the upper and lower four points are stored in the registers (a1 to a4) of the work RAM 13 (step SC5). FIG. 10 is a diagram showing coordinates a1 and a2 at both ends of the mouth and upper and lower coordinates a3 and a4 of the mouth. The opening degree of the mouth can be detected by the coordinates a1 to a4.

  After storing the coordinates a1 to a4 in step SC5 of FIG. 8, the process returns to the flowchart of FIG. 6 and proceeds to step SB3. In step SB3, the stored coordinates a1 to a4 are stored (copied) in the registers Fa1 to Fa4. Next, the mouth size D is calculated based on a1 to a4 (step SB4). That is, in FIG. 10, the mouth size D is calculated based on the widths of both ends of the mouth (a2-a1) and the vertical distance (a4-a3) of the mouth. Further, the calculated D is stored in the registers FD and FFD (step SB5). That is, after initialization, the initial data of D, FD, and FFD are the same.

As described above, after the initialization of the main routine, steps SB2 to SB5 are executed, and the calculated D is stored in the FD and FFD. Thereafter, in step SB1, the music start time (first mouth recognition) is stored. ), The process proceeds from NO in step SB1 to step SB6, and the mouth recognition process for the second time and thereafter shown in FIG. 8 is executed. By this mouth recognition process, the coordinates a1 and a2 of the both ends of the mouth and the coordinates a3 and a4 of the top and bottom of the mouth are newly stored in the work RAM 13. Therefore, the previous coordinates of both ends of the mouth and the upper and lower coordinates of the mouth are stored in Fa1 to Fa4, and the current coordinates of both ends of the mouth and the upper and lower coordinates of the mouth are stored in a1 to a4. Next, the difference between a1 to a4 and Fa1 to Fa4, that is, the difference between the x coordinate in the horizontal direction and the y coordinate in the vertical direction is obtained as follows (step SB7).
Δx1 = x1−xF1, Δy1 = y1−yF1
Δx2 = x2-xF2, Δy2 = y2-yF2
Δx3 = x3-xF3, Δy3 = y3-yF3
Δx4 = x4-xF4, Δy4 = y4-yF4
Then, a1 to a4 are stored in Fa1 to Fa4 (step SB8).

Next, the same direction moving component is calculated based on the obtained difference (step SB9). The same direction movement component can be determined by the presence or absence of movement of the x coordinate of the center of two points at both ends of the mouth and the y coordinate of the center of two points above and below the mouth. Therefore, the same direction moving components Δx and Δy are calculated by the following calculation.
Δx = (Δx1 + Δx2) / 2− (ΔFx1 + ΔFx2) / 2
Δy = (Δy1 + Δy2) / 2− (ΔFy1 + ΔFy2) / 2
Next, the movement components (Δx, Δy) in the same direction are subtracted as follows using a1 to a4 (step SB10).
Δx1 ′ = Δx1−Δx, Δy1 ′ = Δy1−Δy
Δx2 ′ = Δx2−Δx, Δy2 ′ = Δy2−Δy
Δx3 ′ = Δx3−Δx, Δy3 ′ = Δy3−Δy
Δx4 ′ = Δx4−Δx, Δy4 ′ = Δy4−Δy
Since the image is enlarged or reduced depending on the distance between the mouth and the microphone, the coordinates of the four points change. Considering this case, the area of the dotted line in FIG. 10, that is, the area of the mouth is calculated after the mouth recognition, and the coordinate data is normalized. Such normalization is well known in general image processing techniques, and therefore detailed description of the arithmetic processing is omitted.

  Next, the mouth size D is calculated from a1 to a4 (step SB11). And it is discriminate | determined whether D is larger than predetermined value (step SB12). The predetermined value is an upper limit value when the mouth opening method is not singing. Therefore, when D is larger than a predetermined value, it is determined that the user is singing. In this case, it is determined whether or not the previous mouth size FD is greater than or equal to the previous mouth size FFD (step SB13). If FD is greater than or equal to FFD, FD is the current mouth size FFD. It is determined whether or not it is larger than the size (step SB14). For example, as shown in FIG. 11, when the mouth size FFD in the state (A) changes to a large FD in the state (B) and changes to a small D again in the state (C), In the process of transition from FFD to D, it shows that the magnitude of FD is a maximum value. That is, it shows that the mouth has become larger and smaller again. In this case, the value of the variable N indicating the number of opening / closing of the mouth is incremented (step SB15).

  After incrementing the value of N or when FD is less than or equal to D in step SB14, if FD is smaller than FFD in step SB13, that is, the size of FD is not a maximum value and the mouth has not been opened or closed If the value of D is equal to or smaller than a predetermined value in step SB12, the value of FD is stored in FFD (step SB16), and the value of D is stored in FD (step SB17). Then, the process returns to the main routine of FIG.

  It should be noted that instead of the calculation processing from step SB7 to step SB11 or in combination with these calculation processing, the widths FFH, FH, H of the left and right sides of the mouth shown in FIG. The respective ratios FFV / FFH, FV / FH, and V / H are calculated to determine the opening / closing of the mouth, or the image is enlarged or reduced depending on the translation of the mouth and the distance between the mouth and the microphone. It may be configured to perform the correction process.

  FIG. 9 is a flowchart of the DSP control process of the first embodiment in step SA9 of the main routine. It is determined whether or not the timer has counted a certain time, which is an effect processing interval (step SD1). If the certain time has not been counted, the process returns to the main routine. A send coefficient is generated based on the value (step SD2). The send coefficient is a parameter that determines a feedback component (amount or the like) in the feedback unit 184 of the DSP unit 18 in FIG. Further, a time coefficient is generated based on the value of N (step SD3). The time coefficient is a parameter that determines the delay time in the delay unit 183 of the DSP unit 18. Next, the generated send coefficient and time coefficient are supplied to the DSP unit 18 (step SD4). Thereafter, the initial value 0 is stored as the value of N (step SD5), the timer is cleared and started again (step SD6). Then, the process returns to the main routine of FIG.

  FIG. 12 is a diagram showing the characteristic of the coefficient Cdsp for the DSP unit 18 with respect to the opening / closing number N of the mouth. The coefficient Cdsp includes a delay coefficient Cdsp (delay time) for the delay unit 183 of the DSP unit 18 and a feedback coefficient Cdsp (send). As shown in FIG. 12, the smaller the value of N, that is, the lower the opening / closing frequency of the mouth, the larger both coefficients. As a result, when a song with a mood is sung without much movement of the mouth, the reverb and the echo are deeply pronounced, so that the singing ability can be improved more than actual. On the other hand, if you are singing crisply with your mouth open frequently, there are many people who originally have singing ability, so you can use the singing ability by cutting the reverb and echo and outputting the audio signal as it is. To pronounce.

For example, in a plate-type reverb, when the reverb time is 4 seconds and the pre-delay time is 10 ms, the reverb is not applied when the opening / closing frequency of the mouth is high and the value of N is large. On the other hand, when the opening / closing frequency of the mouth is a medium value of N, the send amount (change amount) is −10 dB, and when the opening / closing frequency of the mouth is low and the value of N is small, the send amount is −3 dB. The reverb is gradually deepened as the opening and closing frequency of the mouth decreases.
Alternatively, when the send amount is constant -5 dB regardless of the opening / closing frequency of the mouth, the reverb time is 0.5 seconds when the opening / closing frequency of the mouth is high, and the reverb time is 2. When the open / close frequency is high for 8 seconds, the reverb time is set to 4.8 seconds, and the reverb is gradually deepened as the open / close frequency is low.
Alternatively, with the delay time fixed at 150 ms, when the mouth opening / closing frequency is high, no reverb is applied, and when the mouth opening / closing frequency is a medium N value, the send amount is set to −10 dB, When the switching frequency is low and the value of N is small, the send amount is set to -3 dB and the reverb is gradually deepened.
Alternatively, with the send amount fixed at −5 dB, when the mouth opening / closing frequency is high, the delay time is set to 5 ms, when the mouth opening / closing frequency is medium, the delay time is set to 50 ms, and the mouth opening / closing frequency is low. Increases the delay time to 500 ms and gradually deepens the reverb.
Also, as shown in FIG. 11B, the delay time is set according to the average interval from the maximum value at which the mouth opens to the next maximum value, that is, the tempo of singing. It may be configured to realize a delay time called.

As described above, the karaoke system according to the first embodiment is housed in the microphone 1 and outputs a sound signal according to the input sound, and the microphone 1 accommodates the video of the singer's mouth. And a camera unit 6 that captures an image and outputs an image signal. The CPU 10 detects a difference relationship between the audio signal output from the microphone unit 5 and the image signal output from the camera unit 6, and gives a control signal to the DSP unit 18 according to the difference relationship data, Effect processing is performed on the audio signal output from the unit 5.
Therefore, the singing ability can be enhanced more than actual using the correlation between the movement of the singer's mouth and the tone of the singing voice.

Next, a second embodiment of the present invention will be described.
FIG. 13 is a block diagram showing a configuration of a karaoke system in the second embodiment using the voice control device according to the present invention. In FIG. 13, a CPU 10 is connected via a system bus 11 to a program ROM 12, a work RAM 13, a switch unit 14, a display unit 15, a sound source 16, a song data ROM 17, an A / D conversion circuit 20, and the camera unit shown in FIG. 6 is connected. The CPU 10 controls the entire karaoke system by exchanging data and commands with the above-described units connected via the system bus 11. The microphone unit 5 shown in FIG. 2 inputs an audio signal to the A / D conversion circuit 20. The A / D conversion circuit 20 converts the audio signal from analog to digital and inputs it to the DSP unit 18. The internal configuration of the DSP unit 18 is the same as that of the first embodiment shown in FIG.
As described above, the karaoke system in the second embodiment is almost the same as the configuration of the first embodiment, but in the second embodiment, the output of the A / D conversion circuit 20 is connected to the system bus 11. . As described below, the CPU 10 takes in an audio signal obtained from the A / D conversion circuit 20 and controls the DSP unit 18.

FIG. 14 is a flowchart of the main routine of the CPU 10 in the second embodiment. First, after predetermined initialization (step SG1), music selection processing is performed in accordance with the music selection operation (step SG2), and it is determined whether or not the music start key is turned on (step SG3). When the song start key is turned on, a timer is started (step SG4). Next, both the register FFE for storing the previous value of the envelope of the audio signal and the register FE for storing the previous value are cleared to 0 (step SG5). Further, a variable N representing the number of times of opening and closing the mouth and M representing the number of peaks (maximum values) of the envelope of the audio signal are both set to 0 (step SG6). Next, the selected music data is read from the music data ROM 17 (step SG7) and sent to the sound source 16 (step SG8). Next, image recognition processing is executed based on the image signal from the camera unit 6 (step SG9), and microphone input control processing is executed based on the audio signal obtained from the microphone unit 5 via the A / D conversion circuit 20. (Step SG10), and DSP control processing is executed (step SA11). After this, it is determined whether or not the song is finished or the stop key is turned on (step SA12). If the song is not finished and the stop key is not turned on, the process proceeds to step SG7 and the song data is transferred. Repeat reading. In step SG12, if the song end or stop key is on, the process proceeds to step SG2, and the next song is selected in accordance with the operation of the switch unit.
In this main routine, the image processing in step SG9 and the mouth recognition processing in the image processing are the same as those in the first embodiment shown in FIGS. 6 and 7 and the first embodiment shown in FIG. This is the same as the mouth recognition process.

  FIG. 15 is a flowchart of the microphone input control process in step SG10. The envelope of the audio signal output from the A / D conversion circuit 20 is extracted (step SH1), and the envelope value is stored in the register E (step SH2). Then, it is determined whether or not the envelope value stored in E is larger than a predetermined value (step SH3). The predetermined value is an upper limit value at which it is determined that the singer is not singing toward the microphone 1. If the envelope value of E is larger than the predetermined value, that is, if it is determined that the singer is singing into the microphone 1, the previous envelope value stored in the FE is It is determined whether or not the value is greater than or equal to the envelope value (step SH4). If the FE value is greater than or equal to the FFE value, it is further determined whether or not the FE value is greater than the current envelope value stored in E (step SH5). Immediately after the initialization of step SG1 in FIG. 14, since the values of FE and FFE are initialized to 0 in step SG5, the process of step SH6 is skipped and the process proceeds to step SH7. Is repeated twice, the previous envelope value is stored in the FFE, and the previous envelope value is stored in the FE.

  In the third microphone input control process, when the value of FE is larger than the value of E, for example, in the transition of the envelope shown in FIG. When the value reaches the value E and then falls to the value E, the envelope of the FE is a maximum value. That is, it can be determined that the singer is in a state of opening his mouth and speaking out. Therefore, in this case, the value of M representing the peak of the envelope is incremented (step SH6). After this, or when the value of FE is not larger than the value of E at step SH5, when the value of FE is smaller than the value of FFE at step SH4, or when the value of E is less than or equal to a predetermined value at step SH3 In step SH7, the value of FE is stored in FFE, and the value of E is stored in FE (step SH8). Then, the process returns to the main routine of FIG.

FIG. 16 is a flowchart of the DSP control process in step SG11 in the main routine. It is determined whether or not the timer has counted a certain time as an effect processing interval (step SJ1). If the certain time has not been counted, the process returns to the main routine, but if the certain time has been counted, The absolute value of the difference between the value and the value of N is calculated and stored in the register α (step SJ2). Then, a send coefficient is generated based on α (step SJ3). As in the first embodiment, the send coefficient is a parameter that determines a feedback component (such as an amount) in the feedback unit 184 of the DSP unit 18 in FIG. Further, a time coefficient is generated based on the value of α (step SJ4). The time coefficient is also a parameter for determining the delay time in the delay unit 183 of the DSP unit 18 as in the first embodiment. Next, the generated send coefficient and time coefficient are supplied to the DSP unit 18 (step SJ5). After this, the initial value 0 is stored in the values of M and N (step SJ6), the timer is cleared and restarted (step SJ7). Then, the process returns to the main routine of FIG.

  FIG. 18 is a diagram illustrating the characteristic of the coefficient Cdsp for the DSP unit 18 with respect to α, which is the absolute value of the difference between the number M of the envelopes of the audio signal and the number N of mouth opening / closing. Similar to the first embodiment, the coefficient Cdsp includes a delay coefficient Cdsp (delay time) and a feedback coefficient Cdsp (send) for the delay unit 183 of the DSP unit 18. As shown in FIG. 18, as the value of α increases, both coefficients increase. As a result, when you are singing a moody song without moving your mouth too much, if you detect a peak of the envelope of the audio signal, you can use reverb and echo to pronounce it, improving the singing ability more than it actually is be able to. On the other hand, if you are singing crisply with your mouth open frequently, there are many people who originally have singing ability, so you can use the singing ability by cutting the reverb and echo and outputting the audio signal as it is. To pronounce. Also, if you are singing like a humming or shout with your mouth closed at the interlude part of the song, or if your mouth is wide open during the ending of the song, make your voice lower and immerse yourself in the finish. If so, effect processing is performed to increase the mood by deepening the reverb.

Specifically, the three cases of a state where the mouth opening / closing frequency is high (high frequency), a state where the mouth opening / closing frequency is medium (medium frequency), or a state where the mouth opening / closing frequency is low (frequency) are further When the sound signal envelope is classified into a state where the envelope of the audio signal is larger than the predetermined value (occurrence occurs) or the state where the envelope of the audio signal is equal to or smaller than the predetermined value (occurrence occurs), the control purpose of the effect processing in these six singing states is as follows: become.
(1) <High frequency, with voice: normal tempo singing voice> Moderate effect processing (2) <High frequency, no voice> Since no voice is produced, unneeded parts can be heard so that only the accompaniment is heard Effect processing to cut and make it easy to hear (3) <With voice during frequency: Singing voice of slow tempo> Effect processing to deepen the sound and excite the atmosphere (4) <No voice during frequency> Since there is no voice, Effect processing that cuts unnecessary parts and makes it easier to hear so that only the accompaniment is heard (5) <Infrequent, with voice: humming and shout singing voice> Effect processing that stretches slowly and long (6) < Low frequency, no voice> Since there is no voice, effect processing that makes it easy to hear by cutting unnecessary parts so that only the accompaniment can be heard purely. nd (reverb, delay), an example of a reverberation time, and delay time in FIG.

As described above, the karaoke system according to the second embodiment captures an image of a singer's mouth accommodated in the microphone 1 and the microphone unit 5 that outputs an audio signal according to the sound accommodated in the microphone 1 and input. And a camera unit 6 for outputting an image signal. The CPU 10 detects the differential relationship between the envelope of the audio signal output from the microphone unit 5 and the image signal output from the camera unit 6, and gives a control signal to the DSP unit 18 according to the data of the differential relationship. Then, effect processing is performed on the audio signal output from the microphone unit 5.
Therefore, by controlling the effect according to the difference between the singing voice of the singer and the movement of the singer's mouth, the correlation between the singing's mouth movement and the timbre of the singing voice is used, and the singing ability is more than actual. Can be increased.

In the first and second embodiments, the correlation between the audio signal output from the microphone unit 5 and the image signal output from the camera unit 6 is the difference relationship between the two, but the correlation is the difference relationship. It is not limited.
The correlation between the aspect ratio of the mouth and the waveform data of the audio signal shown in FIG. 11 is detected, control data corresponding to the correlation is supplied to the DSP unit 18, and the audio signal output from the microphone unit 5 is output. Alternatively, the effect processing may be performed. For example, when the ratio of the aspect ratio of the audio signal is small and the frequency of the waveform data of the audio signal is fluctuating, it can be considered that the voice is sung without shaking your mouth so much, so an effect that emphasizes vibrato Apply processing. Alternatively, when the ratio of the vertical / horizontal ratio is large and the frequency of the waveform data of the audio signal is high, it can be considered that the voice is being sung in an effort to produce a high voice.
In addition, the mouth shape pattern of the image signal corresponding to each of a plurality of types of waveform data of the audio signal and the control signal data corresponding to each pattern are stored in advance, and the image signal output by the camera unit 6 is supported. The control signal data to be read may be read out, and the control signal may be provided to the DSP unit 18 to effect the audio signal output from the microphone unit 5.

The external view of the microphone in each embodiment of this invention. The figure which shows the internal structure of the microphone of FIG. The block diagram which shows the structure of the karaoke system of 1st Embodiment. The block diagram which shows the internal structure of the DSP part in FIG. The flowchart of the main routine of CPU in 1st Embodiment. 6 is a flowchart of image recognition processing in FIG. 5. The flowchart of the image recognition process following FIG. The flowchart of the mouth recognition process in FIG. 6 is a flowchart of DSP control processing in FIG. The figure which shows the coordinate of four points of the both ends of an opening | mouth and upper and lower sides in 1st Embodiment. The figure which shows transition of opening and closing of the opening | mouth in 1st Embodiment. The figure which shows the characteristic of the coefficient with respect to the DSP part with respect to the opening-and-closing number of mouths in 1st Embodiment. The block diagram which shows the structure of the karaoke system of 2nd Embodiment. The flowchart of the main routine of CPU in 2nd Embodiment. The flowchart of the microphone input control process in FIG. The flowchart of the DSP control processing in FIG. The figure which shows the envelope of the audio | voice signal in 2nd Embodiment. The figure which shows the characteristic of the coefficient with respect to the DSP part with respect to the difference of the opening / closing number of mouths and the envelope of an audio | voice signal in 2nd Embodiment. The figure which shows the specific example of the coefficient with respect to the DSP part with respect to the state of the mouth in 2nd Embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Microphone 2 Microphone cover 3 Transparent protective cover 4 Board | substrate 5 Microphone part 6 Camera part 10 CPU
12 Program ROM
13 Work RAM
16 sound source 17 song data ROM
18 DSP unit 20 A / D conversion circuit 181 Effector 182 Signal synthesis unit 183 Delay unit 184 Feedback unit

Claims (14)

A signal generating means that is accommodated in a microphone and outputs an audio signal according to the input audio;
Imaging means housed in the microphone and imaging a singer's mouth and outputting an image signal;
Correlation detecting means for detecting the correlation between the audio signal output from the signal generating means and the image signal output by the imaging means;
Signal processing means for effecting the audio signal output from the signal generating means according to the correlation data detected by the correlation detecting means;
A voice control device. The correlation detection unit includes a difference detection unit that detects a difference relationship between an image signal output from the imaging unit and an audio signal output from the signal generation unit, and the signal processing unit includes the difference detection unit. The sound control apparatus according to claim 1, wherein effect processing is performed on the sound signal output from the signal generating unit based on the difference relation data detected by the step .   The voice control apparatus according to claim 1, wherein the signal processing unit is accommodated in the microphone.   The audio control apparatus according to claim 1, wherein the signal processing unit controls a feedback component of reverberation processing. The voice control apparatus according to claim 1, wherein the signal processing unit controls a reverberation time of a reverb process.   The voice control apparatus according to claim 1, wherein the signal processing unit controls a feedback component of delay processing. The voice control apparatus according to claim 1, wherein the signal processing unit controls a delay time of delay processing. Detecting a sound signal output from the microphone in accordance with the input sound; and
A step B of detecting an image signal output from an image pickup means for picking up an image of a singer's mouth contained in the microphone;
Detecting a correlation between the audio signal detected in step A and the image signal detected in step B;
Performing an effect process on the audio signal output from the microphone according to the correlation data detected in the step C; and
Voice control method to execute.   The step C includes a step E for detecting a difference relationship between the audio signal detected in the step A and the image signal output in the step B, and the step D includes the difference detected in the step E. 9. The audio control method according to claim 8, wherein effect processing is performed on an audio signal output from the microphone based on related data.   The voice control method according to claim 8 or 9, wherein in step D, effect processing is performed by signal processing means accommodated in the microphone.   The voice control method according to claim 8 or 9, wherein the step D controls a feedback component of reverberation processing. The voice control method according to claim 8 or 9, wherein the step D controls a reverb time of a reverb process.   The voice control method according to claim 8 or 9, wherein the step D controls a feedback component of delay processing. The voice control method according to claim 8 or 9, wherein the step D controls a delay time of delay processing.

Images