JP3535360B2 - Sound generation method, sound generation device, and recording medium - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a sound generation device that generates sound based on an acquired image and a three-dimensional arrangement position of acoustic elements.

[0002]

2. Description of the Related Art To play music, a musical instrument such as a piano or a guitar is usually used. If it's a piano,
A piece of music is created by playing the keyboard according to the score. With a guitar, you can play music by flipping the strings with a nail or a pickup. As described above, in the conventional musical instruments, the playing method and the sound to be played are different for each musical instrument.

An electronic musical instrument such as a synthesizer is made in order to overcome such a difficulty, and playing a keyboard is the same as playing a piano, but what kind of sound is played? You can select the timbre. When a tone color or the like is selected, parameters such as the tone color are changed and a sound is generated under the control of the built-in microcomputer.

With the synthesizer, it has become possible to play and enjoy music with various tones.
However, it is a fact that considerable training is required in order to be able to enjoy it as you like while playing the keyboard and changing the timbre. Also, music software for playing music by selecting a keyboard or slide bar of a synthesizer displayed on the screen with a mouse is sold. Certainly, there is no need for a large keyboard, so there is a merit that you can play music even in a small space with one PC. On the other hand, instead of playing the keyboard, the mouse is used to select each one, which causes a problem that it takes time.

The research of a new musical instrument Hyper Instruments (hereinafter referred to as a super musical instrument) that breaks the limit of such a conventional electronic musical instrument is conducted by To of MIT Media Lab.
conducted by Mmachover et al. For example, using a data glove (a glove-shaped optical fiber is wired to the finger part and the bending angle of the finger joint is measured by the continuity of the optical fiber), a gesture is input and a guitar or the like is virtually input. Playing Bug-Mudr
We developed a. However, it is difficult to measure the angle of the knuckle in a robust manner because wearing the data glove becomes a burden and the glove may shift during the performance even if the data glove is calibrated. .

[0006] Machover et al. Also measure the movement of the human body by utilizing the fact that a human sits in a chair forming an electric field and the electric field changes when the human moves, and a sound is generated according to the result. Do Sensor Sensor
We also developed that. Similarly, an electric field is formed in a rectangular frame that hangs in the air, and the movement of the hand inserted into the square frame is measured by the change in the electric field to generate sound.
s was also developed. However, the sounds played by these are
It is quite different from the sound of conventional music, and there is a problem that it takes more technology than conventional classical musical instruments to play like music.

On the other hand, there is a program for generating a sound by converting a color at a position pointed by a mouse pointer on the screen into a sound. In this case, the correspondence between the color and the sound is predetermined, and since the color at the position of the mouse pointer is one color, only one sound is generated at the same time. Using this, it is very difficult to keep clicking the mouse button and generate a melody line well.
Since it is only a single note, it is difficult to generate music above the melody line.

[0008]

In order to realize a more intuitive playing method, which is not dependent on the conventional playing method of a classical musical instrument such as playing the keyboard or playing the guitar,
od Machover et al.'s Sensor Hair
Like Frames, it sounds too avant-garde and requires some training to play like music.

The method of simply converting the display color at the position pointed to by the mouse pointer to a color is too simple to produce a bloated sound as music, and it is difficult to keep pressing the mouse pointer. There is. Therefore, an object of the present invention is to provide a sound generation method and a sound generation apparatus using the same, which can easily generate a desired sound by a simple operation.

[0010]

(1) The sound generation method of the present invention (claim 1) is based on the depth information of the acquired image and the positional relationship of the acoustic elements of the image with respect to the two-dimensional arrangement position. A desired sound can be easily generated by a simple operation by calculating the power spectrum of the phoneme and generating the sound based on the calculated power spectrum and the phoneme.

According to another aspect of the present invention, there is provided a sound generation apparatus (claim 3), which is a sound element storage means for storing a two-dimensional arrangement position of a sound element, an image acquisition means for acquiring an image, and an image acquired by the image acquisition means. Calculating means for calculating the power spectrum of the acoustic element based on the positional relationship with respect to the arrangement position of the acoustic element stored in the acoustic element storage means and depth information of the image, and the power spectrum calculated by the calculating means and the Generating means for generating sound based on the phoneme. (2) The sound generation method (claim 2) of the present invention calculates the power spectrum of the acoustic element based on the acquired depth information of the image and the positional relationship of the acoustic element of the image with respect to the three-dimensional arrangement position, By generating a sound based on the calculated power spectrum and the phoneme, a desired sound can be easily generated by a simple operation.

Further, the sound producing apparatus of the present invention (claim 4).
Is an acoustic element storage unit that stores a three-dimensional arrangement position of the acoustic element, an image acquisition unit that acquires an image, and an arrangement position of the acoustic element stored in the acoustic element storage unit of the image acquired by the image acquisition unit. Calculating means for calculating a power spectrum of the acoustic element based on the positional relationship with respect to the image and depth information of the image, and generating means for generating a sound based on the power spectrum calculated by the calculating means and the acoustic element. To have. (3) Further, the sound generation apparatus (Claim 5) of the present invention comprises an image storage means for storing the image acquired by the image acquisition means, and the calculation means is stored in the image storage means. By generating a power spectrum of the acoustic elements based on the positional relationship of the image elements with respect to the arrangement position of the acoustic elements and depth information of the image, the same sound is generated many times when the array of the acoustic elements is unchanged. You can

Further, the sound producing device of the present invention (claim 6).
Since the change means for changing the arrangement position of the acoustic element is provided, the desired sound can be easily generated by a simple operation. For example, the generated sound can be further changed so as to match the acquired gesture or hand gesture image.

Further, the sound producing device of the present invention (claim 7).
By including a changing unit that changes at least one of the power spectrum and the arrangement position of the phoneme according to a predetermined rule, the desired sound can be easily generated by a simple operation. For example, even when the motion of the acquired gesture or hand gesture image is monotonous, a varying sound can be generated.

The sound producing apparatus of the present invention (claim 8) comprises a changing means for changing at least one of the power spectrum and the arrangement position of the acoustic element based on information extracted from a given music file. By doing
A desired sound can be easily generated by a simple operation. For example, it is possible to generate a sound according to BGM (background music).

[0016]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to the drawings. (First Embodiment) FIG. 1 schematically shows the configuration of a sound generation apparatus according to the first embodiment. As shown in FIG. 1, the sound generation device is an image of a hand gesture, a gesture, or the like, and is configured by an image capturing unit such as a camera for acquiring an image (hereinafter referred to as a depth image) whose depth value can be determined. Extraction from the acquisition unit 1, a phoneme storage unit 2 that stores a two-dimensional or three-dimensional arrangement position of a plurality of phonemes having different frequencies, and a depth image of a hand gesture or a gesture acquired by the image acquisition unit 1 From the image of the xy plane to be stored and the arrangement position of the acoustic element stored in the acoustic element storage unit 2, the acoustic element arranged at a position overlapping the image of the xy plane is selected, and the selected acoustic element and the depth are selected. Power used by the acoustic element calculation unit 4 for generating sound from the power spectrum of each selected acoustic element calculated based on the z value (depth information) determined from the image, and the power used by the three-dimensional acoustic element calculation unit 4. Spectrum A power spectrum calculating unit 5 that calculates, sound generating unit 7 for presenting the calculated result of the acoustic element calculation unit 4 as sound
And a display unit 6 that displays the placement positions of the phonemes stored in the phoneme storage unit 2, depth images such as hand gestures and gestures acquired by the image acquisition unit 1, and the calculation results of the phoneme calculation unit 4.
And an information management unit 3 that controls exchange of various information between the above units.

FIG. 2 shows an example of a storage format of placement positions of the phonemes stored in the phoneme storage unit 2. There are a plurality of types of phonemes and positions where the phonemes of that type are installed. It is stored.

Here, as the type of the phoneme, for example, the type depending on the difference in frequency is used. Further, according to the storage format shown in FIG. 2, not only acoustic elements of a certain frequency can be arranged in only one place, but also acoustic elements can be scattered here and there.

Since FIG. 2 shows an example in which the phonemes are arranged two-dimensionally, a combination of the x coordinate and the y coordinate is stored in each position coordinate. Subsequent addresses are stored in the last cell of each address. If "NULL" is written in this part,
Indicates that there is no next data.

FIG. 3 shows an example of a screen display when the phonemes stored in the storage format of FIG. 2 are displayed on the display unit 6. In FIG. 3, acoustic elements having frequencies of 100 Hz to 4.0 KHz are arranged. For reference, the sound of "A" (corresponding to "La") used for the articulation of the orchestra is 440 Hz.

It should be noted that how to arrange the acoustic elements of which frequency can be arbitrarily set, for example, according to the application of the sound generating apparatus of the present invention. Two phonemes are shown in FIG.
Although the example of storing the phonemes in the case of arranging the elements three-dimensionally has been shown, the present invention is not necessarily limited to this. For example, it is possible to arrange them three-dimensionally.

FIG. 4 is another example of the storage position of the acoustic elements stored in the acoustic element storage unit 2 and shows a case where the acoustic elements are three-dimensionally arranged. In FIG. 4, the stored position coordinates of each phoneme are a set of three coordinates of x coordinate, y coordinate, and z coordinate.

FIG. 5 shows an example of a screen display when the phonemes stored in the storage format of FIG. 4 are displayed on the display unit 6, and each phoneme is three-dimensionally arranged. In FIG. 5, each acoustic element is displayed in the shape of a cylinder, but the present invention is not limited to this. It can be a sphere, a cone, a cube, or a more complex shape.

FIG. 6 shows a hand image as an example of the image acquired by the image acquisition unit 1. This kind of image is x
It is a three-dimensional image having coordinates, y coordinates, and z coordinates. Regarding the acquisition method, there is a method of analyzing a moving image acquired by a video camera by conventional image processing and cutting out a shape such as an edge to cut out a shape. However, with conventional image processing, it is not possible to cut out to the depth value, so here,
The operation of the present invention will be described on the assumption that an image (depth image) whose depth value (z coordinate value) can be determined has been acquired by the image acquisition method described in Japanese Patent Application No. 9-299648. That is, it is assumed that the depth value is a value based on the gradation degree due to the shading of the pixel.

FIG. 7 shows a case where sounds are generated (played) from the two-dimensionally arranged phonemes as shown in FIG. 3, using the depth image of the hand shown in FIG. 3 shows that the image of the hand acquired by the image acquisition unit 1 overlaps the image of the two-dimensionally arranged acoustic elements of FIG. 3 already displayed on the display unit 6. Therefore, it may be displayed on the display unit 6 as shown in FIG.

Similarly, FIG. 8 uses the depth image of the hand shown in FIG. 6 to generate sounds from the three-dimensionally arranged phonemes as shown in FIG. 5, for example. Exist)
Here is what it looks like.

In order to simplify the explanation, the operation of the sound producing apparatus of FIG. 1 will be described below by taking the case of playing the two-dimensionally arranged acoustic elements of FIG. 7 as an example. FIG. 9 is a flowchart for explaining the operation of the sound generation device of FIG. 1, and under the control of the information management unit 3, each unit of FIG. 1 executes the following operations.

First, the information management unit 3 reads the acoustic element data stored in advance in the acoustic element storage unit 2, for example, in the format shown in FIG. 2 (step S
1), send the read phoneme data to the display unit 6,
For example, it is displayed as shown in FIG. 3 (step S
2). After that, the process waits for an image to be input from the image acquisition unit 1.

When the image acquisition unit 1 starts image acquisition, the information management unit 3 operates at 30 frames per second, for example, as shown in FIG. 6, unless the image acquisition unit 1 gives an instruction to end the image acquisition. Acquire an image (steps S3 to S)
4). If there is an end instruction (step S3),
The process ends.

The image acquired by the image acquisition unit 1 is, for example,
The depth image has a format as shown in FIG. FIG. 10 shows an image having 16 pixels in the x-axis (horizontal) direction, 16 pixels in the y-axis (vertical) direction, and 256 gradations in the z-axis (depth) direction. In other words, in a 16x16 matrix,
A value in the depth direction (depth value) is stored. Figure 10
In the example shown in (3), the larger the depth value, the closer the position of the phoneme. That is, the depth value "0"
Indicates that there is no imaged object, or even if there is no imaged object, the depth value “255” indicates that there is an imaged object at the closest position.

Where the phonemes are already displayed on the display unit 6 as shown in FIG. 3, an image obtained by extracting pixels having a depth value greater than a certain threshold value from the acquired depth images is overlapped. Then, for example, it is displayed as shown in FIG. 7 (step S5).

Next, in step S6, the power spectrum calculating section 5 is acquired by the image acquiring section 1 as shown in FIG.
The power spectrum is calculated using the depth value of the image as shown in FIG. As a method of calculating the power spectrum, for example, for a phoneme in which a hand image interferes, the depth value of the interference portion may be converted as a power value of the phoneme.

In order to generate a three-dimensional (deep) sound on the basis of, for example, the phoneme with which the hand image interferes and the calculated power spectrum of the phoneme, the phoneme computing unit 4 The sound generation unit 7 generates an actual sound based on the calculation result (step S7).

Next, the method of calculating the power spectrum in the acoustic element calculator 4 and the power spectrum calculator 5 will be described with reference to the flowchart shown in FIG.
Further, FIG. 12 is a graph showing an example of the calculated power spectrum of each phoneme.

First, initialization is performed (step S11).
That is, a matrix Xij (hereinafter simply referred to as a power matrix) that stores a power spectrum for each phoneme as a calculation result, and each phoneme for each phoneme type (for example, frequency as a type) in the power matrix Xij. So that the power value can be stored, the variable i for identifying the type is cleared and the address S for reading the placement position of the phoneme stored in the phoneme storage unit 2 is read at the top address (see FIG. In case of 2, "p1") is set.

Next, from the position (coordinate value) of the phoneme stored at the address S, the presence or absence of interference with the depth image (see FIG. 10) (the range of the position of the phoneme based on the placement position of the depth image of the phoneme). It is checked whether or not there is a portion overlapping the pixel) (step S12).

If the phoneme interferes with one or a plurality of pixels of the depth image, first, the phoneme type of the address S (in the case of FIG. 2, when S = p1, the phoneme type is "sound"). 1 ”) is stored in the matrix Xi1. Then, the depth values of the interfering pixels are sequentially superimposed on Xi2 (step S
13).

As shown in FIG. 2, the same address "p1" (that is, the same type,
In this case, a plurality of positions are stored for the same phoneme of the same frequency), and it is checked whether there are phonemes arranged at different positions (step S14). If there is, the process returns to step S12, the presence or absence of interference between the acoustic element at that position and the pixel is checked, and if there is interference, step S12.
At 13, the depth value of the interfering pixel is superimposed (for example, added) on the power matrix Xi2. Similarly, in step S12, even if there is no pixel interfering with the position of the phoneme, it is possible that the same type of phoneme placed at a different position may be interfering. If so, the process proceeds to step S14.

In step S14, if all the phonemes of the same type (that is, the same frequency, for example, stored in the same address) have been checked for interference, the process proceeds to step S15, and the next type is detected. The address variable S and the variable i are incremented in order to perform the same processing (steps S12 to S14) on the phoneme (i.e., the address "p2"). In the case of the storage format as shown in FIG. 2, since the address of the storage area storing the placement position of another type of phoneme is stored in the final cell of the storage area of each type of phoneme, Is stored in the address variable S. At this time, if the address stored in the final cell is "NULL" (step S16), it means that the calculation for all the phonemes has been completed, so the process is completed.

In step S15, the address variable S and the variable i
Is updated, and the address variable S becomes “NU in step S16.
If it is not "LL", the process returns to step S12 again to repeat the above process.

An example of the power spectrum calculated in this way is shown in FIG. In FIG. 12, the value of the power matrix Xi2 is shown by the length of the bar graph for each phoneme (Xi1) of each frequency that interferes with the acquired image.

The calculation result of the power spectrum as shown in FIG. 12 may be displayed on the display unit 6 as it is. The flow of the process in FIG. 11 will be specifically described by taking as an example the case of calculating the power value of the 100 Hz acoustic element in FIG. In FIG. 7, the index finger portion in the image interferes with the 100-Hz phoneme. Therefore, in step S12, it is determined that there is interference, and the depth value of the interfering pixel (for example, two) is “33” from FIG. 10,
It is assumed to be "74". In step S13, the depth values of these two pixels are superimposed on the power matrix, and as a result, the power value of the phoneme at 100 Hz is 33 + 74 = 10.
Calculated as 7. Since the acoustic element 100 Hz is arranged only in this one place, this “107” becomes the power value of 100 Hz as it is.

In the above description, the sum of the depth values of the pixels that interfere with the acoustic element is used as it is as the power value of the acoustic element for each frequency.
It is not necessary to set the depth value, and for example, when superimposing the depth value of the interfering pixel, the following operation may be performed on the depth value of the pixel.

1) power value = depth value × α (α: arbitrary value) 2) power value = (depth value) 3) power value = k × depth value 4) power value = F (x, y) × depth The value (F (x,
y): Weight value depending on distribution of depth value on xy plane) 5) Power value = F (x) × depth value (F (x): Weight value depending on distribution of depth value on x-axis) 6 ) Power value = F (y) × depth value (F (y): weight value depending on the distribution of the depth value on the y-axis) As shown in FIG. 5, each phoneme is arranged three-dimensionally. If so, in step S12 of FIG. 11, the interference check is performed using the depth value of the pixel. For example, the pixel x
The distance between the pixel and the phoneme is obtained from the coordinate value and the depth value on the y plane and the coordinate value of the phoneme, and when the distance is smaller than a predetermined threshold value, the pixel and the phoneme It is determined that there is interference with the phonemes. Then, when the power value of the phoneme is obtained in step S13 of FIG. 11, the same operation as described above may be performed, but it is not necessarily limited to the depth value. For example, simply, the area (the number of pixels) of the image interfering with the position of the phoneme is unchanged, or the area of the image is compared with the above 1) to 6).
The operation may be performed to calculate the power value.

Now, if each of the phonemes is sounded with the power spectrum as shown in FIG. 12, the sound can be generated according to the movement of the hand. However, a continuous sound cannot be obtained by simply sounding each phoneme with the power as shown in FIG.

Therefore, in the acoustic element calculator 4, the spectrum on the frequency axis calculated by the power spectrum calculator 5 as shown in FIG. 12 is changed into a waveform displaced on the time axis by the inverse Fourier transform. Let That is, a sine wave having a frequency of each phoneme is generated with the calculated power of each phoneme as the amplitude (for example, a waveform table for storing a waveform corresponding to the type of each phoneme may be provided. ),
The final overall waveform is generated by convolving and integrating this.

More specifically, referring to FIG. 12, first, a sine wave is generated at each frequency with the power shown in FIG. 12 as the amplitude. For example, 100H in FIG.
Since the power value of the acoustic element of z is 107 dB, as shown in FIG. 13, a 100 Hz sine wave having this amplitude is generated. Similarly, for the 200 Hz acoustic element in FIG. 12, a 200 Hz sine wave having the power value as the amplitude is generated as shown in FIG. In this way, FIG.
By integrating the sine wave (trigonometric function) calculated for each frequency of 2 for all frequencies, a waveform as shown in FIG. 15 is obtained.

The sound generation unit 7 generates a sound having the waveform shown in FIG. If the image acquired by the image acquisition unit 1 is 30 frames per second, the sound of the waveform shown in FIG. 15 is generated every 1/30 second. That is, 30 kinds of sounds are generated every second. (Effects of the First Embodiment) As described above, according to the first embodiment, the acoustic element storage unit 2 that stores the two-dimensional or three-dimensional arrangement position of the acoustic element and the image acquisition unit 1 are used. When the image (depth image) of the desired object is acquired, the power spectrum calculation unit 5 has a positional relationship between the depth information of the image and the two-dimensional or three-dimensional arrangement position of the acoustic elements stored in the acoustic element storage unit 2. The power spectrum of the phoneme is calculated based on the above, and the phoneme computing unit 4 generates a sound based on the calculated power spectrum and the phoneme, thereby performing a simple operation (for example, moving a hand or a body). ) Makes it possible to easily generate a timbre with a bulge.

Since a phoneme can be pronounced in real time using a depth image at a very low cost and with a small calculation load, anyone from small children to elderly people can intuitively do almost no practice. , You can enjoy playing and the effect is great. (Modified Example of First Embodiment) In the first embodiment, the case where a depth image such as a human hand gesture or a gesture is acquired and calculated as an acoustic element has been described. However, the present invention is not limited to this. It is not limited to. For example,
It is also possible to capture an image of a pet such as a goldfish or a small bird, instead of a human hand gesture or a gesture, calculate these depth images and acoustic elements, and generate environmental music (BGM) by the pet.

In this case, for example, in order to be able to apply different types and arrangements of the acoustic elements depending on the characteristics of various pets, the acoustic element storage unit 2 has the acoustic elements corresponding to the characteristics of various pets as shown in FIG. A plurality of array tables of are stored. When the user specifies the type of pet to the information management device 3, the arrangement table of the phonemes may be switched according to the type of the pet.

Further, in the first embodiment, it is assumed that the sound is produced with the power obtained from the interference between the range image and the phoneme in a completely silent state, but the present invention is not necessarily limited to this. Absent. For example, some phonemes have a certain power value and are sounded (active)
Then, for example, a performance in which the power of FIG. 12 is superimposed is also possible. (Second Embodiment) FIG. 16 schematically shows the configuration of a sound generation apparatus according to the second embodiment of the present invention. 16, the same parts as those in FIG. 1 are designated by the same reference numerals, and only different parts will be described. That is, in FIG. 16, in addition to the configuration of FIG.
Has been added.

The phoneme editing unit 8 is for changing the arrangement position of the phonemes stored in advance in the phoneme storage unit 2. Thereby, for example, the sound generated in accordance with the acquired gesture or hand gesture can be changed.

FIG. 17 shows an example of a screen display during editing when changing the arrangement of the phonemes displayed on the display section 6. For example, one of the phonemes displayed on the display unit 6 as shown in FIG. 3 is selected, and the phoneme is copied or moved to a desired position on the display screen, or arranged. It is also possible to perform editing such as deleting from the phoneme, and further to save the edited result from the phoneme unit editing unit 8 to the phoneme unit storage unit 2.

FIG. 17 shows that the 400 Hz acoustic element at the upper right of the center is selected (inverted display) and is copied to the lower left of the center. 18 is the same as FIG.
9 is a flowchart for explaining the operation of the sound generation apparatus shown in FIG. 9, and differs from FIG. 9 in that steps S21 and S22 are added between steps S3 and S4. That is, when the image acquisition is completed in step S3, the process proceeds to step S21, and the information management unit 3 first
It is checked whether there is a phoneme edit instruction. When there is an instruction to edit a phoneme, the process proceeds to step S22, and, for example, the phoneme editing process described with reference to FIG. 17 is executed.

It is assumed that the sound generation apparatus shown in FIG. 16 is used to perform a performance once on a certain array of phonemes, and then the array of phonemes is changed using the phoneme editing unit 8. In this case, it is desired to confirm whether or not the effect of editing is satisfactory for the performance before the phoneme is edited. However, it is usually difficult to reproduce the exact same movement as last time. In order to compensate for such a defect, for example, an image storage unit 9 for storing an image (depth image) previously acquired by the image acquisition unit 1 as shown in FIG. 19 is added to the configuration unit of FIG. You may.

The image storage unit 9 stores the series of depth images acquired by the image acquisition unit 1 by compressing and storing them, and after editing the phonemes, calls the stored depth image to extract the depth image and the sound. Based on the arrangement position of the elements, the power spectrum calculation unit 5 and the acoustic element calculation unit 4 execute the same processing as described above to generate a sound in which the arrangement of the edited element is more satisfactory than that before the editing. It is easy to confirm whether or not it has become a thing.

When the series of depth images stored in the image storage unit 9 are displayed on the display unit 6 for confirmation, the depth images can be confirmed as an animation. Specifically, for example, it is possible to confirm how the hand moves as shown in FIG. Alternatively, by displaying a sequence of depth images stored in the image storage unit 9 on the display of the array of phonemes shown in FIG. 3, the performance is performed in accordance with the movement of the hand, as shown in FIG. You can check the sound.

As described above, by storing the depth image in the image storage unit 9 and calling the stored depth image, the same sound can be generated many times when the arrangement of the phonemes is unchanged. . Even in this case, if the arrangement of the phonemes is changed, the sound is also changed. (Effect of the second embodiment)
As described above, according to the second embodiment, by changing (copying, moving, deleting) the placement position of the phoneme in the phoneme editing unit 8, the user can perform the performance according to his or her preference ( Sound generation) can be easily performed.

Further, an image storage unit 9 for storing the image acquired by the image acquisition unit 1 is provided, and a desired image is called from this image storage unit 9, and the power spectrum calculation unit 5 and the acoustic element calculation unit 4 are called. By generating the sound from the called image and the arrangement position of the sound element, the same sound can be generated many times when the arrangement of the sound elements is unchanged. (Third Embodiment) FIG. 20 schematically shows the configuration of a sound generation apparatus according to the third embodiment. In addition,
20, the same parts as those in FIG. 1 are designated by the same reference numerals,
Only different parts will be described. That is, in FIG. 20, in addition to the configuration shown in FIG.
0 is added.

In the first embodiment, when some phonemes have a certain power value and are sounded (active),
In addition, for example, it has been described that the performance in which the power of FIG. 12 is superimposed is also possible, but here, by applying it, the time series transformation unit 10 changes the power spectrum of the phoneme in time series. It has become. That is, the time series transformation unit 10 changes the power value of each phoneme in time series using some function such as a trigonometric function such as sin and cos.

For example, the time-series transformation unit 10 changes the power of some arbitrary phonemes in a time-series manner while the phonemes are being pronounced (for example, each sound in the graph as shown in FIG. 12). In a state where the power value of the element (frequency) changes up and down every 1 / K seconds), the power matrix of each of the phonemes is added to the image acquired by the image acquisition unit 1 for example every 1/30 second. It is also possible to superimpose the power value based on the power value (for example, the power value of each phoneme in FIG. 12) every time it is acquired.

The time-series transforming unit 10 is not limited to the power value of the phoneme, but can similarly change the placement position of the phoneme according to the trigonometric function or the like of the time-series transforming unit 10. . For example, when the phonemes as shown in FIG. 5 are three-dimensionally arranged, the shape of the phonemes may be changed by changing the length of the phonemes in the z-axis direction (depth direction).

The time-series transforming unit 10 transforms the power spectrum, arrangement position, etc. of an acoustic element in time-series.
IDI (Music Instrument Digi
It is also possible to transform a file in which music information is stored, such as a Tal Information file), in time series using the stored music information.

In order to perform the time-series transformation according to the music file in this way, for example, as shown in FIG. 21, a music information input section 11 is further added to the configuration of FIG. Music information from a music file (for example,
Alternatively, the time series transformation unit 10 may change the frequency, the power value, and the like of the phoneme based on the extracted pitch and volume of the sound. (Effects of the Third Embodiment) As described above, according to the third embodiment, at least one of the power spectrum and the arrangement position of the acoustic element in the time-series deformation unit 10.
By changing one according to a predetermined rule (for example, a trigonometric function with time as a variable), for example, when there is no large movement in the object for which an image is acquired, for example, even in the case of a pet such as a goldfish, Since the power spectrum and arrangement position of the phonemes are changed in time series, it is possible to prevent a monotonous performance.

Further, based on the information extracted from the music file (for example, MIDI file) by the music information input unit 11, the time-series transformation unit 10 changes at least one of the power spectrum and arrangement position of the acoustic element, It is possible to generate a sound that matches BGM (background music).

The methods described in the first to third embodiments are magnetic disks (floppy disks, floppy disks, etc.) as programs that can be executed by a computer.
Hard disk, etc., Optical disk (CD-ROM, D
It can also be stored in a recording medium such as a VD) or a semiconductor memory and distributed. That is, even when a personal computer or the like executes a predetermined recording medium that stores a program that describes the processing according to the flowcharts shown in FIGS. 9 and 18, it is the same as the description of the first to third embodiments. Is.

[0067]

As described above, according to the present invention,
A desired sound can be easily generated by a simple operation.

[Brief description of drawings]

FIG. 1 is a diagram showing a configuration example of a sound generation device according to a first embodiment of the present invention.

FIG. 2 is a diagram showing a case where acoustic elements are two-dimensionally arranged, which is a storage example of arrangement positions of acoustic elements stored in an acoustic element storage unit.

FIG. 3 is a diagram showing an example of a screen display when an acoustic element stored in the storage format of FIG. 2 is displayed on a display unit.

FIG. 4 is a diagram showing another example of the storage position of the acoustic elements stored in the acoustic element storage unit, showing a case where the acoustic elements are three-dimensionally arranged.

5 is a diagram showing an example of a screen display when the phonemes stored in the storage format of FIG. 4 are displayed on the display unit.

FIG. 6 is a diagram showing a hand image as an example of an image acquired by an image acquisition unit.

FIG. 7 is a diagram for explaining a case where sound is generated from the two-dimensionally arranged phonemes of FIG. 3 using the depth image of the hand shown in FIG.

8 is a diagram for explaining a case where sound is generated from the three-dimensionally arranged phonemes of FIG. 5 using the depth image of the hand shown in FIG.

9 is a flowchart for explaining the operation of the sound generation device in FIG.

FIG. 10 is an image (depth image) acquired by the image acquisition unit.
For explaining the depth value of the.

FIG. 11 is a flowchart for explaining a power spectrum calculation process.

FIG. 12 is a graph showing an example of the calculated power spectrum of each phoneme.

13 is a diagram showing an example of a sine wave (amplitude is a power value) of the 100-Hz acoustic element in FIG.

14 is a diagram showing an example of a sine wave (amplitude is a power value) of the 200-Hz acoustic element in FIG.

FIG. 15 is a diagram showing an example of a sine wave generated by superimposing a sine wave of each acoustic element whose amplitude is the calculated power value of each acoustic element.

FIG. 16 is a diagram showing a configuration example of a sound generation device according to a second embodiment of the present invention.

FIG. 17 is a diagram showing a screen display example during editing when changing the arrangement of the phonemes displayed on the display unit.

18 is a flowchart for explaining the operation of the sound generation device in FIG.

FIG. 19 is a diagram showing another configuration example of the sound generation device according to the second embodiment of the present invention, which includes an image storage unit.

FIG. 20 is a diagram showing still another configuration example of the sound generation apparatus according to the second embodiment of the present invention, which is provided with a time series deformation unit.

FIG. 21 is a diagram showing a further example of the configuration of the sound generation device according to the second embodiment of the present invention, which is provided with a music information input unit.

[Explanation of symbols]

1 ... Image acquisition unit 2 ... Phoneme storage 3 ... Information Management Department 4 ... Acoustic element calculator 5 ... Power spectrum calculation unit 6 ... Display 7 ... Sound generation unit 8 ... Sound editorial department 9 ... Image storage unit 10 ... Time series transformation section 11 ... Music information input section

Continuation of the front page (56) Reference JP-A 64-91190 (JP, A) JP-A 10-26978 (JP, A) JP-A 10-240282 (JP, A) Jitsuko 3-48638 (JP , Y2) Patent 2629740 (JP, B2) (58) Fields investigated (Int.Cl. ⁷ , DB name) G10H 1/00-7/12 G10K 15/04 302

Claims (9)

(57) [Claims] 1. An acquisition step of acquiring an image of an imaged object, in which each pixel value represents the depth of the imaged object, and a plurality of acoustic elements each defined by coordinates and frequency values.
From among the above, in the image acquired in the acquisition step,
A sound having coordinates included in the image area corresponding to the imaging target
A selection step of selecting Hibikimoto, corresponding to the coordinates included in each acoustic element selected in the selecting step among the pixels of the image area of the imaging target in the image
A calculation step of calculating a power value of a sound corresponding to each of the selected phonemes based on a pixel value of the pixel, and a plurality of frequency values and power values of the selected phonemes. And a sound producing step for producing a sound to be output by superimposing the sound of. 2. Defining each of the plurality of phonemes
The coordinate is a three-dimensional coordinate, The claim 1 characterized in that
The sound generation method described above. 3. An image pickup device for picking up an image of an image pickup target, wherein each pixel value obtains an image representing the depth of the image pickup target, and a plurality of acoustic elements each defined by coordinates and frequency values.
From among the above, the imaging in the image acquired by the acquisition means
A phoneme having coordinates included in the image area corresponding to the target
Based selecting means for selecting the pixel value of <br/> pixel corresponding to the coordinates included in each acoustic element selected by said selection means among the pixels of the image area of the imaging target in the image, the Calculation means for calculating a power value of a sound corresponding to each selected phoneme, and a sound to be output by superposing a plurality of sounds each having a frequency value and the power value of each selected phoneme A sound generation device comprising: a generation unit that generates a. 4. Defining each of the plurality of phonemes
4. The coordinates according to claim 3, wherein the coordinates are three-dimensional coordinates.
On-board sound generator. 5. The sound generation apparatus according to claim 3, further comprising a changing unit that changes the coordinates of the phoneme. 6. The changing means for changing the power value of each of the plurality of sounds based on a predetermined rule or information extracted from a given music file. Sound generator. 7. The sound according to claim 3, further comprising changing means for changing the coordinates of the plurality of phonemes based on a predetermined rule or information extracted from a given music file. Generator. 8. An acquisition step of acquiring an image of an imaged object, each pixel value of which represents the depth of the imaged object, and a plurality of acoustic elements each of which is defined by coordinates and frequency values.
From among the above, in the image acquired in the acquisition step,
A sound having coordinates included in the image area corresponding to the imaging target
A selection step of selecting Hibikimoto, corresponding to the coordinates included in each acoustic element selected in the selecting step among the pixels of the image area of the imaging target in the image
A calculation step of calculating the power value of the sound corresponding to each of the selected phonemes based on the pixel value of the pixel, and a plurality of frequency values and power values of the selected phonemes. A machine-readable recording medium that stores a program that causes a computer to execute a generation step of generating a sound to be output by superimposing the sound of. 9. Defining each of the plurality of phonemes
9. The coordinates are three-dimensional coordinates, according to claim 8.
Recording medium.