JP7633064B2 - Acoustic signal processing method and audio signal processing program - Google Patents

Description

The present invention relates to an audio signal processing method and an audio signal processing program.

When simulating the real world in a game or virtual space, humans can experience virtual reality by perceiving images, sounds, vibrations, etc. with their five senses.

To make virtual reality feel more realistic, it is often necessary to have the computer perform more calculations. For this reason, there is a demand for technology that makes it easier for computers to process virtual reality.

In particular, the following techniques are examples of conventional techniques relating to acoustics.
The positions in the virtual space of a sound source and a listener for performing acoustic expression in the virtual space are identified. A playback means is prepared for playing back a sound emitted by a sound source object set as a virtual sound source in the virtual space. The position of the sound source object is then identified by searching a quadtree having position information of small areas included in a predetermined area in the virtual space. There is a technology that allows a program to generate sound in the virtual space by having the playback means process sound data according to the result of the position identification (see, for example, Patent Document 1).

JP 2020-18620 A

The disclosed technology aims to allow humans to experience sounds that are closer to reality while reducing the processing burden on computers involved in sound processing in virtual reality.

According to the disclosed technology, a sound source and a listener exist in a virtual three-dimensional space including a plurality of subspaces, and an acoustic signal processing method is provided for outputting acoustic processing information used to obtain an acoustic signal including a sound signal perceived by the listener when sound emitted from the sound source propagates through the three-dimensional space and reaches the listener, in which first relational data defining a correspondence between a plurality of subspaces included in the three-dimensional space and an identifier assigned to each of the plurality of subspaces, and second relational data defining a correspondence between the identifier and a signal processing parameter for obtaining the acoustic signal are prepared in a memory, and the acoustic signal processing method includes the steps of: An acoustic signal processing method is provided that includes receiving the position of the sound source and the position of the listener, referring to the first relationship data using the position of the sound source and the position of the listener to identify a first identifier corresponding to the sound source and a second identifier corresponding to the listener, referring to the second relationship data using the first identifier and the second identifier to search for a first parameter corresponding to the first identifier and a second parameter corresponding to the second identifier, and outputting acoustic processing information based on at least the first parameter and the second parameter so as to obtain the acoustic signal.

The disclosed technology allows humans to experience sounds that are closer to reality while reducing the processing burden on computers involved in sound processing in virtual reality.

FIG. 1 shows subspaces, a listener, and a sound source that exist in a virtual space. FIG. 2 shows a process for outputting sound processing information used to obtain a sound signal perceived by a listener from the positions of a sound source and a listener. FIG. 3 shows an example in which a three-dimensional space is divided into three-dimensional meshes. FIG. 4 shows the positional relationship between a plurality of subspaces and a three-dimensional mesh. FIG. 5 shows a three-dimensional space filled with a number of rectangular parallelepipeds each made by a three-dimensional mesh. FIG. 6 shows a rectangular solid with subspaces and overlaps. FIG. 7 shows the positional relationship between a sound source, a listener, and a rectangular parallelepiped when three-dimensional space is viewed from the Z-axis direction. FIG. 8 is a process flow showing in more detail the process of identifying the first identifier and the second identifier. Fig. 9A shows a process flow for selecting parameters related to sound attenuation, etc., based on identifier information, Fig. 9B is a table defining groups of a plurality of layers. FIG. 10 shows a detailed process flow relating to the process of including parameters relating to sound attenuation and the like in the sound processing information. FIG. 11 shows a processing flow for parameters relating to sound attenuation and the like. FIG. 12 shows a processing flow of parameters relating to the generation of environmental sounds or sound reverberation effects. 13A and 13B show how to extract identifiers for each sound source and listener using multiple rectangular parallelepipeds in virtual space when a bridge is present outdoors. FIG. 14 illustrates a process for modifying the first parameter or the second parameter according to a first rule when the first identifier and the second identifier are different. FIG. 15 shows another example of processing when the first identifier and the second identifier are different. 16A and 16B show an example in which a partition exists in one partial space. 17A and 17B show identifiers that are assigned to each rectangular parallelepiped that exists in the virtual space near a partition. Fig. 18A shows detailed processing for determining whether a first identifier and a second identifier have a predetermined relationship, and Fig. 18B shows a table used for determining whether a combination of a first identifier and a second identifier has a predetermined relationship. Fig. 19A shows a table that defines the identifiers of subspaces and the types of subspaces, and Fig. 19B shows a table that defines the amount of sound attenuation and cutoff frequencies that are examples of parameters of low-pass filters assigned to the types of spaces. Fig. 20A shows a table indicating the relationship between the identifier value and the size of the subspace, and Fig. 20B shows a table in which the definition of the environmental sound corresponding to the size of the subspace and the names representing the reverb parameters are defined. FIG. 21 shows the hardware configuration of the embodiment.

Hereinafter, the embodiments will be described with reference to the drawings. Note that the following embodiments are not mutually exclusive. As long as there is no contradiction, a part of one embodiment can be replaced with or combined with a part of another embodiment. Furthermore, as long as there is no contradiction, each step of the disclosed flow can be replaced. Furthermore, each flow can be omitted or executed simultaneously. A part of the embodiment can be realized by either a program or hardware. Furthermore, a part of the program can utilize the function of an operating system. The program can be stored in a non-transitory recording medium.
The following embodiments relate to the generation and processing of sound. The generation and processing of sound may be performed using a pure computer program or using hardware. Furthermore, the processing of sound may be performed digitally or by an analog circuit.

<Embodiment 1>
Fig. 1 shows a subspace, a listener Li1, and a sound source S1 that exist in a virtual space V. In Fig. 1, a bird is depicted as an example of the sound source S1.

A plurality of partial spaces exist in the virtual space V. The virtual space V has five partial spaces: a room A, a room B, a room C, and an outdoor space E. Each of the five partial spaces is assigned the following identifier (ID).
Room A ID=10
Room B ID=20
Room C ID=30
Outdoor E ID=1

A sound (e.g., a bird's song) emitted from a sound source S1 in room B passes through the wall of room B and the wall of room C to reach listener Li1. In addition to passing through walls, sound can reach listener Li1 from sound source S1 via various paths, such as passing through the wall of room B, passing through outside E, passing through the wall of room C, and then reaching listener Li1. In this embodiment, the sound heard by listener Li1 may include components of the sound that passes through the walls of room B and room C. In this way, by considering only the attenuation of sound passing through adjacent walls, calculations related to sound propagation can be simplified.

Furthermore, the calculation of sound attenuation can be further simplified by adopting the sound attenuation in the walls of room B and room C, whichever has the greater attenuation effect, as the sound attenuation. Furthermore, even if there is another room between room B and room C (not shown), for example, the calculation of sound attenuation can be further simplified by omitting the attenuation of sound passing through the other room and the walls of the other room, and considering and selecting only the sound attenuation in the walls of room B where sound source S1 is located and the sound attenuation in the walls of room C where listener Li1 is located, as described above. The sound attenuation can be calculated by adding the sound attenuation in the walls of room B and the sound attenuation in the walls of room C where listener Li1 is located.

Since the human sense of sound is less sensitive than other senses such as vision, even if the acoustic signals such as the amount of sound attenuation are calculated using the above method, it is possible to obtain with a high degree of accuracy the sensation that the sound gives to listener Li1 when it reaches listener Li1. By doing so, it is possible to avoid performing complex calculations through detailed simulations of sound propagation.

For example, in terms of sound propagation, if the attenuation of the wall of room B is −3 dB and the attenuation of the wall of room C is −6 dB, then −6 dB should be adopted since the sound attenuation of room C is greater. By adopting this adopted −6 dB as the attenuation of the sound transmitted from the sound source S1 to the listener Li1, the volume of the sound perceived by the listener Li1 can be obtained.
In addition to attenuating the volume, the effect on sound propagation (filtering, etc.) may be processed in a similar manner to the above-mentioned approach regarding attenuation of the volume.

For example, when sound passes through a wall, the higher the frequency, the greater the attenuation. Therefore, it is desirable to apply a low-pass filter to the sound. For this low-pass filter processing, parameters can be adopted that, for example, result in greater attenuation of high frequencies the thicker the wall.

For example, if the cutoff frequency of the low-pass filter corresponding to the wall of room B is 1200 Hz and the cutoff frequency of the low-pass filter corresponding to the wall of room C is 800 Hz, the wall of room C will pass more low-frequency sounds and will not pass higher-frequency sounds as easily as the wall of room B, so the processing parameters of the low-pass filter of room C can be applied to the sound. The above is one aspect of an embodiment relating to parameter processing related to sound attenuation, etc. Parameters related to sound attenuation, etc. (i.e. sound attenuation parameters and low-pass filter processing parameters, etc.) are an example of a first type of parameter.

On the other hand, parameters relating to sound reverberation and the like can be processed in the following manner.
The effect of sound reverberation tends to be strongly influenced by the reverberant properties of the room that the listener Li1 is in. Therefore, for sound effects such as reverb or echo, it is desirable to employ parameters related to the room of the listener Li1.

Regarding environmental sounds, the environmental sounds in the room of the listener Li1 tend to have a stronger influence on the listener Li1 than the environmental sounds in other rooms. Therefore, it is desirable to employ parameters related to the environmental sounds in the room in which the listener Li1 is present.
Examples of environmental sounds include, for example, the sound of a car passing by on a nearby road in the case of outdoors E, and the sound of wind blowing through a window in the case of room B. For these environmental sounds, sounds with a reverberation effect of the room added in advance may be directly generated, or a reverberation effect (e.g., echo or reverb) may be added to the environmental sounds. In addition, a reverberation effect may be added to a sound that has been subjected to attenuation processing and low-pass filter processing, etc., from the sound source S1 to the listener Li1.
In addition, in outdoor E, for example, a delay effect simulating the effect of an "echo" may be added.

In order to determine which subspace of the virtual space the sound source S1 and the listener Li1 are present in, it is necessary to determine which subspace the three-dimensional coordinates of the sound source S1 and the listener Li1 (or the ears of the listener Li1) in the virtual space are present in. For this determination, when a three-dimensional model in which a solid object such as a room is configured with polygons is used, it is possible to determine whether the sound source S1 and the listener Li1 are present inside or outside the polygon that configures the boundary of which subspace based on the polygon information. Then, an identifier is assigned to each subspace of the three-dimensional model. It is possible to specify the subspace in which the sound source S1 and the listener Li1 are present and the identifier assigned to the subspace according to the model that defines the three-dimensional space.
The three-dimensional model in which each of the subspaces is assigned an identifier is an example of first relational data.
Also, a table that associates various acoustic processing parameters for each room and outdoor area with an identifier (ID) assigned to each room and outdoor area can be stored in memory.
If the identifier of the partial space in which the sound source S1 and the listener Li1 exist can be specified, the parameters of the acoustic processing can be specified by searching this table with the identifier. This table is an example of the second relational data.

For example, if the subspace identifier is set in advance to increase in response to an increase in the effect of a parameter related to sound attenuation, etc., the identifier to be used in determining the parameter related to sound attenuation, etc. to be adopted can be determined by judging the size of the identifier. The same applies to low-pass filter parameters. In this way, it is possible to select and determine the acoustic processing parameters to be adopted (for example, sound attenuation and low-pass filter) by judging the size of the identifier of the subspace in which the sound source S1 and listener Li1 exist.

When the sound source S1 and the listener Li1 are in the same subspace, sound attenuation processing may not be performed. Low-pass filter processing may not be added. Alternatively, the attenuation parameter value may be changed according to the distance between the sound source S1 and the listener Li1, so that the amount of attenuation is large when the distance is large.

Although one identifier is sufficient for the outdoor E, multiple identifiers may be provided. Details will be described later, but one aspect will be briefly described here. For example, assume that identifier 7 is given to the space under a bridge in the outdoor E, and identifier 1 is given to the other space in the outdoor E. Then, assume that listener Li1 is under the bridge, i.e., in the area of identifier 7, and sound source S1 is in the area of identifier 1. When listener Li1 and sound source S1 are at approximately the same height (i.e., listener Li1 and sound source S1 are both under the bridge), the sound reaching listener Li1 from sound source S1 is not blocked by the bridge, so sound attenuation may not be performed. When listener Li1 and sound source S1 are at different heights and there is a height difference between them that would result in a bridge being present between them, the selection of parameters may be changed so that sound attenuation is performed. This parameter change may be performed by storing a table in memory having multiple parameters related to sound attenuation and the like that are predetermined for each case. This processing is also similar to that for the parameters of the low-pass filter.
The above bridge embodiment can be applied not only outdoors but also indoors as well.
The above example of a bridge can also be applied to cases where a partition is present. For example, this can be achieved by applying multiple identifiers to the same subspace. For a bridge, the acoustic parameters can be changed based on the height relationship (layer relationship) between the sound source S1 and the listener Li1. In the case of a partition, the acoustic parameters can be changed based on whether the relationship between the identifier corresponding to the sound source S1 and the identifier corresponding to the listener Li1 is a predetermined relationship or not. These details will be described later.

Note that sounds emitted by the listener Li1 itself (e.g., words uttered by the listener Li1) may be processed using parameters related to reverberation and the like associated with the subspace in which the listener Li1 exists.

Figure 2 is a flowchart for outputting acoustic processing information used to obtain an acoustic signal perceived by listener Li1 from the position of sound source S1 and the position of listener Li1. The acoustic processing information is used as control information for a DSP (Digital Signal Processor) or the like to obtain a sound signal that can be perceived by humans.

Each step in FIG. 2 will be described below. It is assumed that a first table that defines the correspondence between a plurality of subspaces included in a three-dimensional space and an identifier assigned to each of the plurality of subspaces is stored in advance in memory. Similarly, it is assumed that a second table that defines the correspondence between the identifier and a signal processing parameter used when generating an acoustic signal is also stored in advance in memory. The first table and the second table are examples of the first relational data and the second relational data, respectively. It goes without saying that the first table and the second table may be combined and exist in memory as a single table. Alternatively, it is not necessarily required that the information in these tables form a table in memory.

Each step in FIG. 2 is explained below.
[Step S110] The position of the sound source S1 and the position of the listener Li1 in the virtual space are received. The three-dimensional positions may be expressed in three-dimensional coordinates with an X-axis, a Y-axis, and a Z-axis that are perpendicular to each other.

[Step S112] The first table is searched to identify a first identifier and a second identifier corresponding to the position of the sound source S1 and the position of the listener Li1, respectively.

[Step S116] The first and second parameters are obtained by searching the second table from the identifiers corresponding to the position of the sound source S1 and the position of the listener Li1.

[Step S118] Appropriate acoustic processing information is output based on at least the first parameter and the second parameter. Then, for example, the output acoustic processing information is input to a DSP (Digital Signal Processor) or the like to obtain a sound signal that can be perceived by humans.

The outline of how the first and second parameters are used to output the acoustic processing information is explained below, and the details will be described later.

For example, if the first identifier and the second identifier are the same, it is determined that there is no obstruction. That is, the sound source S1 and the listener Li1 are in the same subspace. In this case, the volume from the sound source may not be attenuated or low-pass filter processing may not be applied. That is, since the first identifier and the second identifier are the same, the first parameter and the second parameter are the same, and this parameter may be used. Alternatively, the volume may not be attenuated or low-pass filter processing may not be performed regardless of the parameter value. Alternatively, a parameter (not shown) related to sound attenuation when the first identifier and the second identifier are the same may be stored in the memory. Note that, for parameters related to environmental sound or sound reverberation, when the first identifier and the second identifier are the same, it is desirable to apply the first parameter (or the second parameter having the same value).

In contrast, for example, when the first identifier and the second identifier are different, the sound source S1 and the listener Li1 will be in different subspaces in principle. In this case, either the first parameter or the second parameter may be used. Note that, when the first identifier and the second identifier are different, the process of changing parameters related to sound attenuation and the like in a specific case has already been outlined using the examples of a bridge and a partition, and will be described in the embodiments described below.
<Embodiment 2>

In the first embodiment, in order to identify the space in which the sound source S1 and the listener Li1 exist, the positional relationship between the sound source S1 and the listener Li1 and the walls separating the subspaces is determined in accordance with the three-dimensional model that constitutes the three-dimensional space in the calculation of the subspace. In the first embodiment, it was necessary to calculate this positional relationship (i.e., to determine whether the sound source S1 and the listener Li1 exist inside or outside the subspace). The second embodiment, which makes this calculation even simpler, is described below.

3 shows an example of dividing a three-dimensional space into a three-dimensional mesh. The three-dimensional mesh is created on planes perpendicular to the X-axis, Y-axis, and Z-axis, and spaced apart at a predetermined interval. The virtual space is divided into small spaces by a plurality of rectangular parallelepipeds 300 separated by this three-dimensional mesh. The rectangular parallelepipeds 300 may be cubes.
For convenience, the following description will be given assuming that the XY plane is the horizontal plane in the three-dimensional space and the Z-axis direction is an axis perpendicular to the horizontal plane.

Figure 4 shows the positional relationship between multiple subspaces and a three-dimensional mesh. The relationship between Room A, Room B, Room C, and Outdoor E shown in Figure 1 and the three-dimensional mesh is shown.

FIG. 5 shows a three-dimensional space filled with a number of rectangular parallelepipeds each made by a three-dimensional mesh.
For convenience, a plate-like space composed of multiple rectangular parallelepipeds arranged parallel to a horizontal plane may be called a layer. FIG. 5 shows Layer 1 and Layer 2 as examples. It is preferable that each layer is numbered in order. The position of each rectangular parallelepiped on the layer plane can be specified by this number. Then, by assigning a number to each layer, the position of the layer in the Z-axis direction can be specified. The position of the rectangular parallelepiped on the layer plane and the layer number can be used to specify the position of the rectangular parallelepiped in three-dimensional space. Note that the method for specifying the position of the rectangular parallelepiped in three-dimensional space is not limited to this method.

6 shows a rectangular parallelepiped that overlaps with the subspace. In order to make it easier to see, the rectangular parallelepiped corresponding to the outdoor area E is omitted from FIG. 6.
Then, the same identifier as the identifier corresponding to each partial space is assigned to the corresponding rectangular parallelepiped. Note that, for example, as shown in rectangular parallelepiped 600, there are cases where rectangular parallelepiped 600 and room C do not share any space with each other. In such a case, for example, each rectangular parallelepiped 600 may be associated with the partial space with which it shares the most space. Then, the identifier of the associated partial space may be assigned to rectangular parallelepiped 600.

In this way, it is possible to more easily extract identifiers corresponding to the sound source and the listener by searching for cuboids that exist at the positions of the sound source and the listener and extracting the identifiers assigned to the cuboids that have been searched for.

Figure 7 shows the positional relationship between a sound source S2, a listener Li2, and a rectangular parallelepiped in a certain layer when three-dimensional space is viewed from the Z-axis direction. The identifiers assigned to the rectangular parallelepipeds located at the positions of the sound source S2 and the listener Li2 can be extracted by referring to a first table showing the relationship between the rectangular parallelepipeds and the identifiers. The first table can be stored in memory in advance.

By doing the above, it is possible to search for a rectangular parallelepiped that exists at the position of each of the sound source S2 and the listener Li2, and then refer to the first table to check the correspondence between the rectangular parallelepiped and the identifier, thereby more easily obtaining the identifier associated with each of the sound source S2 and the listener Li2.

Then, by referring to the second table that defines the relationship between the identifiers and the acoustic parameters from the identifiers associated with the sound source S2 and the listener Li2, it is possible to easily obtain various acoustic parameters associated with the sound source S2 and the listener Li2. Based on the obtained acoustic parameters, it is possible to easily output acoustic processing information for generating an acoustic signal that is perceived by the listener.

8 is a process flow showing in more detail step S112 for identifying the first identifier and the second identifier. Each process step will be described below. It is assumed that a first table in which an identifier is associated with each of a plurality of rectangular parallelepipeds is stored in advance in a memory. The first table is an example of first relational data.
[Step S210] Based on the position of the sound source S2 and the position of the listener Li2, a first rectangular parallelepiped and a second rectangular parallelepiped corresponding to the respective positions are searched for.

[Step S212] A first identifier associated with the first rectangular parallelepiped and a second identifier associated with the second rectangular parallelepiped are extracted by referring to the first table using the searched first rectangular parallelepiped and second rectangular parallelepiped. The first identifier is an identifier of a partial space in which a sound source S2 exists, and the second identifier is an identifier of a partial space in which a listener Li2 exists.
In this manner, it is possible to more easily identify the identifiers corresponding to the sound source S2 and the listener Li2.

Fig. 9A shows a process flow for selecting parameters related to sound attenuation, etc., based on identifier information, Fig. 9B is a table defining groups of a plurality of layers.
The process flow of FIG. 9A will be described below.
[Step S310] If the first identifier and the second identifier are different, the first parameter or the second parameter is changed according to the first rule. An example of this case is the presence of an obstruction such as a bridge or a partition. The outline of this case has already been described, but an example of the presence of a bridge will be described below.

When there is an obstacle separating the sound source from the listener, such as a bridge, and the sound source is separated from the listener by the bridge, the sound from the sound source is attenuated by the bridge before it reaches the listener. However, when the sound source and the listener are both under the bridge, there is no sound obstruction by the bridge, so there is no or only a small amount of sound attenuation. The same is true when the sound source and the listener are both on the bridge.
As shown in Fig. 13B, a case is assumed in which an obstruction such as a bridge exists that attenuates sound propagation in the vertical direction. In this case, it is preferable to be able to select a parameter related to sound attenuation corresponding to an identifier assigned to a subspace depending on whether or not the sound source and the listener are both present in the same layer or the same layer group.
When multiple layers are under a bridge, such as when layers L0 to L3 are under the bridge in FIG. 13B, layers L0 to L3 can be defined as forming a layer group.
FIG. 9B is a table that defines layer groups assuming the above example.
Fig. 19A is a table that defines the identifiers of the subspaces and the types of the subspaces, and Fig. 19B is a table that defines the amount of sound attenuation and the cutoff frequency, which is an example of a parameter of a low-pass filter, that are assigned to the types of spaces.
Fig. 9B is a table showing that layers L0 to L3 form layer group 1, and layers 11 and 12 form layer group 2. By referring to Fig. 9B, multiple layers can be treated as a group, and if a sound source and a listener both exist in layers of the same group, the parameters corresponding to the same layer group from the tables of Fig. 19A and Fig. 19B can be adopted.
It is also possible to eliminate the table shown in Fig. 9B and not define layer groups. In this case, if the sound source and the layer where the listener exists match, the parameters of the same layer group may be adopted.
Alternatively, the parameters of the same layer group in FIG. 19B may be adopted by limiting the layer group defined in FIG. 9B.
In this case, effects such as sound attenuation may be provided according to the distance between the sound source and the listener.
For example, when FIGS. 9A and 9B, and FIGS. 19A and 19B are applied to FIG. 13B, the result is as follows.
With reference to Fig. 9B, the sound source S41 and the listener Li4 in Fig. 13B are in different layer groups. The sound source S41 is assigned identifier 1, and the listener Li4 is assigned identifier 7. If it is decided to adopt the parameters of the larger identifier as described below, the parameters of the different layer group of the listener Li4 with identifier 7 in Figs. 19A and 19B are a volume attenuation parameter of -6 dB and a low-pass filter cutoff frequency parameter of 16000 Hz. Therefore, it is sufficient to apply these parameters to the sound from the sound source S41.
Referring to Fig. 9B, the sound source S42 and the listener Li4 in Fig. 13B are in the same layer group. The sound source S42 is assigned an identifier 1, and the listener Li4 is assigned an identifier 7. If it is decided to adopt the parameters of the one with the larger identifier, then in Figs. 19A and 19B, the parameters of the same layer group of the listener Li4 with the identifier 7 are a volume attenuation parameter of 0 dB and a low-pass filter cutoff frequency parameter of "none" (i.e., the same as not applying a low-pass filter). Therefore, there is no sound attenuation (0 dB) and no low-pass filter is applied to the sound from the sound source S42.
Details of Figures 13A and 13B will be described later.

When an obstruction such as a bridge is present, rather than determining whether or not an obstruction is present by applying the above-mentioned processing to the sound from the sound source, it is possible to apply processing according to the propagation of the sound based on the presence or absence of an obstruction by changing the parameters applied depending on the relative positions of the sound source and the listener.
By referring to Figures 19A and 19B, including cases where the sound source and listener are located outdoors, inside a room, in a castle, a cave, etc., it is possible to change the volume attenuation parameters and low-pass filter parameters depending on whether the sound source and listener are in the same layer group or different layer groups.
Note that while Figures 19A and 19B show an example in which the same volume attenuation parameters and low-pass filter parameters are assigned to multiple identifiers, it goes without saying that volume attenuation parameters and low-pass filter parameters may also be assigned to each identifier.

As already mentioned, for sound reflections and environmental sounds in a subspace, the acoustic parameters of the subspace in which the listener is located can be used, and there is no need to consider the effects of obstructions.

FIG. 10 shows a detailed process flow relating to the process of including parameters relating to sound attenuation and the like in the sound processing information.
[Step S410] It is checked whether the first identifier and the second identifier are different. If the first identifier and the second identifier are different (Yes), it is determined that the sound source and the listener are in different subspaces. If the first identifier and the second identifier are the same (No), it is determined that the sound source and the listener are in the same subspace. If this check is Yes, the process proceeds to step S412. If this process is No, the process proceeds to step S414.

[Step S412] The first type of parameters assigned to one of the first and second identifiers selected according to the second rule are included in the acoustic processing information. For example, parameters related to sound attenuation and parameters of a low-pass filter are examples of the first type of parameters. Then, the process returns.
The second rule will be explained later.

[Step S414] The first type of parameters when the sound source and the listener are in the same subspace are included in the acoustic processing information. In this case, since the sound source and the listener are in the same subspace, sound attenuation and low-pass filter processing may not be performed. Alternatively, the distance between the sound source and the listener may be detected, and sound attenuation and low-pass filter processing may be changed according to the distance. Then, the process returns.
The handling of parameters relating to environmental sounds and sound reverberation will be described with reference to FIG.

The second rule will be described below with reference to FIG.
FIG. 11 shows a process flow to which the second rule is applied when the parameters are, for example, sound attenuation and low-pass filter processing.
[Step S510] Parameters corresponding to identifiers having parameters that have a greater effect of shielding sound are included in the sound processing information.
As already mentioned, if the sound processing is sound attenuation or low pass filtering, it is desirable to take the parameter associated with the identifier with the parameter that has the greater amount of attenuation and apply that parameter to the sound. Taking the parameter associated with the identifier with the parameter that has the greater amount of attenuation is an example of a second rule.
By adopting parameters associated with an identifier having parameters with a larger amount of attenuation, the amount of calculation of the amount of sound attenuation can be reduced.
If a parameter with a larger amount of attenuation is associated with a larger identifier, a rule may be used to adopt the parameter related to the attenuation of the sound corresponding to the larger identifier between the first identifier and the second identifier. This rule is an example of the second rule.
Note that parameters relating to sound attenuation and the like associated with the first and second identifiers may be added (or the effects of the parameters may be superimposed) and included in the sound processing information.

FIG. 12 shows a process flow for outputting sound processing information when the sound processing is related to the generation of environmental sounds or the sound reverberation effect.
As already mentioned, when the acoustic processing is processing related to the generation of environmental sounds or the reverberation effect of sounds, it is desirable to generate environmental sounds or add reverberation effects by including parameters related to an identifier corresponding to the subspace in which the listener exists in the acoustic processing information. It is also desirable to apply parameters related to reverberation effects associated with an identifier corresponding to the subspace in which the listener exists to environmental sounds, sounds from a sound source that have been subjected to attenuation processing, and sounds emitted by the listener himself.
In addition, parameters related to the sound reverberation effect associated with the subspace in which the sound source exists (i.e., the identifier corresponding to the sound source) may be applied to the sound from the sound source.

Figures 13A and 13B show how to extract identifiers for each sound source and listener using multiple cuboids in virtual space when a bridge is present outdoors.

Figure 13A shows a bridge 1720 outdoors. The bridge 1720 is built on a riverbank 1750. A listener Li4 is present in a space 1730 directly below the bridge. River water 1740 exists below the bridge. A sound source S41 exists directly above the bridge 1720. A sound source S42 exists at a lower position in the Z-axis direction than the bridge 1720, but is not directly below the bridge 1720.

Fig. 13B shows a cross section taken along the line E-E of Fig. 13A. Fig. 13B shows identifiers assigned to a number of rectangular parallelepipeds filling a three-dimensional space. Identifier 7 is assigned to the rectangular parallelepiped under the bridge 1720. Identifier 1 is assigned to the rectangular parallelepipeds in the outdoor space other than directly under the bridge 1720.
The identifier of the partial space in which the listener Li4 exists is 7, and the identifiers of the partial spaces in which the sound source S41 and the sound source S42 exist are both 1. Note that the listener Li4 or the sound source S42 spans multiple rectangular parallelepipeds. When the identifiers given to the multiple rectangular parallelepipeds are different, for example, the identifier given to the rectangular parallelepiped that overlaps largely with the listener Li4 or the sound source S42 may be associated with the listener Li4 or the sound source S42.
In this case, the first rule for changing the first parameter or the second parameter is as follows:

(First rule: Example of a bridge) Even if the combination of the identifiers of listener Li4 and sound source S42 is 1 and 7, if the layers to which listener Li4 and sound source S42 belong are both layers L0 to L3, they are determined to belong to the same layer group. In this case, listener Li4 and sound source S42, which belong to the same layer group, are both located at approximately horizontal positions, and there is no bridge between them that blocks the propagation of sound.

Even if the combination of identifiers for listener Li4 and sound source S41 is 1 and 7, if the layer to which one of listener Li4 and sound source S41 belongs is one of layers L0 to L3 and the layer to which the other belongs is other than layers L0 to L3, then it is determined that the two belong to different layer groups.

By applying the above first rule, even in a space where a bridge or the like is present outdoors 1710 (or inside a room), the sound attenuation or low-pass filter parameters can be appropriately changed by taking into account the presence of the bridge and referring to the tables shown in Figures 9B and 19B.
As already mentioned, in either case, the acoustic processing related to the environmental sound and the sound reflection can be performed by applying the parameters related to the identifier 7 associated with the listener Li4. The environmental sound can be the sound of a flowing river, etc. The parameters related to the sound reflection can be parameters that simulate the sound reflection under a bridge.
<Embodiment 3>

Figure 14 shows the process of changing the first parameter or the second parameter according to the first rule when the first identifier and the second identifier are different. Each flow in Figure 14 is explained below.

[Step S710] It is determined whether the sound source and listener are in the same layer group. If the determination is Yes, processing proceeds to step S712. If the determination is No, processing proceeds to step S714.

[Step S712] When the first identifier and the second identifier are different and the parameters related to sound attenuation, etc. are related to the case where the sound source and the listener are in the same layer group, corresponding first parameters and second parameters are searched for.
This process makes it possible to use the corresponding first and second parameters when the sound source and the listener are in the same layer group.
[Step S714] When the first identifier and the second identifier are different and the sound source and the listener are in different layer groups, corresponding first and second parameters are searched for with respect to parameters related to sound attenuation, etc.
This process allows the first parameter and the second parameter to be changed depending on whether the sound source and the listener are in different layer groups.
The process in FIG. 14 is an example of the first rule.

<Embodiment 4>
(First rule: Example of a partition)
Fig. 15 shows an example of another process for changing the first parameter or the second parameter according to the first rule when the first identifier and the second identifier are different. This process flow is a process flow for a case where a partition is present as shown in Figs. 16A, 16B, 17A, 17B, 18A, and 18B.

[Step S812] It is determined whether the first identifier and the second identifier have a predetermined relationship. If this check is positive (Yes), the process proceeds to step S814. If this check is negative (No), the process returns.

[Step S814] With regard to parameters relating to sound attenuation and the like, if the first identifier and the second identifier match, the parameters are determined as the first parameter and the second parameter.
In this case, the first and second parameters are not obtained from the table shown in Fig. 19B, but the parameters when the first and second identifiers match (i.e., when the sound source and the listener exist in the same subspace) are set as the first and second parameters. By doing so, even if the first and second identifiers are different, the sound source and the listener belong to the same subspace, and the same processing can be performed as when there is no obstruction between them.
The process in FIG. 15 is an example in which the first rule is applied to a partition.
A specific example of the process shown in FIG. 15 will be described below using a partition as an example.

16A and 16B show an example in which a partition 180 exists in one partial space.
16A, a listener Li5 is located in front of a partition 180. A sound source S51 is located on the opposite side of the partition 180. A sound source S52 is located to the side of the partition 180.

Figure 16B is an XY plane view crossing listener Li5 in the virtual space of Figure 16A. The mesh shown in Figure 16B is a three-dimensional mesh in which the boundaries formed by each rectangular solid are indicated by dashed lines when viewed from the Z-axis direction.

As shown in FIG. 16B, a rectangular parallelepiped with identifier 5 exists on the listener Li5 side of the partition 180. In addition, a rectangular parallelepiped with identifier 6 exists on the sound source S51 side of the partition 180. The surrounding rectangular parallelepipeds other than these rectangular parallelepipeds have been assigned identifier 1. In this case, identifiers 1, 5, and 6 have been assigned to the rectangular parallelepipeds that share the outdoor space. A barrier (partition 180) exists between the space in which the rectangular parallelepiped with identifier 5 exists and the space in which the rectangular parallelepiped with identifier 6 exists. There are no obstructions between the space in which the cuboid with identifier 5 exists and the space in which the cuboid with identifier 1 exists, and between the space in which the cuboid with identifier 6 exists and the space in which the cuboid with identifier 1 exists. (Note that there are also combinations of cuboids with identifier 1 in which there is an obstruction (partition 180), but because the partition 180 is far away between these cuboids, taking into account sound deflection, it is possible to perform a more realistic simulation of sound propagation by assuming that no obstruction exists.)

17A and 17B show identifiers that are assigned to each rectangular parallelepiped that exists in the virtual space near a partition 180. FIG.
As shown in FIG. 17A , a subspace 1902 exists on one side near a partition 1900, and a subspace 1904 exists on the other side near the partition 1900.

17B , an identifier 5 is assigned to each of a set of rectangular parallelepipeds 1952 that overlap with partial spaces 1902 located on both sides of partition 1900. An identifier 6 is assigned to each of a set of rectangular parallelepipeds 1954 that overlaps with partial space 1904.
By doing so, it is possible to associate the identifier 1, the identifier 5, or the identifier 6 with the sound source S51 or the sound source S52 and the listener Li5.

Returning to FIG. 16B, if identifier 5 is associated with the position of listener Li5 and identifier 6 is associated with sound source S51, processing may be performed to indicate the presence of partition 180 (i.e., an obstruction).

In contrast, if identifier 5 is associated with the position of listener Li5 and identifier 1 is associated with sound source S52, processing may be performed in the same manner as if partition 180 (i.e., an obstruction) were not present.

Alternatively, when identifier 1 is associated with the position of listener Li5 and identifier 1 is also associated with sound source S52 (not shown), processing can be performed in the same manner as if partition 180 (i.e., an obstruction) were not present.

By doing this, even when sound source S51 or sound source S52 and listener Li5 are present in the same subspace, it is possible to perform acoustic processing that takes into account the presence of objects that attenuate sound propagation, such as partitions.

Figure 18A shows detailed processing for determining whether a first identifier and a second identifier have a specified relationship (step S710 in Figure 15). Figure 18B shows a table used to determine when a combination of a first identifier and a second identifier has a specified relationship. Each step of the processing flow in Figure 18A is explained below.

[Step S912] It is determined whether the combination of the first identifier and the second identifier exists in a predetermined table. If the result of the determination is positive, for example, as shown in table 1800 in Fig. 18B, the combination of identifiers is 1 and 5 or 1 and 6 (Yes), the process proceeds to step S914. If the result of the determination is negative, for example, the combination of identifiers is 5 and 6 (No), the process returns.
[Step S914] It is determined that the first identifier and the second identifier have a predetermined relationship.
According to the processing flows of Figures 15 and 18A, if the combination of identifiers is 1 and 5, or 1 and 6, it is determined that the first identifier and the second identifier have a predetermined relationship, and in step S814 of Figure 15, the parameters when the first identifier and the second identifier match are set as the first parameter and the second parameter.
By the above process, if the sound source S51 is associated with the identifier 6 and the listener Li5 is associated with the identifier 5, parameters related to sound attenuation and the like associated with the identifier 6, which has the larger identifier value, are searched for in tables similar to those in Figures 19A and 19B. Note that in this case, it is not necessary to determine whether the layer groups are different.
As a modified example, when there is a space having a structure that combines a partition and a bridge, parameters related to sound attenuation, etc. may be searched for taking into account the differences and similarities between layer groups (not shown).
The process flow in FIG. 18A using table 1800 in FIG. 18B is an example of the first rule.
In this manner, even when a partition or the like is present in the partial space, processing related to sound propagation can be performed appropriately.
If the identifier 1 and the identifier 2 corresponding to the sound source and the listener, respectively, are both "1", the result in step S410 in FIG. 10 is No, and the process in step S414 is performed.

<Embodiment 5>
19A shows a table 1900 that defines identifiers of partial spaces and types of partial spaces. A plurality of identifiers may be assigned to each of outdoors, inside a room, a castle, a cave, and the like.

Figure 19B shows a table 1901 that defines sound attenuation parameters and cutoff frequencies, which are examples of low-pass filter (LPF) parameters, assigned to space types.

Table 1901 shows two parameters, one for the same layer group and one for different layer groups. By preparing a table in which sound attenuation and low-pass filter parameters differ depending on whether the sound source and listener are in different layer groups, it is possible to process sound propagation when a bridge or the like is present, as already mentioned.
It is preferable to store a table (eg, FIG. 9B) that defines layer groups in advance in memory.

20A shows a table 2000 showing the relationship between the identifier value and the size of the subspace. The size of the subspace may be expressed by the length of one side of a rectangular parallelepiped having the same volume as the subspace. Alternatively, the size of the subspace may be expressed by, for example, the volume of the subspace.
FIG. 20B shows a table 2001 in which names representative of environmental sound definitions and reverb parameters corresponding to the sizes of subspaces are defined.

The environmental sounds shown in Table 2001 are examples. Also, reverb is an example of an effect parameter that indicates the reverberation of sound. The reverberation of sound is not limited to reverb, but may be echo or delay, etc.

It goes without saying that the sound effects are not limited to the examples given above, and various sound effects such as a high-pass filter, a phase shifter, distortion, and chorus can be used.
Table 1900, table 1901, table 2000, and table 2001 are examples of the second relational data.

In addition, in the above embodiment, sound was used as an example, but vibration may also be used.

Figure 21 shows the hardware configuration of the embodiment. The hardware configuration includes a CPU 1001, a ROM 1002 in which the program and data of the embodiment can be stored, a RAM 1003, a network interface 1005, an input interface 1006, a display interface 1007, and an external memory interface 1008. These pieces of hardware are interconnected by a bus 1004.

The network interface 1005 is connected to a network 1015. The network 1015 may be a wired LAN, a wireless LAN, the Internet, a telephone network, or the like. The input interface 1006 is connected to an input unit 1016. The display interface 1007 is connected to a display unit 1017. The external memory interface 1008 is connected to a storage medium 1018. The storage medium 1018 may be a RAM, a ROM, a CD-ROM, a DVD-ROM, a hard disk, a memory card, a USB memory, or the like.
The program and method for implementing the above-described embodiment can be executed by a computer having the hardware configuration shown in FIG.
It should be noted that there may be a plurality of listeners and a plurality of sound sources.
Furthermore, the first relationship data may be changed when the partial spaces are connected, for example, when a door separating adjacent partial spaces is opened. Alternatively, first relationship data assuming a case where a door is opened may be prepared separately, and when the door is opened, a pointer indicating the position of the first relationship data in memory may be changed, thereby appropriately switching between different first relationship data for use.
The above embodiments are illustrative of the invention defined in the claims and should not be construed as limiting.

1004 Bus 1005 Network interface 1006 Input interface 1007 Display interface 1008 External memory interface 1015 Network 1016 Input unit 1017 Display unit

Claims (10)

1. An acoustic signal processing method for outputting acoustic processing information used to obtain an acoustic signal including a sound signal perceived by a listener when a sound source and a listener exist in a virtual three-dimensional space including a plurality of subspaces and a sound emitted from the sound source propagates through the three-dimensional space and reaches the listener, the method comprising:
a first relational data defining a correspondence relationship between a plurality of subspaces included in the three-dimensional space and identifiers assigned to each of the plurality of subspaces, and a second relational data defining a correspondence relationship between the identifiers and parameters of signal processing for obtaining the acoustic signal, are prepared in a memory;
The acoustic signal processing method includes:
receiving a position of the sound source and a position of the listener;
identifying a first identifier corresponding to the sound source and a second identifier corresponding to the listener by referring to the first relationship data using the position of the sound source and the position of the listener;
referring to the second relationship data using the first identifier and the second identifier to search for a first parameter corresponding to the first identifier and a second parameter corresponding to the second identifier;
outputting acoustic processing information based on at least the first parameter and the second parameter so as to obtain the acoustic signal;
An acoustic signal processing method comprising: the three-dimensional space is divided into a plurality of rectangular parallelepipeds having vertices at four mutually adjacent intersections of a three-dimensional grid formed of a plurality of grid lines parallel to each of X, Y and Z axes which are orthogonal coordinate axes placed in the three-dimensional space, an identifier assigned to the partial space is associated with each of a set of the rectangular parallelepipeds which overlaps with the partial space, and the first relationship data is constituted by defining a correspondence relationship between each of the plurality of rectangular parallelepipeds and an identifier;
The identifying step includes:
Searching for a first rectangular parallelepiped located at the position of the sound source and a second rectangular parallelepiped located at the position of the listener;
extracting the first identifier associated with the first rectangular parallelepiped and the second identifier associated with the second rectangular parallelepiped by referring to the first relationship data;
The method of claim 1 , further comprising: The outputting step includes:
if the first identifier and the second identifier are different, modifying the first parameter or the second parameter according to a first rule;
Including,
3. The method for processing an acoustic signal according to claim 1. The acoustic signal processing method according to any one of claims 1 to 3, wherein the output includes, in the acoustic processing information, a first type of parameter assigned to one identifier selected according to a second rule from among the first identifier and the second identifier when the first identifier and the second identifier are different. The acoustic signal processing method according to claim 4, wherein the second rule is a rule for selecting an identifier having a parameter that has a greater effect of shielding acoustic signals. The acoustic signal processing method according to any one of claims 1 to 5, wherein the output includes, among the parameters assigned to the second identifier, parameters related to environmental sound or sound reverberation included in the acoustic processing information. The acoustic signal processing method according to claim 3, wherein the first rule is a rule for determining whether the positions of the sound source and the listener are in a predetermined positional relationship. The acoustic signal processing method according to claim 3, wherein the first rule is a rule for determining whether the first identifier and the second identifier have a predetermined relationship. 9. An audio signal processing program for causing a computer to execute the audio signal processing method according to claim 1. A storage medium storing the acoustic signal processing program according to claim 9 .

Images