JP7847447B2 - Acoustic signal processing device, and program - Google Patents

Description

This invention relates to an acoustic signal processing device and a program, and more particularly to a technique for realizing binaural playback.

Augmented Reality (AR) and Virtual Reality (VR) content is sometimes created or presented by rendering sound in accordance with the movements of video objects and the viewer. Sound rendering is used to create an effective sense of presence. AR/VR content is sometimes created with the assumption that it will be presented while wearing headphones or a head-mounted display (HMD) with built-in headphones. Binaural playback technology may be employed as a technique to virtually recreate a three-dimensional sound space using headphones.

Binaural playback is achieved by reproducing the sound pressure obtained when sound waves emitted from a sound source placed at a predetermined position reach the entrance of the ear canal of both ears of the listener. In binaural playback, sound is presented directly using playback sound sources close to each ear, such as headphones or earphones. The sound transmission characteristics from the sound source to each ear include the effects of reflection, diffraction, and attenuation of sound waves in the listener's head, auricle, torso, etc. (hereinafter collectively referred to as "head, etc."). By presenting sound with these transmission characteristics added, virtually placed at a set position, binaural playback can provide the listener with a high sense of presence.

In binaural playback, the Head-Related Transfer Function (HRTF) or Head-Related Impulse Response (HRIR) is used as a feature quantity that indicates the transfer characteristics from the sound source position in space to the listening position (receiving point). While HRTF is expressed in the frequency domain, HRIR is the time-domain representation of HRTF. Generally, HRTF or HRIR is measured in a special acoustic environment such as an anechoic chamber. In the measurement, sound based on the measurement signal is emitted from speakers placed around the listener, and the sound is captured using a microphone placed at the entrance of the listener's ear canal. The HRTF or HRIR (hereinafter collectively referred to as "HRIR, etc.") is obtained using the captured signal and a known measurement signal.

A binaural signal is a two-channel acoustic signal containing a left-ear acoustic signal (hereinafter referred to as the "left-ear signal") obtained by convolving the left ear's HRIR into the input acoustic signal (hereinafter referred to as the "input signal"), and a right-ear signal obtained by convolving the right ear's HRIR into the input signal. Each ear's acoustic signal (hereinafter referred to as the "each ear's signal") y is obtained by performing a convolving operation with the input signal using HRIR, as illustrated in equation (1). In equation (1), y(t), x(t), and h(t) represent the sample values of each ear's signal, the input signal, and the HRIR at time t, respectively.

Each ear signal y(t) can also be obtained by calculating the product Y(ω) of the Fourier transform X(ω) of the input signal and HRTF H(ω), as exemplified in equation (2), and then performing an inverse Fourier transform on the product Y(ω). In equation (2), ω represents frequency.

As described above, binaural playback is achieved by presenting sounds based on individual ear signals to the corresponding ears using headphones. The sound pressure of the sound presented to the external auditory canals of both ears is equivalent to the sound pressure from sound waves arriving from the sound source location used in measurements such as HRIR. Therefore, the listener can perceive a virtual sound image at that sound source location.

In recent years, the notation of sound fields based on spatial Fourier series expansion has attracted attention as a technique for representing acoustics in three-dimensional space. A representative example is a method based on spherical harmonic expansion (for example, Non-Patent Document 1). This method is based on the fact that the sound pressure distribution p(r, θ, φ, ω) in three-dimensional space can be expressed as a linear combination of basis functions of the spherical harmonic expansion, as shown in equation (3). In other words, the spherical harmonic expansion describes the sound pressure distribution in three-dimensional space in a form in which the variables are separated into radial and angular components. Equation (3) corresponds to the general solution of the three-dimensional wave equation given in polar coordinates by transforming the variables of the three-dimensional wave equation given in Cartesian coordinates. In equation (3), (r, θ, φ) represents the three-dimensional coordinates expressed in polar coordinates. _{h n} ⁽²⁾ represents the nth-order second-kind spherical Hankel function. The second-kind spherical Hankel function gives the orthogonal basis for the radial r component. Y _n ^m represents the nth-th order m-th spherical harmonics. Spherical harmonics provide an orthogonal basis in the angular direction. _{A n} ^m represents the spherical harmonic spectrum. The spherical harmonic spectrum can represent the spatial distribution of a sound field, such as the directivity of a sound source, by using the product of the second kind spherical Hankel function and the spherical harmonics as weighting coefficients for the basis functions of the spherical harmonic expansion. To date, applications of the spherical harmonic spectrum to sound field control, such as rotation of a virtual sound source in an arbitrary direction and interpolation of sound pressure at an arbitrary position, have been proposed.

Ambisonics (e.g., Non-Patent Document 2) is a known technology that utilizes this principle. This technology is also employed and standardized in MPEG-H 3DA (Non-Patent Document 3), a next-generation speech encoding scheme. Furthermore, a method has been proposed to encode binaural signals using ambisonics with HRTF measured in three-dimensional space (Patent Document 1).

Patent No. 6067934

Yoichi Haneda, "Signal Processing in the Wavenumber Domain of Sound," Fundamentals Review, IEICE Fundamentals and Boundary Society, Vol. 11, No. 4, pp. 243-255, 2017. D. H. Cooper, T. Shiga, Discrete-matrix multichannel stereo. Journal of Audio Engineering Society 20(5), pp.346-360, 1972. ISO/IEC 23008-3:2019 “Information technology - High efficiency coding and media delivery in heterogeneous environments - Part 3: 3D audio, Second edition” (2019) A. V. Oppenheim, R. W. Schafer, Digital signal processing, Englewood Cliffs, N.J.: Prentice-Hall, 1975. P. A. Martin, Multiple Scattering: Interaction of Time-Harmonic Waves with N obstacles, Cambridge university press, 2006.

HRIRs (Hearing, Resonance, and Infrared) are determined for each pair of sound source and listening point within the acoustic space. In binaural playback using a single pair of HRIRs, a sound image is perceived as stationary at the sound source location, assuming the listener is present at that location. To represent the movement of the sound image or the listener, it is common practice to switch HRIRs according to changes in the relative positions of the sound source and listening point. However, since the number of sound source and listening point pairs involved in HRIR measurement is finite, even if the sound source location is fixed, the positions of the sound image that can be represented are limited. Simply switching HRIRs makes it difficult to move the listening point along an arbitrary trajectory and allow the listener to perceive a smoothly moving sound image.

In binaural playback using HRTF, the sound source signal is Fourier transformed buffer by buffer, and rendering is performed by multiplying it with the HRTF in the frequency domain. Each buffer uses a different HRTF, representing a different sound source position or listening point. By presenting a sound mixed from the output signals of at least two buffers, the sound source position or listening point of the sound image is linearly approximated between the buffers. To perceive a smoothly moving sound image, rendering in time-sample units is desirable. Furthermore, mixing output signals based on different HRTFs in the time domain may disrupt the frequency characteristics of individual HRTFs, which serve as cues for sound image localization.

One of the objectives of this invention is to render sound in the frequency domain when the sound source or listening point moves arbitrarily.

[1] One aspect of the present invention is an acoustic signal processing device comprising: a spherical harmonic spectrum calculation unit that calculates conversion coefficients from basis functions of a spherical harmonic expansion at a sound source position with respect to the listening point to basis functions at a sound source position with respect to the origin, and for each ear, calculates a spherical harmonic spectrum of a sound field based on the spherical harmonic spectrum of the head-related transfer function, the conversion coefficients, and the sound source signal; and a binaural signal generation unit that calculates a sound pressure spectrum for each ear by linearly combining the basis functions using the spherical harmonic spectrum of the sound field, and converts the sound pressure spectrum into an acoustic signal in the time domain.
According to the configuration of [1], for each ear, a spherical harmonic spectrum representing the sound field at the listening point is calculated without recalculating the spherical harmonic spectrum of the head-related transfer function in the frequency domain. By linearly combining the basis functions in the spherical harmonic expansion using the calculated spherical harmonic spectrum, it is possible to render in the frequency domain a sound in which the sound source or listening point moves arbitrarily. Through rendering, the frequency characteristics of the sound continuously change in accordance with the smooth fluctuation of the sound source position or listening point.

[2] One aspect of the present invention is the above-described acoustic signal processing device, wherein the listening point is the head position, and the reference point for acquiring the head-related transfer function is the head position.
According to the configuration in [2], using the head position as the listening point simplifies the rendering operations based on the listening point compared to when the position of each ear is used. Furthermore, using the head position as the reference point for acquiring the head-related transfer function makes it easier to acquire and manage the head-related transfer functions for each ear simultaneously.

[3] One aspect of the present invention is the above-described acoustic signal processing device, wherein the listening point is the position of each ear, and the reference point for acquiring the head-related transfer function is the position of each ear.
According to the configuration of [3], the position of each ear is used as a listening point, and that position is used as the expansion center for the spherical harmonic expansion of the head-related transfer function. Therefore, the estimation accuracy of the calculated sound pressure spectrum can be improved compared to when the head position is used as the listening point.

[4] One aspect of the present invention is a program that causes a computer to function as the above-described acoustic signal processing device.
According to the configuration in [4], the position of each ear is used as the listening point, and that position is used as the expansion center for the spherical harmonic expansion of the head-related transfer function. Therefore, the estimation accuracy of the calculated sound pressure spectrum can be improved compared to when the center of the head is used as the listening point.

According to this invention, it is possible to render sound in the frequency domain even when the sound source or listening point moves arbitrarily.

This is a schematic block diagram showing an example of the functional configuration of the acoustic signal processing device according to the first embodiment. This figure illustrates the global coordinate system and local coordinate system according to the first embodiment. This is a flowchart illustrating the acoustic signal processing according to the first embodiment. This figure illustrates the global coordinate system and local coordinate system according to the second embodiment. This is a flowchart illustrating the acoustic signal processing according to the second embodiment.

<First Embodiment>
Embodiments of the present invention will be described below with reference to the drawings. First, an example of the functional configuration of the acoustic signal processing device 10 according to the first embodiment will be described. Figure 1 is a schematic block diagram showing an example of the functional configuration of the acoustic signal processing device 10 according to this embodiment.
The acoustic signal processing device 10 receives the coordinates of the listening point and the sound source signal for each ear (left and right). The acoustic signal processing device 10 calculates conversion coefficients from the basis functions of the spherical harmonic expansion at the sound source position relative to the listening point to the basis functions at the sound source position relative to the origin. For each ear, the acoustic signal processing device 10 calculates the spherical harmonic spectrum of the sound field based on the spherical harmonic spectrum of the head-related transfer function, the conversion coefficients, and the sound source signal. For each ear, the acoustic signal processing device 10 calculates the sound pressure spectrum by linearly combining the basis functions using the spherical harmonic spectrum of the sound field, and converts the sound pressure spectrum into a time-domain acoustic signal.

The acoustic signal processing device 10 comprises an input unit 110, a control unit 120, a storage unit 130, and an output unit 140.
The input unit 110 receives listening point information indicating the coordinates of the listening point and a sound source signal. The input unit 110 outputs the listening point information and the sound source signal to the control unit 120. The listening point is, for example, the position of the listener's head in three-dimensional space (hereinafter referred to as "head position"). For example, the position of the center of the head is indicated as the head position. The time series of the listening point for each time corresponds to the movement trajectory. The input unit 110 may input and output listening point information for each time, or it may input and output listening point information indicating the movement trajectory over a certain period. The input unit 110 may acquire the sound source signal for each time, or it may acquire the sound source signal for that period all at once. The input unit 110 is, for example, an input interface.

The control unit 120 performs processing to realize the functions of the acoustic signal processing device 10. The control unit 120 includes a spherical harmonic spectrum calculation unit 122, a binaural signal generation unit 124, and a spherical harmonic expansion unit 126.

The spherical harmonic spectrum calculation unit 122 calculates, for each time τ, the conversion coefficient S n m ν μ ([ _{r h} (τ)]) or S' _n ^m _ν ^μ ([r _h (τ)]) for each frequency from the basis function in the n-th ^order m- _th order ^spherical harmonic expansion at the head position [r _h (τ)] as the listening point. In this text, [...] indicates a vector. However, in the formulas described later, vectors are shown in bold. The head position [r _h (τ)] is expressed in three-dimensional spherical coordinates [r _h (τ), θ _h (τ), φ _h (τ)]. _{r h} (τ), θ _h (τ), and φ _h (τ) represent the radial, polar, and azimuthal angles of the head position, respectively. The transformation coefficients S _n ^m _ν ^μ ([r _h (τ)]) and S' _n ^m _ν ^μ ([r _h (τ)]) are obtained using the addition theorem for spherical Bessel functions (described later in equation (7)) to obtain an nth-order m- _th basis function, which is the product of the nth-order (n is an integer between 0 and N, and N is a predetermined integer between 0 and N) spherical Bessel function j _n (kr _h (τ)) (k is the wavenumber) at head position r h (τ) and the nth-order m-th (m is an integer between -n and N) spherical harmonic function Y _n ^m θ _h (τ), φ _h (τ)) at sound source position r in a global coordinate system covering the entire sound field, and the ν-order (ν is an integer between 0 and N) second kind spherical Hankel function h _ν ⁽²⁾ (kr) and the νth-order μ-th (μ is an integer between -ν and ν) spherical harmonic function Y _ν ^μ This corresponds to a weighting coefficient determined such that the order of the ν-th order μ-th basis function of the spherical harmonic expansion, which is the _product of (θ, φ) or the product of the ν-th order spherical Bessel function j _ν (kr) and the ν-th order μ-th spherical harmonic function Y ν ^μ (θ, φ), and the weighted sum between the orders are equal. The spherical harmonic spectrum calculation unit 122 calculates the transformation coefficient S _n ^m _ν ^μ when r < r _s (τ), and calculates the transformation coefficient S' _n ^m _ν ^μ when r > r _s (τ). r and r _s represent the radial movement of the sound source position in the global coordinate system and the radial movement of the sound source position in the local coordinate system with respect to the listening point, respectively. In other words, the conversion coefficients S _n ^m _ν ^μ ([r _h (τ)]) and S' _n ^m _ν ^μ ([ _{r h} (τ)]) indicate the degree of contribution of each basis function in the conversion from the nth-order m-th basis function of the spherical harmonic expansion at the sound source position [r _s ] relative to the listening point [r h (τ)] to a weighted sum of the ν-order μ-th basis functions at the sound source position [r] relative to the origin.

The spherical harmonic spectrum calculation unit 122 performs a Fourier transform on the converted sound source signal z(t) (described later), which is obtained by multiplying the sound source signal s(τ) input from the input unit 110 by the calculated conversion coefficient S _n ^m _ν ^μ ([ _{r h (} τ)]) or S' _n ^{m ν} _μ ⁽ [r h (τ)]), and converts it into a converted sound source spectrum Z _n ^m _ν ^μ (ω) in the frequency domain.
The spherical harmonic spectrum calculation unit 122 reads the spherical harmonic spectrum α _n ^m (ω) of the HRTF for each ear from the memory unit 130, and multiplies the read spherical harmonic spectrum α _n ^m (ω) of the HRTF by the converted sound source spectrum to calculate the spherical harmonic spectrum P _ν ^μ (ω) of the sound field. The spherical harmonic spectrum P _ν ^μ (ω) of the sound field is calculated using equation (13) (described later). The spherical harmonic spectrum calculation unit 122 outputs the spherical harmonic spectrum P _ν ^μ (ω) of the sound field calculated for each ear to the binaural signal generation unit 124.

The binaural signal generation unit 124 uses the spherical harmonic spectrum P _ν ^μ (ω) of the sound field input from the spherical harmonic spectrum calculation unit 122 as weighting coefficients, and calculates the sound pressure spectrum P(r,ω) as a weighted sum using the product of the spherical Bessel function j _ν (kr) and the spherical harmonic function Y _ν ^μ (θ,φ) or the product of the second kind spherical Hankel function h _ν ⁽²⁾ (kr) and the spherical harmonic function Y _ν ^μ (θ,φ) as basis functions. The sound pressure spectrum P(r,ω) is calculated using equation (12) (described later). The binaural signal generation unit 124 performs an inverse Fourier transform on the frequency domain sound pressure spectrum P(r,ω) calculated for each ear to generate time domain acoustic signals as signals for each ear. The binaural signal generation unit 124 outputs the binaural signal consisting of each ear signal as an output signal via the output unit 140.

The spherical harmonic expansion unit 126 acquires the HRTF for each pair of listening point and sound source position for each left and right ear, performs a spherical harmonic expansion on the acquired HRTF, and pre-calculates a common spherical harmonic spectrum α _n ^m (ω) for the listening point and sound source position pairs. Each HRTF is expressed as a linear combination using the product of an nth-order second-kind spherical Hankel function and an nth-order m-th spherical harmonic function as basis functions by spherical harmonic expansion, as shown in equation (6) (described later). The spherical harmonic expansion unit 126 can calculate a common weight coefficient as the spherical harmonic spectrum α n m (ω) for the listening point and sound source position pairs such that the dependence in the radial direction is explained by the nth-order second-kind spherical Hankel function and the dependence in the angular direction is explained by _{the nth-order} ^m -th spherical harmonic function. The spherical harmonic expansion unit 126 stores the HRTF data showing the calculated spherical harmonic spectrum α _n ^m (ω) in the storage unit 130.

The storage unit 130 stores data used for processing in the control unit 120 and data acquired by the control unit 120. The storage unit 130 is configured to include storage media such as RAM (Random Access memory) and ROM (Read Only Memory).
The output unit 140 outputs the binaural signal input from the binaural signal generation unit 124 to the outside. The output unit 140 is, for example, an output interface. The output unit 140 may be integrated with the input unit 110 and configured as an input/output interface.

According to the method described above, the spherical harmonic spectrum of the HRTF based on the listening point, which is the origin of the local coordinate system, is converted to the spherical harmonic spectrum of the HRTF based on the origin of the global coordinate system, and separated into radial and angular components. Then, the converted sound source spectrum is obtained from the conversion coefficients of the basis functions from the local coordinate system to the global coordinate system and the sound source signal. Furthermore, the converted sound source spectrum and the spherical harmonic spectrum of the HRTF common to the sound source position and the listening point are used to convert it into a sound pressure spectrum. As a result, the sound source signal with the HRTF convolved is interpolated in the frequency domain for any listening point and sound source position. Since the frequency characteristics of the sound (spectral cue), which are clues for sound image localization, are not disturbed by the calculation, more reliable sound image localization to the sound source position can be expected for the listener. In contrast, in the overlapping summation method described in Non-Patent Literature 4, binaural signals relating to different target directions are added in the time domain. Interference occurs between the two due to the phase difference of the HRTF contained in each binaural signal. Due to interference-induced distortion of frequency characteristics, it was sometimes impossible to achieve sound image localization in the target direction. This embodiment can serve as a solution to this problem.

Next, the acoustic signal processing according to this embodiment will be described in more detail. This embodiment is based on the spherical harmonic expansion of HRTF in three-dimensional space. Here, the global coordinate system that covers the entire sound field including the playback sound source and the listener, and the local coordinate system with the listener's head position as the origin will be explained using Figure 2. [r] represents a vector representing an arbitrary position S from the origin O in the global coordinate system, [r _h (t)] represents the position H of the head moving along an arbitrary trajectory at time t (hereinafter sometimes referred to as "head position"), and [r _s (t)] represents a vector representing an arbitrary position S in the local coordinate system with the head position H as the origin. The vectors [r], [r _h (t)], and [r _s (t)] can be expressed as three-dimensional spherical coordinates [r (t), θ (t), φ (t)], [r _h (t), θ _h (t), φ _h (t)], and [r _s (t), θ _s (t), φ _s (t)], respectively.

In HRTF measurement, assuming that position S is the sound source position where the measurement speaker is installed, the transfer function representing the frequency domain transfer characteristics of sound from sound source position S to head position H corresponds to HRTF, and the impulse response representing that transfer characteristic in the time domain corresponds to HRIR. In this embodiment, the movement of the head position H as the listening point is described as "the movement of the head with directivity represented by a spherical harmonic spectrum," obtained by expanding HRTF using spherical harmonics. Note that since HRTF is determined by the relative positional relationship between the sound source and the head, it may be considered as the movement of a target point instead of the listening point. The target point refers to the position where the sound source is virtually installed and which serves as the target for sound image localization.

Assume that in the global coordinate system, the listening point H moves along an arbitrary trajectory, and sound is emitted from the sound source position S. In this case, the impulse response g([r] - [r _h (τ)]) from the sound source position S (coordinate [ _r ]) to the listening point H (coordinate [r h (τ)]) at time τ is a time-varying impulse response. As shown in equation (4), the binaural signal p([r], t) at time t is obtained by performing a convolution operation on the sound source signal s(t) using the impulse response g([r] - [r _h (τ)]). The spectrum of the binaural signal in the frequency domain is obtained by performing a Fourier transform on the binaural signal p([r], t) in the time domain, as shown in equation (5).

Since HRTF is measured using a microphone placed on the head, it is represented as a distribution around the head by performing a spherical harmonic expansion in a local coordinate system with the head position as the origin. The transfer function in the frequency domain from the sound source position S (coordinate [r _s ]) to the head position H (coordinate [r _h ]) in the local coordinate system, i.e., the HRTF, is expressed as G([r _s ] - [r _h (τ)], ω). According to the spherical harmonic expansion, as shown in equation (6), the HRTF is transformed into a linear combination with basis functions being the product of an nth-order second kind spherical Hankel function and an nth-order m-order spherical harmonic function, that is, a weighted sum that crosses order and degree. Therefore, the HRTF is represented by a spherical harmonic spectrum α _n ^m (ω) consisting of weight coefficients multiplied by each individual basis function. Since HRTF is acquired separately for the left and right ears, the spherical harmonic spectrum α _n ^m (ω) also differs between the left and right ears.

According to the spherical harmonic expansion shown in equation (6), the head position H, which is the center of the expansion, changes over time. Therefore, the orthogonal basis functions, the second kind of spherical Hankel function and the spherical harmonics, also need to be recalculated each time the head position H moves. In this embodiment, the expansion center is shifted to the origin of the global coordinate system using the addition theorem of spherical Bessel functions, and the sound field is represented using orthogonal basis functions relating to a fixed sound source position, regardless of the head position H. The addition theorem of spherical Bessel functions states that when [r ₂ ] = [r _{1 ]} + [ _b ] (where _b is the coordinate between the three-dimensional coordinates [r ₁ ] and [r 2]) between the three-dimensional coordinates [r ₁ ] (= [r ₁ , θ ₁ , φ ₁ ]) and [r ₂ ] (= [r ₂ , θ 2, φ 2]), the relationship shown in equation (7) holds. In equation (7), the conversion coefficients S _n ^m _ν ^μ ([b]) and S' _n ^m _ν ^μ ([b]) are given by equations (8) and (9), respectively.

In equations (8) and (9), Y ^* _q ^{μ - m} (θ _b , φ _b ) represents the complex conjugate of the spherical harmonic function Y _q ^{μ - m} (θ _b , φ _b ). _W1 and _W2 represent the Wigner 3j notation in equation (10).

By applying the addition formula for spherical Bessel functions to the HRTF shown in equation (6) and shifting the expansion center of the spherical harmonics to the origin O of the global coordinate system, the HRTF can be expressed as a linear combination of basis functions relating to the sound source position in the global coordinate system. Specifically, by substituting the coordinates of the sound source position [r] and the head position [ _{r h (} τ)] relative to the origin O in the global coordinate system into the three-dimensional coordinates [r 1] and [b] of equation (7), respectively, and applying this to equation (6), the HRTF is transformed as shown in equation (11). Then, substituting the HRTF expressed in equation (11) into equation (5), the sound pressure spectrum P([r], ω) is given as shown in equation (12).

In equation (12), the spherical harmonic spectrum _{P ν} ^μ (ω) of the sound field, which is multiplied by the basis function that is the product of the spherical Bessel function ^j _ν (kr) and the spherical harmonic function Y ν μ (θ, φ), is given by equation (13). The above-mentioned transformed sound source signal z (τ) corresponds to the product of the sound source signal s (τ) in equation (13) and the transformation coefficient S _n ^m _ν ^μ (r _h (τ)). The transformed sound source signal z (τ) in the time domain is transformed into the transformed sound source spectrum Z _n ^m _ν ^μ (ω) in the frequency domain. As shown in equation (13), the transformed sound source spectrum Z _n ^m _ν ^μ (ω) is further multiplied by the spherical harmonic spectrum α _n ^m (ω) of the HRTF.

The spherical harmonic spectrum P _ν ^μ (ω) of the sound field is given with the origin O of the global coordinate system as the unfolding center. Therefore, the spherical harmonic spectrum calculation unit 122 can calculate the spherical harmonic spectrum P ν μ (ω) of the sound field without sequentially calculating the spherical harmonic spectrum α _n ^m (ω) of the HRTF based on the coordinates [r] of the sound source position and the coordinates [ _{r h} (τ)] of the listening point given in the global coordinate system using ^equation (13). Then, the binaural signal generation unit 124 can generate a binaural signal as an output signal by performing an inverse Fourier transform on the spectrum of the binaural signal given by equation (12).

Next, an example of acoustic signal processing according to this embodiment will be described. Figure 3 is a flowchart illustrating acoustic signal processing according to this embodiment.
(Step S102) The input unit 110 receives the coordinates of the listening point and the sound source signal.
(Step S104) The spherical harmonic spectrum calculation unit 122 calculates, for each frequency, the conversion coefficients from the basis functions of the spherical harmonic expansion at the sound source position with the center of the head as the reference point as the listening point to the basis functions at the sound source position with the origin as the expansion center.
(Step S106) The spherical harmonic spectrum calculation unit 122 converts the converted sound source signal obtained by multiplying the sound source signal by a conversion coefficient for each ear into a converted sound source spectrum in the frequency domain. The spherical harmonic spectrum calculation unit 122 calculates the spherical harmonic spectrum of the sound field by multiplying the converted sound source spectrum by the spherical harmonic spectrum of the HRTF according to equation (13).

(Step S108) The binaural signal generation unit 124 calculates the sound pressure spectrum for each ear by linearly combining the basis functions of the spherical harmonic expansion using the spherical harmonic spectrum of the sound field according to equation (12). As a linear combination, the weighted sum of the basis functions of the spherical harmonic expansion, with the spherical harmonic spectrum of the sound field as the weight coefficient, is obtained as the sound pressure spectrum.
(Step S110) The binaural signal generation unit 124 converts the frequency domain sound pressure spectrum for each ear into a time domain output signal and outputs the converted output signal to the output unit 140. After that, the process shown in Figure 3 is completed.

<Second Embodiment>
Next, a second embodiment will be described. The following description will mainly focus on the differences from the first embodiment, and for similarities, refer to the description in the first embodiment.
According to spherical harmonic expansion, the closer a location is to the expansion center, the more accurately the sound field spectrum can be estimated. Typically, in binaural playback, the listener's head position is used as the listening point. However, the entrances to the external auditory canals of each ear are used as measurement points for HRTF. The entrances to the external auditory canals are located about 7 to 10 cm away from the head position. This deviation of the measurement points from the head position can cause a decrease in the accuracy of the sound field spectrum. Therefore, in this embodiment, the expansion center of the spherical harmonic expansion is set to the position of each ear. This is expected to improve the accuracy of sound field spectrum estimation.

Figure 4 shows the relationship between the global coordinate system and the local coordinate system according to this embodiment. However, the left ear is used as an example.
The spherical harmonic spectrum calculation unit 122 receives listening point information indicating the head position via the input unit 110. The spherical harmonic spectrum calculation unit 122 can, for example, determine the position of the left ear as a position located a predetermined distance from the center of the head and to the left in a predetermined direction of the head.

Here, the vector indicating the position of the left ear E relative to the head position H is denoted as [r _e ] (= [r _e , θ _e , φ _e ]), the vector indicating the sound source position S relative to the left ear E is denoted as [r' _s (τ)] (= [r' _s (τ), θ' _s (τ), φ' _s (τ)]), and the vector indicating the position of the ear E relative to the origin O in the global coordinate system is denoted as [r' _h (τ)] (= [r' _h (τ), θ' _h (τ), φ' _h (τ)]). Using the addition theorem for spherical Bessel functions, in the spherical harmonic expansion of the HRTF expressed in equation (6), if the expansion center is shifted from the head position H to the left ear E, the HRTF _GL ([r' _s ] - [r' _h (τ)], ω) from the sound source to the left ear is given as shown in equation (14).

In equation (14), the spherical harmonic spectrum _{βνμ(ω) of the left ear, multiplied by a basis function that is the product of the second kind of spherical Hankel function jν(2) (kr's(τ)) and the spherical harmonic function Yνμ (} ^θ _'s ⁽ _{τ )} , ^φ _'s (τ)), is given by equation (15).
The coordinates [r _e ] of the left ear E may be predetermined by the distance and direction from the center of the head. In that case, the spherical harmonic expansion unit 126 may calculate a conversion coefficient S _n ^m _ν ^μ ([r _e ]) for the coordinates [r _e ] of the left ear E relative to the head position H using equation (8), and correct the calculated conversion coefficient S _n ^m _ν ^μ ([r _e ]) and the spherical harmonic spectrum α _n ^m (ω) of the left ear's HRTF to a spherical harmonic spectrum β _n ^m (ω). The spherical harmonic expansion unit 126 can also correct the spherical harmonic spectrum β _n ^m (ω) of the right ear's HRTF using the same method. The spherical harmonic expansion unit 126 stores the HRTF data showing the corrected spherical harmonic spectrum β _n ^m (ω) for each ear in the storage unit 130 in advance.

Substituting ^{the spherical harmonic spectrum βνμ} ₍ ω) from equation (15) into equation (14), the HRTF G _L ([r _'s ] - [ _r'h (τ)], ω) for the left ear is transformed as shown in equation (16). The transformed HRTF has the same form as the HRTF shown in equation (6), except that _{the corrected spherical harmonic spectrum βνμ} ^{( ω} ) is used instead of the spherical harmonic spectrum _αnm (ω) of the HRTF in equation (6). This correction of the spherical harmonic spectrum can also be considered as a shift of the reference point for acquiring the HRTF from the center of the head to the position of the left ear.

Next, using the addition formula for spherical harmonics, the expansion center in the spherical harmonic expansion of HRTF in equation (16) is shifted to the origin O in the global coordinate system. Substituting the coordinates of the sound source position [r] and the position of the left ear [r' _h (τ)], respectively, with respect to the origin O in the global coordinate system, into the three-dimensional coordinates [r ₁ ] and [b] in equation (7), and applying this to equation (16), the HRTF is transformed as shown in equation (17). Then, substituting the HRTF expressed in equation (17) into equation (5), the sound pressure spectrum P _L ([r], ω) of the left ear signal is obtained as shown in equation (18).

In equation (18), the spherical harmonic spectrum ^{P Lηξ} (ω) of the sound field obtained by multiplying by an η-order ξ basis function which is the product of the η-order sphere Bessel function j _η (kr) and the η-order ξ-order spherical harmonic function Y _η ξ (θ, φ), or by an η-order ξ basis function which is the product of the η-order second kind sphere Hankel function h _η ⁽²⁾ (kr) and the η-order ξ-order spherical harmonic function _{Y η ξ} ^{( θ} , φ), is expressed in equation (19).

Therefore, the spherical harmonic spectrum calculation unit 122 can calculate the spherical harmonic spectrum P Lη ξ (ω) of the sound field using the coordinates [r] of the sound source position S given in the global coordinate system, the coordinates [r' _h ^{(τ)] of the left ear E as the listening point, and the spherical harmonic spectrum β ν μ} _{( ω} ⁾ of the left ear E read from the storage unit 130. The binaural signal generation unit 124 can then generate the left ear signal by performing an inverse Fourier transform on the spectrum for the left ear given by equation (18).

The spherical harmonic spectrum calculation unit 122 can calculate the spherical harmonic spectrum P _Rη ^ξ (ω) for the right ear using the same method as for the left ear. The binaural signal generation unit 124 can also calculate the sound pressure spectrum for the right ear from the spherical harmonic spectrum P _Rη ^ξ (ω) and the sound source signal s(τ) using the same method as for the left ear, and generate a signal for the right ear by performing an inverse Fourier transform on the calculated sound pressure spectrum.

Next, an example of acoustic signal processing according to this embodiment will be described. Figure 5 is a flowchart illustrating acoustic signal processing according to this embodiment.
(Step S122) The input unit 110 receives the coordinates of the center of the head and the sound source signal as the listening point.
(Step S124) The spherical harmonic spectrum calculation unit 122 calculates, for each frequency, the conversion coefficients from the basis functions of the spherical harmonic expansion at the sound source position, with the position of each ear as the listening point as the reference, to the basis functions at the sound source position, with the origin as the expansion center as the reference, at each time point.
(Step S126) The spherical harmonic spectrum calculation unit 122 converts the converted sound source signal obtained by multiplying the sound source signal by a conversion coefficient for each ear into a converted sound source spectrum in the frequency domain. The spherical harmonic spectrum calculation unit 122 calculates the spherical harmonic spectrum of the sound field by multiplying the converted sound source spectrum obtained using equation (19) by the corrected spherical harmonic spectrum.

(Step S128) The binaural signal generation unit 124 calculates a sound pressure spectrum for each ear by linearly combining the basis functions of the spherical harmonic expansion using the spherical harmonic spectrum of the sound field according to equation (18).
(Step S130) The binaural signal generation unit 124 converts the frequency domain sound pressure spectrum for each ear into a time domain output signal and outputs the converted output signal to the output unit 140. After that, the process shown in Figure 5 is completed.

In the above explanation, it is assumed that the head position is used as the listening point for measuring HRTF, and that a pair of HRTF values for the left and right ears are associated with the position of the center of each ear. If, for each ear, the position and the HRTF measured for that ear are associated as the reference for measuring HRTF, the spherical harmonic expansion unit 126 may perform spherical harmonic expansion on the HRTF measured for each sound source position for each ear to calculate the spherical harmonic spectrum α _n ^m (ω) of the HRTF for that ear. In that case, the spherical harmonic spectrum α _n ^m (ω) corresponds to the corrected spherical harmonic spectrum β _ν ^μ (ω) for that ear. Therefore, in steps S124 and S126, the spherical harmonic spectrum calculation unit 122 may use the calculated spherical harmonic spectrum α _n ^m (ω) instead of the corrected spherical harmonic spectrum β _ν ^μ (ω). Furthermore, the binaural signal generation unit 124 may use the spherical harmonic spectrum α _n ^m (ω) calculated in step S128 instead of the corrected spherical harmonic spectrum β _ν ^μ (ω).

Generally, HRTF (Heat-Range Frequency) is determined by the relative positional relationship between the sound source and the listening point. While the above explanation uses the example of a listening point that moves and a sound source that remains stationary, it is not limited to this case. This embodiment can also be applied, for example, to a situation where the listening point is stationary and the sound source moves.

The above description uses the example of the spherical harmonic spectrum calculation unit 122 and the binaural signal generation unit 124 generating one binaural signal for one sound source, but it is not limited to this. The input unit 110 may receive input for each of multiple sound sources, associated with listening point information and sound source signals. The input unit 110 may also receive input associated with sound source position information indicating the sound source location of each sound source. Furthermore, the control unit 120 may have the sound source position set for each individual sound source. The spherical harmonic spectrum calculation unit 122 and the binaural signal generation unit 124 may generate binaural signals for each sound source based on the sound source signal and sound source position, and the generated binaural signals may be mixed between the sound sources to obtain an acoustic signal which is then output as an output signal via the output unit 140.

When the control unit 120 acquires a sound source signal and listening point information from an external device, it may read the sound source signal and listening point information previously stored in the storage unit 130 instead of using the input unit 110. Also, instead of outputting the output signal to an external device using the output unit 140, the control unit 120 may store it in the storage unit 130, or output it to a playback unit (speaker) installed in or connected to the sound signal processing device 10 and emit sound.
In the acoustic signal processing device 10, the spherical harmonic expansion unit 126 may be omitted. The storage unit 130 may have HRTF data acquired from an external device pre-stored in it.

As described above, the acoustic signal processing device 10 according to this embodiment comprises a spherical harmonic spectrum calculation unit 122 and a binaural signal generation unit 124. The spherical harmonic spectrum calculation unit 122 calculates conversion coefficients from the basis functions of the spherical harmonic expansion at the sound source position with respect to the listening point to the basis functions at the sound source position with respect to the origin, and calculates the spherical harmonic spectrum of the sound field for each ear based on the spherical harmonic spectrum of the head-related transfer function, the conversion coefficients, and the sound source signal. The binaural signal generation unit 124 calculates the sound pressure spectrum of the output signal for each ear by linearly combining the basis functions using the spherical harmonic spectrum of the sound field, and converts the sound pressure spectrum into a time-domain output signal.
With this configuration, for each ear, the spherical harmonic spectrum representing the sound field at the listening point is calculated without recalculating the spherical harmonic spectrum of the head-related transfer function in the frequency domain. By linearly combining the basis functions in the spherical harmonic expansion using the calculated spherical harmonic spectrum, it is possible to render sound in the frequency domain with the sound source or listening point moving arbitrarily. Through rendering, the frequency characteristics of the sound continuously change in accordance with the smooth fluctuations of the sound source position or listening point. Furthermore, since it does not involve the addition of the head-related transfer function or the sound source signal convolved with the head-related transfer function in the time domain, the sound pressure spectrum of the sound field at the listening point is estimated without disturbing the frequency characteristics that provide clues for sound image localization. Therefore, sound image localization to that sound source position is not hindered.

Furthermore, the listening point and the reference point for acquiring the head-related transfer function may both be head positions.
By using the head position as the listening point, the rendering operations related to the listening point can be simplified compared to when the position of each ear is used. Furthermore, by using the head position as the reference point for acquiring head-related transfer functions, it becomes easier to acquire and manage head-related transfer functions for each ear simultaneously.

Furthermore, the listening point and the reference point for acquiring the head-related transfer function may be the position of each ear, respectively.
By using the position of each ear as the listening point, that position is used as the expansion center for the spherical harmonic expansion of the head-related transfer function. Therefore, the estimation accuracy of the calculated sound pressure spectrum can be improved compared to when the center of the head is used as the listening point.

The audio signal processing device 10 may be implemented as a dedicated audio signal processing device, or it may be implemented as a device whose primary function is not audio signal processing, such as a personal computer or tablet device. The audio signal processing device 10 may also be implemented as part of equipment related to the production, editing, and distribution (including broadcasting) of various types of content (for example, a mixing console).

Furthermore, some or all of the above-described acoustic signal processing device 10 may be configured using dedicated components (such as integrated circuits) or implemented using a computer. For example, either the spherical harmonic spectrum calculation unit 122 or the binaural signal generation unit 124, or a combination thereof, may be implemented by a general-purpose arithmetic processing unit such as a CPU (Central Processing Unit) executing processes instructed by commands written in a predetermined program read from a storage medium such as ROM (Read Only Memory).

Although one embodiment of this invention has been described in detail above with reference to the drawings, the specific configuration is not limited to that described above, and various design modifications can be made without departing from the spirit of this invention.

10…Acoustic signal processing unit, 110…Input unit, 120…Control unit, 122…Spherical harmonic spectrum calculation unit, 124…Binaural signal generation unit, 126…Spherical harmonic expansion unit, 130…Storage unit, 140…Output unit

Claims (4)

The conversion coefficients from the basis functions of the spherical harmonic expansion at the sound source position relative to the listening point to the basis functions at the sound source position relative to the origin are calculated.
For each ear, a spherical harmonic spectrum calculation unit calculates the spherical harmonic spectrum of the sound field based on the spherical harmonic spectrum of the head-related transfer function, the conversion coefficient, and the sound source signal.
For each ear, the sound pressure spectrum is calculated by linearly combining the basis functions using the spherical harmonic spectrum of the sound field.
An acoustic signal processing device comprising: a binaural signal generation unit that converts the sound pressure spectrum into an acoustic signal in the time domain. The receiving point is the head position,
The acoustic signal processing apparatus according to claim 1, wherein the reference point for obtaining the head-related transfer function is the head position. The aforementioned listening points are the positions of each ear.
The acoustic signal processing device according to claim 1, wherein the reference points for obtaining the head-related transfer function are the positions of each ear. A program for causing a computer to function as an acoustic signal processing device according to any one of claims 1 to 3.