AU2017355584B2

AU2017355584B2 - Intelligent hearing aid

Info

Publication number: AU2017355584B2
Application number: AU2017355584A
Authority: AU
Inventors: Yujie Li; Jianbang Liu; Zhihui WEN
Original assignee: Chears Tech Co Ltd
Current assignee: Chears Technology Co Ltd
Priority date: 2016-11-02
Filing date: 2017-10-25
Publication date: 2020-02-20
Anticipated expiration: 2037-10-25
Also published as: AU2017355584A1; CN110268726A; WO2018083570A1

Abstract

An intelligent hearing aid, comprising a microphone, a smartphone and an earphone operatively connected with each other, wherein the microphone is configured for receiving, recording, and transferring environmental audio data / signals to the smartphone; the smartphone is configured for carrying out noise suppression and data amplification on the audio data receiving from the microphone to generate and transfer an enhanced audio data to the earphone; and the earphone is configured for receiving and outputting the enhanced audio data; wherein the noise suppression combines knowledge of pattern recognition and signal processing and probability theory to facilitate estimation of noise power spectral density.

Description

INTELLIGENT HEARING AID

Technical Field [001] The present disclosure relates to intelligent hearing aids, and more particularly, to an intelligent hearing aid with enhanced audio processing performance comprising a microphone, a smartphone and an earphone operatively connected with each other.

Background Art [002] There are needs of intelligent hearing aids with enhanced audio processing performance for hearing-impaired elderly and such enhanced intelligent hearing aids usually require dedicated high end or expensive components, such as a faster CPU and relatively high quality audio unit. It is desirable to make use of existing and commonly available equipment of hearing-impaired elderly to provide an enhanced hearing aid.

Brief Summary of Invention [003] There is disclosed an intelligent hearing aid, comprising a microphone, a smartphone and an earphone operatively connected with each other, wherein the microphone is configured for receiving, recording, and transferring environmental audio data / signals to the smartphone; the smartphone is configured for carrying out noise suppression and data amplification on the audio data receiving from the microphone to generate and transfer an enhanced audio data to the earphone; and the earphone is configured for receiving and outputting the enhanced audio data; wherein the noise suppression combines knowledge of pattern recognition and signal processing and probability theory to facilitate estimation of noise power spectral density, and the estimation is conducted under assumptions including:

(1) speech and noise signals are both addictive in short-time Fourier domain; (2) the speech and noise signals are assumed to be zero-mean and independent of each other and presented as

F(|T|^{1 2}) = F(|S|²) + E(\N\²), where E(·) being statistical expectation operator; (3) the speech and noise signals are assumed to have spectral coefficients having a complex Gaussian distribution, to deduce Minimum mean-square error estimator, lⁿl^z

PN(n)=^-e ^ση, σ^π _K z x zr2

WO 2018/083570

PCT/IB2017/056613 (4) the noise signal is assumed to be more stationary than the speech signal.

Brief Description of Drawings [004] The disclosure will be described by way of example with reference to the accompanying drawings, in which:

[005] Figure 1 is a flow chart of a noise suppression algorithm of an example intelligent hearing aid of the present disclosure.

Detailed Description of Invention [006] In order to address the needs of hearing-impaired elderly, the applicant has designed a hearing aid app for them to listen just by connecting a smartphone with earphone. Different noise filters are devised according to different environment at which users are located. The present disclosure adopts warm tone color in the user-interface for easy-reading to the users.

[007] After downloading and installing the app, the user needs to select different scene mode such as Indoor mode, Outdoor mode, Spacious area mode, and the like, according to audio environment. The environmental sound will be received via the microphone and processed by the App, and then amplified clean audio would play out from the earphone. The app support background running then the application and mobile screen are not necessary to keep on.

[008] The hearing aid of the present disclosure is combining smartphone and headset/earphone as an intelligent hearing aid. First of all, the microphone on earphone keep recording the audio sound in the environment to smartphone operation system. Within the smartphone operation system, the input audio data would be amplified and noise suppression processed. Thirdly, the amplified and clean audio would be transmitted to earphone output to play out for hearing impaired people.

[009] The present disclosure adopts circular buffers and multi-threading mechanism to improve the audio quality without negative effect. Circular buffer had separated the recording thread and processing thread to preserve the continuation of audio output. For implementation, multi-threading was using to divide a first buffer in processing thread and a second buffer in recording thread.

[0010] According to the present disclosure, the recording process contains two major steps. One is called analog to digital conversion, the other one is called data streaming and storing. The buffer is a memory structure used in the data streaming process to temporarily hold the

WO 2018/083570

PCT/IB2017/056613 collected digital voice data. Once the buffer is filled with data, the data will be encapsulated as a package and passed to next process. The size of the buffer cannot be chosen randomly. The buffer size should be determined by the sampling frequency of the analog to digital convertor.

[0011] One critical issue kept bothering the implement of digital hearing aids is the latency. When people are talking, they will most likely expect two kinds of signal, the movement of the speaker’s mouth and the sound they are producing. Since the light will travel much faster than the sound, people will observe the movement before they hear the sound. However, human brain cannot notice the time gap between two signals if the gap is smaller than around 30 ms. In the case of digital hearing aid application, the buffer will not transfer any data out before it is full. This character will introduce much delay. The delay will be long enough for the brain to identify it, which means the user will experience the mismatch between the observed movement of the speaker’s mouth and the sound they heard. This defect will bring the user tremendous difficulty in receiving and understanding the content of the voice.

[0012] Therefore, the present disclosure deploys a function in the AudioRecord class (getMinBufferSize) to identify the least size of the buffer that won’t introduce any side effect. Further information about this function can be found at https://developer.android.eom/reference/android/media/AudioRecord.html#getMinBuf ferSize%28int,%20int,%20int%29 .

[0013] To improve the audio processing performance of hearing aid app, the real time noise suppression algorithm need to be implemented. For this purpose, the present disclosure have modified the existing prior art noise suppression algorithm from audio-file-based to real time processing. On the other hand, for better implementation of real time noise suppression processing, the computation complexity of original algorithm had also been simplified.

[0014] The prior art statistical-model-based algorithm cannot perform such function. The prior art statistical-model-based algorithm allows the users estimate the noise power spectrum density by fitting the input data into certain statistical model. The estimation result cannot ensure certain level of confidence if the input data failed to exceed certain amount data. The estimation result will be untrustworthy if no enough data was given. The algorithm has to wait for relatively long time to collect enough voice data. However, the characteristic of real-time process violate this condition. The latency of process won’t be tolerated if the algorithm needs to wait for the voice and collect large amount of voice input data. Therefore, there is a tradeoff between the accuracy of the estimation and the processing latency.

WO 2018/083570

PCT/IB2017/056613 [0015] The modification of the present disclosure allows the relevant algorithm to perform these functions. To cope with the demand of real-time processing, historical data are involved in the estimation process. A short period of present audio data will be attached to a long period of historical one, which are generated last iteration and stored in a register with certain volume. During each iteration, only the data in the slide window will be involved and the window will slide a few data point to next position after the estimation is done in this iteration. This data arrangement mechanism is inspired by some signal compression technique in the digital communication area. Not only the duration of estimation has been curtailed, but also the level of confidence of the estimation has been raised.

[0016] The Comparison between the existing algorithm and modified algorithm:

	Modified Algorithm	Existing Algorithm
The Size of Input Data Frame Filtered in Single Processing Cycle	256	2048
Corresponding Transmission Latency (48000HZ)	5.33ms « 30ms	42.67ms > 30ms
The Size of Historical Data Register	3840	NA
The Pre-defined A Priori SNR When Speech Present	Depend on the scenario. (e.x.15 dB for quiet indoor environment; 5dB for noisy outdoor environment;)	15dB
Input Samples Requirement	Audio Buffer (256 Samples)	Audio File (10240 Samples +)
Real-Time Processing Supported	Yes	No
Noise Suppression Effectiveness (Stable Background Noise)	-80%	-80%
Noise Suppression Effectiveness (Non-Stable Background Noise)	-50%	-50%
Computation Complexity	Low	Low

[0017] Referring to Figure 1, which is a flow chart of a noise suppression algorithm of an example intelligent hearing aid of the present disclosure. The present disclosure adopts an enhanced noise cancellation algorithm, which aims at eliminate the corruption of signal, and which was mainly introduced to the single channel audio signal by non-stationary noise. It combines the knowledge of pattern recognition and signal processing to facilitate the estimation of the noise power spectral density. The probability theory was also involved in the developing of this algorithm to build a theoretical noise pattern for the speech enhancement.

WO 2018/083570

PCT/IB2017/056613 [0018] The estimation is conducted under several assumptions: (1) The speech and the noise signals are both addictive in the short-time Fourier domain. (2) The signals (speech and noise) are assumed to be zero-mean and independent of each other, which can be presented as

WI²) = £(|S|²) +wi²), with £() being the statistical expectation operator. (3) The signals are assumed to have a complex Gaussian distribution in the spectral coefficients to deduce the Minimum meansquare error (MMSE) estimator, lⁿl^z

PN(n)=^-e ^ση, σ^π σ/π (4) It is reasonable to assume the noise signal is more stationary than the speech signal.

[0019] The design case is that a microphone will record the audio signals at certain frequency and the audio signal sequence will be stored in a buffer, where can contain 256 signals. Once the buffer is full, the 256 signals will be forward to processor. There are THREE main procedures in the iterative noise cancellation. The first key step is aligning the new coming data with historical data reserved by a register. The noise estimation and elimination are done in the second process. The last step is called clean signal finalization. The clean audio signal can be achievable after the final state.

[0020] Each audio signal is an 8-bit (1 byte) double digit. The data frame fed into the iterative noise elimination process are designed to be 2048 byte. Therefore, we assign a register with a 1792-byte storage size to keep the historical data. To ensure the reliability of the estimation, this algorithm combines the current data with historical data. Once there is a new 256-byte data stream coming in, this new data stream will be aligned at the back of the data stored in the register, then multiply with a 2048-point coefficient sequence of Hanning window. That’s what must be done in the first data alignment process. As one cycle of noise cancellation requires only 256 new recorded data, real-time processing can be achieved with an inappreciable delay.

[0021] Before the highlight of this algorithm, the iterative parameters are all set to zero for convenience except the a priori probability is set to 0.5, which means the algorithm assumes the mean of a posteriori probability of the speech presence is also 50%. The data stream will be transform to the frequency domain by Fourier transform and only half of the frequency coefficients on the positive axis will be remained as the signal power spectrum to decrease

WO 2018/083570

PCT/IB2017/056613 the computational complexity, since those coefficients are symmetric. The posteriori SNR estimation is based on the new signal power spectrum and the noise power spectrum computed in last iteration. Then, the a posteriori speech presence probability (SPP) presents by (

nfqr Ί 1^1___\ where is the a priori SNR for speech presence in our model, and P(Xi) = o).

[0022] The present disclosure employs the spectral noise power estimation of the previous frame, = σβ(1- 1). The noise power estimation and the estimation of a posteriori SPP are to some extent in inverse proportion. Therefore, the underestimation of σ/ will slow down the converging to the true value of it. To avoid stagnation, the technical solution of the present disclosure recursively smooth PC/fJy) over time, as

P(Z) = 0.9P(Z -1) + O.lPCTfJyG)), and enforce the current a posteriori SPP PfXilyG)) to be lower than 0.99 if the PG) larger than 0.99, as

PCHiMO) - l^min (“•W») > ⁰⁹⁹

I P(7Cily(0) ,else.

[0023] The MMSE estimator under speech presence can be computed as follows.

E(|/V|² |y) — P(3P₀ly) |y|² + P^lyj^G - 1).

Then, the estimated spectral noise power is obtained by this recursive smoothing of F(|/V|²|y(Z)).

<TvG) = -1) + (1- «pow)P’(l/V|² |y(Z)), where a_pow = 0.8 is the smoothing factor. According to this estimated spectral nose power density, the estimated a priori SNR can be presented as ξ ly(0l²

I°n (01² |y(Z-1)|² _{r Ί} ^a i^rni2 + (1 - «)G - 1)] , for I = 1 , else.

[0024] Therefore, the estimated spectral coefficients of clean audio signal are given by

WO 2018/083570

PCT/IB2017/056613

K0 =

KO

KO +1 y(0[0025] The clean signal in real time can be easily collected after applying inverse the Fourier transform to the complete spectral domain coefficients of the data stream. (The negative half of the domain was excluded after the data was transform to frequency domain.) [0026] In the finalization of clean signal, the data must multiply a synthesis to avoid distortion due to the data segmentation, the new clean data will be appended to the earlier attained clean data, which means this new 2048-point signal will be add to the earlier reserved clean signals. The first 256 data will be converted to audio signal and removed from the signal stream. The remaining 1792-byte data will be saved in certain register waiting for the next iteration.

[0027] Through all the computation described above, a relative noise-free audio signal will be generated and heard. There are evidences that this algorithm can approach the optimal solution in a fast speed.

[0028] Features set out in the claims hereto (jointly and severally where appropriate) are to form part of this disclosure and are incorporated herein by reference.

[0029] While various examples or embodiments have been described herein, it should be appreciated that they are for illustration and are not for scope restriction. It should be appreciated that portions or parts of the various example embodiments can be excerpted for combination and/or mix-and-match where appropriate to form other variants without loss of generality.

Claims

Claims

1. An intelligent hearing aid, comprising a microphone, a smartphone and an earphone operatively connected with each other, wherein the microphone is configured for receiving, recording, and transferring environmental audio data / signals to the smartphone;

the smartphone is configured for carrying out noise suppression and data amplification on the audio data receiving from the microphone to generate and transfer an enhanced audio data to the earphone; and the earphone is configured for receiving and outputting the enhanced audio data;

wherein the noise suppression combines knowledge of pattern recognition and signal processing and probability theory to facilitate estimation of noise power spectral density, and the estimation is conducted under assumptions including: (1) speech and noise signals are both addictive in short-time Fourier domain; (2) the speech and noise signals are assumed to be zero-mean and independent of each other and presented as

F(|Y|²) = F(|S|²) + F(|/V|²), where E(·) being statistical expectation operator; (3) the speech and noise signals are assumed to have spectral coefficients having a complex Gaussian distribution, to deduce Minimum mean-square error estimator,

1 lⁿl^z σ^π

Ps(s) = -j-e ^σ% σ/π (4) the noise signal is assumed to be more stationary than the speech signal.
2. An intelligent hearing aid of Claim 1, wherein the audio data / signals are recorded at predetermined frequencies and an audio data / signal sequence is formed and stored in a buffer, and the sequence comprises preferably 256 data / signals, wherein each data / signals is 1 byte in length; and/or wherein the noise suppression comprises an iterative noise cancellation process requiring only 256 byte of newly recorded data per noise cancellation cycle for achieving a real-time processing with an inappreciable delay.

WO 2018/083570

PCT/IB2017/056613
3. An intelligent hearing aid of Claim 2, wherein data frame fed into the iterative noise cancellation process are designed to be 2048 byte, wherein 1792 bytes are historical data, which combine with the 256 byte of newly recorded data to ensure reliability of estimation; wherein the newly recorded data are appended to the historical data, then multiply with a 2048-point coefficient sequence of Hanning window; and the last 1792 bytes of the data frame are used as the historical data of next noise cancellation cycle.
4. An intelligent hearing aid of Claim 3, wherein the data frame is transform to frequency domain by Fourier transform and only half of frequency coefficients on positive axis are remained as signal power spectrum to decrease computational complexity.
5. An intelligent hearing aid of Claim 4, wherein the noise suppression comprises a posteriori signal to noise ratio (SNR) estimation based on new signal power spectrum and noise power spectrum computed in last noise cancellation cycle.
6. An intelligent hearing aid of Claim 5, wherein a posteriori speech presence probability (SPP) is presented by (

ΡΓ7-Γ Ί _____\ ¹⁺ϊ^(^1+ξ^^{6 σ} ^1+6,1 i ’ where ξ_Ήι is the a priori SNR for speech presence in our model, and =P(Mi\y, fyq = 0).
7. An intelligent hearing aid of Claim 6, wherein PC/fyly) is recursively smoothed over time in the noise cancellation cycles to avoid stagnation, as

P(Z) = 0.9P(Z -1) + O.lPCJfilyCO), and the current a posteriori SPP PC/GlyCO) is enforced to be lower than 0.99 if the

P(Z) larger than 0.99, as

P(7fily(0) min (0.99,^(7+^(/)))

P(7fily(0) ,P(/) > 0.99 , else.
8. An intelligent hearing aid of Claim 7, wherein the MMSE estimator under speech presence can be computed as,

E(|/V|²|y)-P(T+₀|y) |y|² + P^y^l - 1).
9. An intelligent hearing aid of Claim 8, wherein the estimated spectral noise power is

WO 2018/083570

PCT/IB2017/056613 obtained by recursive smoothing of i’d/V|²|y(0), <Tv(0 = -1) + (1- «_pOwMlN|²|y(0)>

where a_pow = 0.8 is the smoothing factor, and the estimated a priori SNR is presented as ξ ly(0l²l^v(0l²|y(l — 1)|^{2 a} lasmi² , for I = 1 , else.
10. An intelligent hearing aid of Claim 9, wherein the estimated spectral coefficients of processed / clean audio signal are given by $(0 =

K0

K0 + 1 y(0io