yongxuustc / sednn

deep learning based speech enhancement using keras or pytorch, make it easy to use

deep_learning_for_speech_enhancement_keras_python

deep learning based speech enhancement using keras python

Authors: YONG XU & QIUQIANG KONG

Goal:

Make the GPU-C++ code project convert to python code which is much easier for the community to follow and use. The training and decoding code will be unified into the python code. Keras will be used as the toolkit.

Invitation:

I want to invite you to be one of the contributors of this project, please contact me if you have interest. [email protected]

My final goal is to build a universal & robust deep learning based speech enhancement front end. And aslo try to adapt it to really serve for the speech recognition back-end.

Ref:

The original GPU-C++ code: https://github.com/yongxuUSTC/DNN-for-speech-enhancement

Please cite the following papers if you use this code:

[1] A Regression Approach to Speech Enhancement Based on Deep Neural Networks. Yong Xu, Jun Du,Li-Rong Dai and Chin-Hui Lee, IEEE/ACM Transactions on Audio,Speech, and Language Processing,P.7-19,Vol.23,No.1, 2015 (2018 IEEE SPS Best paper award, citations > 600)

[2] An Experimental Study on Speech Enhancement Based on Deep Neural Networks. Yong Xu, Jun Du, Li-Rong Dai and Chin-Hui Lee,IEEE signal processing letters, p. 65-68,vol.21,no. 1,January 2014 (citations > 550)

[3] Multi-Objective Learning and Mask-Based Post-Processing for Deep Neural Network Based Speech Enhancement, Yong Xu, Jun Du, Zhen Huang, Li-Rong Dai, Chin-Hui Lee, Interspeech2015

Some DNN based speech enhancemen demos:

http://staff.ustc.edu.cn/~jundu/The%20team/yongxu/demo/SE_DNN_taslp.html

http://staff.ustc.edu.cn/~jundu/The%20team/yongxu/demo/IS15.html

sednn's Issues

如何使用多种信噪比数据来训练一个model？

你好，很感谢您的代码，然我对se有了更深的理解。
现在我有一些问题需要您的帮助。
1.您的代码中prepare_data.py中的pack_features中
这一部分我有点疑惑“Pad start and finish of the spectrogram with boarder values.”
这里为什么要使用边界值来填充谱图呢？

2.您的代码中是针对单独的SNR来进行增强过程的，我想问的是，我能不能通过多种信噪比的混合语音训练一个泛化性能较好的模型？具体实现方法您可不可以给一些建议？

3.您可不可以帮忙解释一个问题：
进行短时傅里叶变换的方法之间的区别 一个是使用librosa库，一个是您代码中的scipy库？

期待您的回复！

Unique model for every SNR

Hi,
In the current python implementation, a separate model is generated for each SNR value.

Are the results reported in the paper for different SNR test data, computed against a single model or model pertaining to particular SNR?

Is it -

1. model(SNR10db) * Test(10db) = clean speech
   or
2. model(all SNRs) * Test(any SNR) = clean speech

Problem in the size of training set

Hello Yong, qiuqiang kong,

I am really impressed with your work. I'm trying to use the python code to train my own model with TIMIT data set. However, I found it's difficult to use all of 4620 audios as the training set, due to the limitation of CPU memory. Could you teach me how many audios did you use to train your best model? How many noise did you use? (I'm using n1~n100, total 100 noise audios, and which value of magnification should I use?) Since if I use 4620 audios as clean speech, and use 100 noise audios to apply on every clean speech, it will generate a super large training set.. ,which cannot be loaded in CPU memory...

Thank you very much!

BRs,
Lupeng Liu

增强语音与原语音长度不一致

你好！
在使用你的代码增强语音后，我试图计算语音的STOI，发现enh_wav的长度与原来的语音长度不一致；我想问一下这是什么原因导致的 以及是否会影响增强语音的性能

你好，请问噪声感知训练部分怎么没有了

增强后语音频谱图问题

你好，我用了thchs30中的一万条噪声数据训练出了一个模型。但是我看了测试集的增强结果，发现语谱图中某些固定的频率上的能量一直不变化，反应在图上就会有细条纹出现，而且语音听起来存在失真现象。下面三幅图分别是带噪声语音语谱图（主要是一些低频噪声），原始无噪声语音语谱图和增强后的语谱图。
带噪

无噪

增强

和原始的语谱图比起来会多一些奇怪的频率能量。
但是我用你们的训练好的matlab工具进行降噪以后就没有这样的现象。下图是我用你们团队提供的matlab工具进行增强的语谱图，降噪效果很好，并且失真现象也减少了很多。

请问是我训练数据太少的缘故吗，我该怎么去提升增强后的语音质量能。
谢谢

mixture2clean_dnn

您好，很感谢您的分享。有几个问题不是很懂想请教下。

1. 训练的数据标签没搞懂代表什么含义。
2. 在evaluate.py中，没有见到将计算的结果写到“_pesq_results.txt”的程序。
   谢谢!

ValueError

你好，我用的自己的语料，运行时出现一下问题
mixed_audio = s + n
ValueError: operands could not be broadcast together with shapes (34719,) (33910,)

关于打他、

the problem of main_dnn.py

the program main_dnn.py, eval_te_gen and eval_tr_gen is same, both of their type is 'test', i think eval_tr_gen's type should changed to 'train'

tensorflow函数过期的问题

请问一下这个问题怎么解决呢，最后运行中，发现这个是关于tensorflow使用函数过期的问题，但是代码里面确找不到这个函数，所以进程卡在了use tf.cast instead这里，想求教一下这里是怎么跑通的

Difficulty with training model for multiple SNR

Hi Yong Xu,

Thank you for your prompt replies on queries. I have been able to train model for a single SNR. Now I am trying to train a model on 5 SNRs (0db, 10db, 15db, 20db, 30 db). I have generated the combined data.h5 file but is has huge size of 27GB. Now, when I am trying to read this file to generate scalar.p my system is hanging. Machine config: GTX 1060, 32 GB RAM, 1TB Hard drive. Can you please help me out with the same?

Thanks
Akshaya

关于数据预处理

@yongxuUSTC @qiuqiangkong
您好，我在看mixture2clean_dnn代码的时候，有一些困惑。
1.pach_features当中的参数n_hop的作用是？为什么要跳过几帧呢？
2.这样跳过之后，平行语料在训练时候的对应关系是什么样的呢？
期待您的回复，谢谢！

复现代码结果不理想

您好，在使用了您的代码后，我有一些地方不确定，期待您能在百忙之中抽空解答一下：

1. 测试代码时，使用的语音和噪声数据与您所给一致且采样频率为16KHz，所有参数未做修改，train_loss与您所给一致，但test_loss却偏大，如图所示
   

PESQ结果如下，仅为1.83

在查看代码后冒昧地问一下是不是您的代码只是DNN的baseline，并没有NAT ，dropout和post-processed的部分，所以复现代码时的结果不良好，如有还望您指出在哪一部分。

2. 在prepare_data.py中，read_audio的部分有resample的操作，想问一下如果我的原始语音和噪声都不是16KHz的，resample之后再训练和测试 和 直接用16KHz的数据训练和测试 的实验结果会有差别么？

3. 在prepare_data.py中，您的操作是先生成了mixture_csvs，然后按照mixture_csvs中的noise_onset 和noise_offset 来对噪声进行截取或补齐。那如果如问题2中那样，原始数据的采样频率不是16KHz， 而是由8KHz、 16KHz、20KHz等不同采样频率的语音和噪声来组成，那在calculate_mixture_features的resample阶段，它们的长度就会变化，但这里还是用mixture_csvs中的noise_onset 和noise_offset 来确定数据长度，就有可能导致noise_offset 比resample后的数据长度要大，使得additive_mixing时，s与n的shape不同而无法相加。比如，原来噪声长度是80000，resample后 长度变为64000，但noise_offset介于[64000 80000]，这就使得64000之后的数据取不到，语音和噪声的shape不同而无法相加。不知您是否遇到过这种问题，该怎样处理呢？

本人初学者，如有理解错误之处，望您指出。
谢谢！

关于训练后生成model

您好，我看到您这个项目的基于TIMIT数据集训练出的DNN model 的 enhancement-demo-tool，想请问一下我用自己的数据训练的model想转换成这样的可执行文件，要怎么操作呢。

PESQ

Hey yongxuUSTC! Really cool project I really liked it! I am currently trying to evaluate my own speech enhancement model. I have a set of clean spectograms and a set of spectograms outputted from the model. I've been trying to udnerstand you PESQ code:

Args:
workspace: str, path of workspace.
speech_dir: str, path of clean speech.
te_snr: float, testing SNR.

I would be really grateful if you could clarify what exactly the path to the workspace is, and also what the testing SNR is.

Thank You in advance!

About global variance

Hi Yong Xu,

I'm really appreciate for your project. I have some questions.
First, does this project include global variance equalization? If so, which part?

and I'm trained this model with my data. (I think about 20 hours data for train) However, PESQ results with enhancemented data were worse than comparing noisy and clean.

I founded that enhancemented waves are strange..

This is mixed audio below.

and this is enhancemented audio below.

Why did this result come out?

Thank you!

it's not easy to reproduce the project

Dear,

When I start the project ,
I find some difficult problem,
first, how to get the data,
then, how to change the config.py
and four files are independent?
finally,
any one has got success ?
who can help me ?

如何使用代码

我看了您的代码，可是看的自己一头雾水，这四个文件夹中的代码有什么区别呢，而且在我改了第三个文件夹中的runme.sh文件中的路径之后，其他参数均没有改变的情况下，还会报错。
请问一下，这里程序的运行顺序应该是怎么样的呢？

problem with overfitting

Hello Yong, qiuqiangkong
I am very much impressed with your work. I am trying to use the code , knowing that i split training set on many phases (each time pass 100 speech mixing them by 3 noises chosen randomly of 100 noises and for the Test i try with different size 3..100 ) of by saving model and weight and reload it each time for retraining (cause i had issue with machine performance (cpu and limited memory ) , and i 'm facing this problem the testing loss increases while training accuracy is still improving . also i get wav file audible but distorted.
i wish if you could upload the trained model (weight) Please . thanks very much .

Regards,
noor

模型輸入特徵問題請教

您好:
非常感謝你們所完成的程式，希望能抽空回答一個小小的問題。模型輸入特徵取完頻譜大小之後有做絕對值，但好像沒有取平方，但在A Regression Approach to Speech Enhancement Based on Deep Neural Networks這篇paper中特徵為log-power spectral，這是有什麼原因嗎?感謝你們~~~

gpu没有加速

按照requestment.txt 安装了库，使用cuda8.0 但是gpu没有加速，请问怎么回事？

Issue with Training and testing

Hello Yong,
I am very much impressed with work. I am trying to use the code on my data. My setup is as follows:

1. Right now I am using Python 3.6 on windows 10.
2. I have implemented runme.sh in a python code.
3. For training I am using 1350 sentences and 8 types of noises of duration same as combined duration of sentences.
4. For testing I am using 150 sentences with some noise same as training noise and some others.
   Now, the issue is that the training is less but not close to 0 and testing loss is way more than expected and enhanced wav files have no sign of speech in any noise condition.
   What should be done in this case? I am also attaching the plot of training stat.
   

Regards,
Ajay

An error in function additive _mixing()

/home/panxin/anaconda2/lib/python2.7/site-packages/numpy/core/fromnumeric.py:2920: RuntimeWarning: Mean of empty slice.
out=out, **kwargs)
/home/panxin/anaconda2/lib/python2.7/site-packages/numpy/core/_methods.py:85: RuntimeWarning: invalid value encountered in double_scalars
ret = ret.dtype.type(ret / rcount)
Traceback (most recent call last):
File "prepare_data.py", line 477, in
calculate_mixture_features(args)
File "prepare_data.py", line 179, in calculate_mixture_features
(mixed_audio, speech_audio, noise_audio, alpha) = additive_mixing(speech_audio, noise_audio)
File "prepare_data.py", line 247, in additive_mixing
alpha = 1. / np.max(np.abs(mixed_audio))
File "/home/panxin/anaconda2/lib/python2.7/site-packages/numpy/core/fromnumeric.py", line 2334, in amax
initial=initial)
File "/home/panxin/anaconda2/lib/python2.7/site-packages/numpy/core/fromnumeric.py", line 83, in _wrapreduction
return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
ValueError: zero-size array to reduction operation maximum which has no identity

I think this error happened because of the return value of np.max(np.abs(mixed_audio)) is too small, then computers ignore it as a zero(0), so the error occurs, so does my thought is right? I am looking forword to your receive. Thanks a lot

Global name 'librosa' is not defined

Sednn/mixture2clean_dnn/preparedata.py

def read_audio(path, target_fs=None):
    (audio, fs) = soundfile.read(path)
    if audio.ndim > 1:
        audio = np.mean(audio, axis=1)
    if target_fs is not None and fs != target_fs:
        audio = librosa.resample(audio, orig_sr=fs, target_sr=target_fs)
        fs = target_fs
    return audio, fs

• librosa is not define, Should import pack librosa.
• Moreover librosa will also cause TypeError: expected string or buffer
pip install 'joblib==0.11' --force-reinstall will solve it.

how many audio samples needed in training?

Hi, yongxu. Your speech enchancement project is wonderful. I have test the code and have a question about data: how many audio samples needed in traning for a "working" model?
Thanks a lot.

测试结果不理想

你好我用这个网络在0db情况下进行了训练，训练集能到到0.66的loss，但是测试集一直是0.90，我用的是timi库的训练集和测试集，请问是哪里出了问题吗

关于mixture2clean的keras和pytorch不同实现

您好，我在运行代码的时候遇到这样一个问题。将keras版本的mixture2clean_dnn中的DNN模型直接替换为pytorch版本的DNN，其他配置一样，然后pytorch-DNN的效果要差很多。
请问这大概是什么原因呢？
期待您的回复，感谢！

memory error报错

您好，程序在小数据量下运行可行。但当增大数据量后，出现了内存报错。
(注：实验使用了4000条纯净语音和10种噪声0db混合作为训练集，即40000条数据)

①在pack_features过程临时存储超内存报错，我改用分批量追加到.h5文件，避免了临时存储结构的内存报错。
但在train阶段，要从data.h5文件中读取所有的 x 放到一个np.array中，然后随机生成batch。由于内存问题，读取到array的过程又会报错。我理解的是，内存报错都是在临时存储数据时，能不能跳过这一步，直接从data.h5文件中随机生成batch？
或者您是如何处理大量数据的读取存储问题的？

②在prepare_data.py的compute_scaler函数中，data.h5文件中读取出来的数据存入np.array时报内存错误，超出最大内存
with h5py.File(hdf5_path, 'r') as hf:
x = hf.get('x')
x = np.array(x)

但下一步计算标准化参数时需要一次性输入x中的数据
x2d = x.reshape((n_segs * n_concat, n_freq))
scaler = preprocessing.StandardScaler(with_mean=True, with_std=True).fit(x2d)

如果希望处理大量数据，该如何得到scaler呢？

③inference函数里
if scale:
mixed_x = pp_data.scale_on_2d(mixed_x, scaler)
speech_x = pp_data.scale_on_2d(speech_x, scaler)
其中scaler是通过训练集的带噪语音计算出来的，
为什么【测试集的标准化】要用【从训练集中计算得到的scaler】来计算呢？

期待您的回复与指导。谢谢！

训练数据的问题

你好由于目前我实验室所用的服务器只有32G的内存没办法使用你们代码使用的那么大的训练集导致测试集的loss一直不太理想，你们能够把你们训练好的模型权重文件发我一份吗

problem in understanding the code

Where is the RBM part in this code

about stoi

how to use?

This seems like an interesting project. I was wondering if you can just give me a quick rundown on how to use this framework to denoise 44.1kHz wav mixtures? From training to enhancement. Thank you!

关于不同生成snr带躁语音训练的问题

你好,感谢你们的工作,
我想请问一下,目前你们原始代码只能对一个固定信噪比的条件下进行训练和测试,
但是如果你论文所说,我们需要去解决对不同 SNR的语音进行增强, 所以我希望能用不同的snr的语音数据拿去训练得到一个比较general的模型. 之前也有类似的issue问过这个问题,只是我不太理解那个答案.

我的做法是在pack features那一步把之前生成的不同snr的mixture features集合起来到一个data.h5中去, 但问题是,这样在compute scaler的时候生成scaler.p文件会有问题,因为data.h5的信噪比不同了. 导致了在main_dnn.py里面训练时候scale data时会有问题, 请问你们是如何做的

谢谢

为什么输出y选择7帧语音中间的一帧呢

训练阶段，x是连续7帧语音，y是中间的一帧，为什么这样设定呢？会对恢复语音精度有影响吗

Script for testing trained model on own sample recording

Hi Yong Xu,
I have finished training model. I am stuck on testing the output of trained model on new recording. Is there a script for this? Or if someone could point out the changes in prepare_data.py and main_dnn.py for this.
The issue is I have a recording with noise so cannot define signal to noise ratio.

Thank you
Akshaya

pretrained model for mixture2clean_dnn?

Hi ! Thanks for the great work, I found that there is pretrained model for clean2clean_verify, and it works well for my custom wave samples. But why don't you share pretrained mdoel for mixture2clean_dnn?

Thanks!

Pre-trained model usage

Hello!
I've discovered that you've shared a pre-trained model on TIMIT dataset, but is it possible to run it on own data?
I've managed to reuse the trained model, generated in the process of running sednn/mixture2clean_dnn/runme.sh. I trained the model on the subsample of LibriSpeech dataset, but I'm not too happy with the quality of enhancing. When I tried to substitute LibriSpeech model with your TIMIT model, I couldn't run the decoding due to the input shapes mismatch.
So, my question is, how can I get to work the pre-trained TIMIT model on my own data in python? My assumpion is that default input data format differs in the clean2clean_verify and mixture2clean_dnn subdirectories. In that case where can I retrieve the information (scripts in the subdirectory sednn/clean2clean_verify/; papers)?

Best regards,
Yakovenko Olga

关于data_prepare.py中的bug

函数calculate_mixture_features中，对于语音数据首先做了一个read_audio函数，如果speech和noise的fs不等于16000的话会对其进行重采样。但是后面的操作，如果len(noise_audio) >len(speech_audio)的情况下，noise_audio = noise_audio[noise_onset : noise_offset]
我现在遇到的情况是语音长度为63000，噪声n72.wav长度为80000，fs = 20000，将其重采样到16000后长度变成了64000，而后面修正长度的时候还是用了noise_onset:noise_offset。而noise_onset和noise_offset在取值的时候是在没有重采样的情况下取的，例如onset = 16221，offset = 79812。这样的话当noise被重采样以后就会导致speech和noise的长度不一样。。所以在生成csv的时候最好也给read_audio加入采样率的参数

some question of pad_with_border

Hello:
Really impressed by your work and got a few questions in terms of how you process the data.

Do pad_with_border mean this?

Many thanks,
Nick

多输入，多输出问题

您好，看了你的论文，非常厉害，
问题1：想实现一下其中的噪声告知训练，选择语音前几帧作为平均之后作为输入，请问是是不是简单的concate在之前7帧能量的后面吗？？例如语音7帧为7256，噪声为1256,是否是将它拼接成8*256，送进网络。
问题2：拜读了你师弟高天博士的论文，他提到了双输出dnn增强网络,如果之前的输入是问题一所说的那样，双输出的话，请问输入依然是前几帧噪声的平均？目标参数是噪声7帧中的一帧吗？

总结下：关于这个多输入，和多输出，我想问的是，简单的在输入后面拼接矩阵，还是需要在网络上添加新的输入？谢谢了

建议：关于论文中和代码不太一致问题

您好，
在读您的论文，An experimental study on speech enhancement based on deep neural networks 和 A regression approach to speech enhancement based on deep neural networks. 发现文中提取特征都是用的
log-power spectral features。而您的python代码实现中是用log spectral features来做的，abs只取了一次。

some questions

@yongxuUSTC @qiuqiangkong
您好，读了mixture2clean_dnn中的代码之后，我有一些不明白的地方，期待您能在百忙之中抽空解答一下：

1.runme.sh 文件中，第26行，设置了train SNR为0，test SNR为0，是不是每种SNR单独训练出一个模型？论文《A Regression Approach to Speech Enhancement Based on Deep Neural Networks》中也是这么做的吗？

2.prepare_data.py 文件中，第260行，求频谱时有magnitude和complex两种选项，当输入是纯净语音时，使用的是magnitude选项，当输入是含噪语音时，使用的是complex选项。为什么会有这种区别？

3.prepare_data.py文件中，第313行，将所有的feature保存到y_all变量中。如果训练集太大，feature太多，电脑内存能够容纳这么多特征吗？

4.prepare_data.py 文件中，第382行，计算feature的scaler，没有计算label的scaler。第389行，保存feature的scaler。之后在main_dnn.py的第86行，读取feature的scaler用于标准化。但label可以用feature的scaler进行标准化吗？

谢谢！

about pesq

Hi @yongxuUSTC
I can't find the pesq_tool document in (https://www.itu.int/rec/T-REC-P.862-200102-I/en) and compile the code by $ gcc -o pesq pesq_tool/*.c -lm

I just find the T-REC-P.862-200102-I!!SOFT-ZST-E.zip
the strucure is as follows:
T-REC-P.862-200102-I!!SOFT-ZST-E
---P862
---Software
---Conform
---source
---readme.txt
---P862E.doc
---P862.pdf

so I don't know how to compile these file and get the compiled executable pesq to mixture2clean_dnn/, can you give me the detail of this issue?

为啥降噪后的语音能量比原始语音还大

packed_features、trainning_stats、models文件没有生成

您好，我在换上使用mini_data训练时可以成功得到结果，但是在换成自己数据训练时没有生成packed_features、trainning_stats、models这三个文件夹，请问这个会是什么原因造成的呢。

pytorch: CUDA out of memory

您好！我在尝试运行pytorch部分代码时会提示“CUDA out of memory”，具体如下：
File "/sednn/mixture2clean_dnn_pytorch/prepare_data.py", line 335, in log return torch.log(x + eps) RuntimeError: CUDA out of memory. Tried to allocate 3.50 MiB (GPU 0; 15.90 GiB total capacity; 15.02 GiB already allocated; 1.88 MiB free; 62.70 MiB cached) ('cuda:', True)
在CPU上运行时正常
所用数据仅仅为mini_data，不知道是什么原因，希望您能告诉我，谢谢！

Wrong noise trunication?

In prepare_data.py in calculate_mixture_features

# Truncate noise to the same length as speech. 
else:
    noise_audio = noise_audio[noise_onset : noise_offset]

Shouldn't the range here be noise_audio[:len(speech_audio)] for instance? Otherwise it results in shapes mismatch later in the code.

Some questions in terms of Features Extraction

Really impressed by your work and got a few questions in terms of how you process the data.

In pack_feature function, you convert the 2D array to 3D array (mat_2d_to_3d function). And use the 3D array as the input to DNN model. What is the reason to do this process? Why not just use 2D array as the input to the DNN model?

Many thanks,
Robert

Problem with "magnitude" mode

Hi Yong Xu,
I have read your code and meet some problem of the "magnitude" mode in func calc_sp

sednn/mixture2clean_dnn/prepare_data.py

Line 261 in c7efa60

x = x.astype(np.float32)

The operation "astype" just throws away the imag part of a complex, which i have tested under numpy 1.13.3. Use np.abs to take magnitude. Is this a bug or something else ?