sterling239 / audio-emotion-recognition

Audio emotion recognition

Description

The goal of this project is to provide script to verify our emotion recognition approach.

Requirements

• numpy
• sklearn
• librosa (for pitch estimation, optional)
• keras

Project structure

The code consist of 3 main .py files.

• model.py - containt keras implementation of LSTM, MPC neural networks, sklearn models

Main methods to use in emotion recognition:

• train_mpc(train_x, train_y, test_x, test_y) - train multilayer perceptron.
• train_lstm(train_x, train_y, test_x, test_y) - train LSTM NN
• train_rfc(train_x, train_y, options) - train Random Forest classifier

All methods return Model object and also show precision and loss on test sample to inline validation.

• calculate_features.py - script for features estimation

Method calculate_features(signal, freq, options) returns features set for all 0.2 sec frames in signal. Freq is a signal framerate. If use_derivatives flag is true, method also include 1st and 2nd time deltas of features. To calculate features cf.py code is used. It based on https://github.com/tyiannak/pyAudioAnalysis/blob/master/audioFeatureExtraction.py with minor improvements.

As default returns 32 mfcc, spectral and chromagram features

• code_yan_lstm.py - implementation of LSTM NN in emotion recognition.

The code can be divided into 3 blocks

• Data reading
• Data preprocessing
• The main part includes model buildg and validation

To get the data successfully, choose regime from ['aibo4', 'aibo5', 'iemocap'] and correct paths to wavs, labels, etc. aibo4 and aibo5 regimes correspondings to different labels over AIBO database.

Data preprocessing consist of normalization, padding sequences, transfromation labels from categorical to vector, balancing and resampling.

The main part is a 5-fold cross-validation procedure over readed database. It splits sample into train and test parts, trains models on train and validates on test. By ending cross-validation procedure it plots confusion matrix aomparing prediction and expected output.

Running

To run the code, enter python code_yan_lstm.py in a command line or IDE. In addition to correction paths and settings, create folder for feature samples in the working directory.

audio-emotion-recognition's Issues

ImportError: cannot import name Merge

I was trying to run your work, but I got an error as follows.

/usr/local/lib/python2.7/dist-packages/h5py/init.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
from ._conv import register_converters as _register_converters
Using TensorFlow backend.
Traceback (most recent call last):
File "code_yan_lstm.py", line 11, in
import models
File "/opt/xuyan/tensorflow/audio-emotion-recognition-master/models.py", line 11, in
from keras.layers.core import Dense, Dropout, Activation, Merge, TimeDistributedDense
ImportError: cannot import name Merge

Error in reading wave file

Traceback (most recent call last):
File "code_yan_lstm.py", line 390, in
data = np.array(read_data())
File "code_yan_lstm.py", line 233, in read_data
return read_iemocap_data()
File "code_yan_lstm.py", line 217, in read_iemocap_data
wav = open_wav(path_to_wav, f + '.wav')
File "code_yan_lstm.py", line 64, in open_wav
wav = wave.open(path_to_wav + filename, mode="r")
File "/usr/lib/python3.5/wave.py", line 499, in open
return Wave_read(f)
File "/usr/lib/python3.5/wave.py", line 163, in init
self.initfp(f)
File "/usr/lib/python3.5/wave.py", line 130, in initfp
raise Error('file does not start with RIFF id')
wave.Error: file does not start with RIFF id

I'm using same IEMOCAP database but above error is showing up.
Please help me out.