tyiannak / multimodalanalysis

Python examples for the course "Multimodal Information Processing & Analysis" of the MSc in Data Science in NCSR Demokritos

multimodalAnalysis

Related News

Special issue in Pattern Recognition in Multimedia Signal Analysis, Deadline 2021 28 February

General

This code contains sample code for the Courses

• "Machine Learning for Multimodal Data" of the MSc in Artificial Intelligence, of the University of Pireaus and the National Centre for Scientific Research "Demokritos".
• "Multimodal Information Processing and Analysis" of the MSc in Data Science, of the National Centre for Scientific Research "Demokritos" and the University of Pelloponese. .

This material covers introductory issues with regards to audio segmentation and classification, image processing, image feature extraction, segmentation and classification, video analysis and multimodal fusion.

Dependencies

All code has been tested in Python3. Dependencies can be installed using pip and the requirements.txt files in each folder (e.g. audio/requirements.txt).

Course Presentations

Link	Title
Course 1	Intro to Multimodal Signal Analysis
Course 2	Audio Representations and Feature Extraction
Course 3	Audio Classification / Regression
Course 4	Audio Segmentation
Course 5	Image Feature Extraction - 1
Course 6	Image Feature Extraction - 2
Course 7	Video Feature Extraction - 1
Course 8	Audio Fingerprinting
Course 9	DL 1
Course 10	DL 2

Author

Theodoros Giannakopoulos, Director of Machine Learning at Behavioral Signals

multimodalanalysis's Issues

Example 10->file_path and folder ../data/general not included

Hello, I am from the MsC !

Example 10->male and female sound files don't exist in the path given and folder ../data/general not included
The dir "../data/general" doesn't exist. Also when I tried to check in the dir "../data/gender/..", only this file exists m1_neu-m1-l1.wav.wav. I can't find f1_neu-f1-l2.wav.wav, it seems that there are similar female sound wav files but they are not the same.

fp1 = "../data/general/speech/m1_neu-m1-l1.wav.wav" # male
fp2 = "../data/general/speech/f1_neu-f1-l2.wav.wav" # female

eyed3 and/or libmagic dependency problem

I've tried running the audio/example08.py with no success.
With eyed3==0.7.11 and eyed3==0.8.0 it produces the following error:

Error: file not found or other I/O error. (DECODING FAILED)
Traceback (most recent call last):
  File "example08.py", line 16, in <module>
    duration = len(s) / float(fs)
TypeError: object of type 'int' has no len()

and with latest eyed3 the following error occurs

Traceback (most recent call last):
  File "example08.py", line 9, in <module>
    from pyAudioAnalysis import audioFeatureExtraction as aF
  File "/Users/vassiliskrikonis/Projects/pyAudioAnalysis/pyAudioAnalysis/audioFeatureExtraction.py", line 10, in <module>
    from pyAudioAnalysis import audioBasicIO
  File "/Users/vassiliskrikonis/Projects/pyAudioAnalysis/pyAudioAnalysis/audioBasicIO.py", line 2, in <module>
    import os, glob, eyed3, ntpath, shutil, numpy
  File "/anaconda3/lib/python3.6/site-packages/eyed3/__init__.py", line 32, in <module>
    from .utils.log import log                                          # noqa: E402
  File "/anaconda3/lib/python3.6/site-packages/eyed3/utils/__init__.py", line 66, in <module>
    _mime_types = MagicTypes()
  File "/anaconda3/lib/python3.6/site-packages/eyed3/utils/__init__.py", line 54, in __init__
    keep_going=False)
TypeError: __init__() got an unexpected keyword argument 'mime'
Exception ignored in: <bound method Magic.__del__ of <eyed3.utils.MagicTypes object at 0x116a22940>>
Traceback (most recent call last):
  File "/anaconda3/lib/python3.6/site-packages/magic/__init__.py", line 19, in __del__
    ffi.close(self.cookie)
AttributeError: 'MagicTypes' object has no attribute 'cookie'

(I'm running MacOS 10.13 and install libmagic with brew)

how to depend audio file duration and frame_size and step

Hi Tyiannal,

frame_size =0.050
step=0.025
[Fs, x] = audioBasicIO.readAudioFile("sample.wav");
F, f_names = audioFeatureExtraction.stFeatureExtraction(x, Fs, 0.050Fs, 0.025Fs);

I want to create a cnn model to predict human emotion from the audio file. To train the model I m using the different duration of audio files, since should I need to modify frame_size and step.
Can u explain how audio duration and frame_size and step depend on each other?

Thank u

Hearing aid