zhuoyulang / if-mmin

IF-MMIN

This repo implements the Invariant Feature aware Missing Modality Imagination Network(IF-MMIN) for the following paper: "Exploiting modality-invariant feature for robust multimodal emotion recognition with missing modalities"

Environment

python 3.8.0
pytorch >= 1.0.0

Usage

First you should change the data folder path in data/config and preprocess your data follwing the code in preprocess/.

You can download the preprocessed feature to run the code.

• For Training IF-MMIN on IEMOCAP:

  First training a model fusion model with all audio, visual and lexical modality as the pretrained encoder.

  bash scripts/CAP_utt_shared.sh AVL [num_of_expr] [GPU_index]

  Then

  bash scripts/CAP_IFMMIN.sh [num_of_expr] [GPU_index]

Note that you can run the code with default hyper-parameters defined in shell scripts, for changing these arguments, please refer to options/get_opt.py and the modify_commandline_options method of each model you choose.

Download the features

Baidu Yun Link IEMOCAP A V L modality Features 链接: https://pan.baidu.com/s/1WmuqNlvcs5XzLKfz5i4iqQ 提取码: gn6w

if-mmin's Issues

Questions about IFIM module encoder inputs

Hello, you wrote in your article that the input of the next encoder in the IFIM module is the output of the previous encoder, but in the code you provided, in the ResidualAE module it says
x_in = x
x_out = shared
latents = []
for i in range(self.n_blocks):
encoder = getattr(self, 'encoder_' + str(i))
decoder = getattr(self, 'decoder_' + str(i))
x_in = x_in + shared # x_out
latent = encoder(x_in)
x_out = decoder(latent)
Should these two lines "x_in = x" "x_in = x_in + shared" be "x_out = x" "x_in = x_out + shared"?

win下运行

您好，如果要在Windows下运行，需要做什么修改？和Linux下运行结果差异大吗？谢谢

May I ask if there have been any studies or research conducted on the IEMOCAP dataset involving a 6-class classification task?

May I ask if there have been any studies or research conducted on the IEMOCAP dataset involving a 6-class classification task?

Questions about input data

Hello, I have some questions about the input data. First, I see that when the original features you provided are loaded, all the three modal features have three dimensions, the first dimension is batch size, and the third dimension is feature dimension, what does the second dimension represent, and why does only the second dimension of the acoustic features change with the samples? Second, the IEMOCAP dataset consists of conversations containing a variable number of utterances, so in your input data, does a batch represent a batch of utterances or a batch of conversations?

数据处理的问题

您好，如果我想从原始IEMOCAP数据集提取可以输入网络的预处理数据，应该以怎么样的顺序运行preprocess文件夹里的代码？谢谢指教。