nustm / mecpe

[TAFFC 2022] Multimodal Emotion-Cause Pair Extraction in Conversations

Multimodal Emotion-Cause Pair Extraction in Conversations

This repository contains the dataset and code for our TAFFC 2022 paper: Multimodal Emotion-Cause Pair Extraction in Conversations Please cite our paper according to the official format.

🌟 Our task paper for SemEval-2024 is available here.

🌟 We have organized a SemEval task based on our ECF dataset, and the source data of the three modalities has been released to Google Drive. Welcome to participate in the competition. Visit SemEval-2024 Task 3: Multimodal Emotion Cause Analysis in Conversations.

Dependencies

• Python 3.6.9 (tested on cuda 10.2 and NVIDIA RTX TITAN)
• Tensorflow 1.15.4
• BERT (The pretrained BERT model "BERT-Base, Cased" is required.)

Note

For your convenience, we also provide our requirements.txt.cuda11+ (tested on cuda 11.3 and python 3.6.13).

Usage

Step1:

# For Task 1: MECPE
python -u step1.py --use_x_a yes  --use_x_v yes --scope BiLSTM_A_V
python -u step1.py --model_type BERTcased --use_x_a yes  --use_x_v yes --scope BERT_A_V
# For Task 2: MECPE-Cat
python -u step1.py --choose_emocate yes --use_x_a yes  --use_x_v yes --scope BiLSTM_A_V_emocate
python -u step1.py --model_type BERTcased --choose_emocate yes --use_x_a yes  --use_x_v yes --scope BERT_A_V_emocate

Step2:

python -u step2.py  --use_x_a yes  --use_x_v yes   --scope  BiLSTM_A_V

Citation

@ARTICLE{wang2023multimodal,
  author={Wang, Fanfan and Ding, Zixiang and Xia, Rui and Li, Zhaoyu and Yu, Jianfei},
  journal={IEEE Transactions on Affective Computing}, 
  title={Multimodal Emotion-Cause Pair Extraction in Conversations}, 
  year={2023},
  volume={14},
  number={3},
  pages={1832-1844},
  doi = {10.1109/TAFFC.2022.3226559}
}

@InProceedings{wang2024SemEval,
  author={Wang, Fanfan  and  Ma, Heqing  and  Xia, Rui  and  Yu, Jianfei  and  Cambria, Erik},
  title={SemEval-2024 Task 3: Multimodal Emotion Cause Analysis in Conversations},
  booktitle={Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024)},
  month={June},
  year={2024},
  address={Mexico City, Mexico},
  publisher={Association for Computational Linguistics},
  pages={2022--2033},
  url = {https://aclanthology.org/2024.semeval2024-1.273}
}

mecpe's Issues

about image and audio feature extraction

I'm interested in your datasets. But when I read the code, I find you have provided a processed visual and acoustic feature. Can you provide me with the processing code for video and audio?

About the Extraction of Audio Embeddings

ffmpeg -i video -ab 160k -ac 2 -ar 44100 -vn input.wav

opensmile/build/progsrc/smilextract/SMILExtract -C opensmile/config/is09-13/IS09_emotion.conf -I input.wav -O output.txt

I used the above command to process video files and extract audio features using the IS09 feature set of opensmile. However, there is a significant difference in dimensionality compared to the 6373 dimensions used in the experiment. Could you please provide more details on this part? How were the 6373-dimensional features extracted from the video?

Thank you very much!

21世纪了，来点现代框架吧

能给个pytorch实现吗，tensorflow静态图很难调试

你们可以提供下 dev和test的json文件吗？

你们可以提供下 dev和test的json文件吗？不提供json文件，没有办法计算指标。

Where is the multimodal emotion cause annotation toolkit?

Where is the multimodal emotion cause annotation toolkit?

Why is the video encoding output mostly negative values？

Why is the video encoding output mostly negative values?

How can I get ralvent files of RoBERTa to change the current model？

video_embedding_4096.npy

cannot reshape array of size 3276784 into shape (13620,4096)