bztbzt / mytextee

TextEE

Updates | Datasets | Models | Environment | Running | Results | Website | Paper

Authors: Kuan-Hao Huang, I-Hung Hsu, Tanmay Parekh, Zhiyu Xie, Zixuan Zhang, Premkumar Natarajan, Kai-Wei Chang, Nanyun Peng, Heng Ji

Introduction

TextEE is a standardized, fair, and reproducible benchmark for evaluating event extraction approaches.

• Standardized data preprocessing for 10+ datasets.
• Standardized data splits for reducing performance variance.
• 10+ implemented event extraction approaches published in recent years.
• Comprehensive reevaluation results for future references.

Please check mroe details our paper A Reevaluation of Event Extraction: Past, Present, and Future Challenges. We will keep adding new datasets and new models!

Updates

• [2023/11/15] We release TextEE, a framework for reevaluation and benchmark for event extraction. Feel free to contact us ([email protected]) if you want to contribute your models or datasets!

Supported Datasets

Dataset Name	Task	Paper Title	Venue
ACE05	E2E, ED, EAE	The Automatic Content Extraction (ACE) Program - Tasks, Data, and Evaluation	LREC 2004
ERE	E2E, ED, EAE	From Light to Rich ERE: Annotation of Entities, Relations, and Events	EVENTS@NAACL 2015
M2E2	E2E, ED, EAE	Cross-media Structured Common Space for Multimedia Event Extraction	ACL 2020
CASIE	E2E, ED, EAE	CASIE: Extracting Cybersecurity Event Information from Text	AAAI 2020
PHEE	E2E, ED, EAE	PHEE: A Dataset for Pharmacovigilance Event Extraction from Text	EMNLP 2022
MEE	ED	MEE: A Novel Multilingual Event Extraction Dataset	EMNLP 2022
FewEvent	ED	Meta-Learning with Dynamic-Memory-Based Prototypical Network for Few-Shot Event Detection	WSDM 2020
MAVEN	ED	MAVEN: A Massive General Domain Event Detection Dataset	EMNLP 2020
RAMS	EAE	Multi-Sentence Argument Linking	ACL 2020
WikiEvents	EAE	Document-Level Event Argument Extraction by Conditional Generation	NAACL 2021
GENEVA	EAE	GENEVA: Benchmarking Generalizability for Event Argument Extraction with Hundreds of Event Types and Argument Roles	ACL 2023

Supported Models

Model Name	Task	Paper Title	Venue
DyGIE++	E2E	Entity, Relation, and Event Extraction with Contextualized Span Representations	EMNLP 2019
OneIE	E2E	A Joint Neural Model for Information Extraction with Global Features	ACL 2020
AMR-IE	E2E	Abstract Meaning Representation Guided Graph Encoding and Decoding for Joint Information Extraction	NAACL 2021
DEGREE	E2E, ED, EAE	DEGREE: A Data-Efficient Generation-Based Event Extraction Model	NAACL 2022
EEQA	ED, EAE	Event Extraction by Answering (Almost) Natural Questions	EMNLP 2020
RCEE	ED, EAE	Event Extraction as Machine Reading Comprehension	EMNLP 2020
Query&Extract	ED, EAE	Query and Extract: Refining Event Extraction as Type-oriented Binary Decoding	ACL-Findings 2022
TagPrime	ED, EAE	TAGPRIME: A Unified Framework for Relational Structure Extraction	ACL 2023
UniST	ED	Unified Semantic Typing with Meaningful Label Inference	NAACL 2022
BART-Gen	EAE	Document-Level Event Argument Extraction by Conditional Generation	NAACL 2021
PAIE	EAE	Prompt for Extraction? PAIE: Prompting Argument Interaction for Event Argument Extraction	ACL 2022
X-Gear	EAE	Multilingual Generative Language Models for Zero-Shot Cross-Lingual Event Argument Extraction	ACL 2022
AMPERE	EAE	AMPERE: AMR-Aware Prefix for Generation-Based Event Argument Extraction Model	ACL 2023

Reevaluation Results

Please check here.

Environment

1. Please install the following packages from both conda and pip.

conda install
  - python 3.8
  - pytorch 2.0.1
  - numpy 1.24.3
  - ipdb 0.13.13
  - tqdm 4.65.0
  - beautifulsoup4 4.11.1
  - lxml 4.9.1
  - jsonlines 3.1.0
  - jsonnet 0.20.0
  - stanza=1.5.0

pip install
  - transformers 4.30.0
  - sentencepiece 0.1.96
  - scipy 1.5.4
  - spacy 3.1.4
  - nltk 3.8.1
  - tensorboardX 2.6
  - keras-preprocessing 1.1.2
  - keras 2.4.3
  - dgl-cu111 0.6.1
  - amrlib 0.7.1
  - cached_property 1.5.2
  - typing-extensions 4.4.0

Alternatively, you can use the following command.

conda env create -f env.yml

2. Run the following command.

python -m spacy download en_core_web_lg

Running

Training

./scripts/train.sh [config]

Evaluation for End-to-End Model

# Evaluating End-to-End
python TextEE/evaluate_end2end.py --task E2E --data [eval_data] --model [saved_model_folder]

# Evaluating EAE
python TextEE/evaluate_end2end.py --task EAE --data [eval_data] --model [saved_model_folder]

Evaluation for Pipeline Model

# Evaluating ED
python TextEE/evaluate_pipeline.py --task ED --data [eval_data] --ed_model [saved_model_folder]

# Evaluating EAE
python TextEE/evaluate_pipeline.py --task EAE --data [eval_data] --eae_model [saved_model_folder]

# Evaluating ED+EAE
python TextEE/evaluate_pipeline.py --task E2E --data [eval_data] --ed_model [saved_model_folder] --eae_model [saved_model_folder]

Citation

@article{Huang23textee,
  author       = {Kuan{-}Hao Huang and
                  I{-}Hung Hsu and
                  Tanmay Parekh and 
                  Zhiyu Xie and
                  Zixuan Zhang and
                  Premkumar Natarajan and
                  Kai{-}Wei Chang and
                  Nanyun Peng and
                  Heng Ji},
  title        = {A Reevaluation of Event Extraction: Past, Present, and Future Challenges},
  journal      = {arXiv preprint arXiv:2311.09562},
  year         = {2023},
}