zjukg / meaformer

[Paper][ACM MM 2023] MEAformer: Multi-modal Entity Alignment Transformer for Meta Modality Hybrid

🏖️ MEAformer: Multi-modal Entity Alignment Transformer for Meta Modality Hybrid

This paper introduces MEAformer, a multi-modal entity alignment transformer approach for meta modality hybrid, which dynamically predicts the mutual correlation coefficients among modalities for more fine-grained entity-level modality fusion and alignment.

👆 Click to see the Video

🔔 News

• 2024-03 Our paper NativE: Multi-modal Knowledge Graph Completion in the Wild [Repo] is accepted by SIGIR 2024 !.
• 2024-02 We preprint our Survey Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Survey [Repo].
• 2024-02 We release the [Repo] for our paper Unleashing the Power of Imbalanced Modality Information for Multi-modal Knowledge Graph Completion , COLING 2024
• 2024-02 We preprint our Paper ASGEA: Exploiting Logic Rules from Align-Subgraphs for Entity Alignment [Repo].
• 2024-01 Our paper Revisit and Outstrip Entity Alignment: A Perspective of Generative Models [Repo] is accepted by ICLR 2024 !
• 2023-07 We release the [Repo] for our paper: Rethinking Uncertainly Missing and Ambiguous Visual Modality in Multi-Modal Entity Alignment ! [Slide], ISWC 2023
• 2023-04 We release the complete code and data for MEAformer ! [Slide] [Vedio], ACM MM 2023

🔬 Dependencies

pip install -r requirement.txt

Details

• Python (>= 3.7)
• PyTorch (>= 1.6.0)
• numpy (>= 1.19.2)
• Transformers (== 4.21.3)
• easydict (>= 1.10)
• unidecode (>= 1.3.6)
• tensorboard (>= 2.11.0)

🚀 Train

• Quick start: Using script file (run.sh)

>> cd MEAformer
>> bash run.sh

• Optional: Using the bash command

>> cd MEAformer
# -----------------------
# ---- non-iterative ----
# -----------------------
# ----  w/o surface  ---- 
# FBDB15K
>> bash run_meaformer.sh 1 FBDB15K norm 0.8 0 
>> bash run_meaformer.sh 1 FBDB15K norm 0.5 0 
>> bash run_meaformer.sh 1 FBDB15K norm 0.2 0 
# FBYG15K
>> bash run_meaformer.sh 1 FBYG15K norm 0.8 0 
>> bash run_meaformer.sh 1 FBYG15K norm 0.5 0 
>> bash run_meaformer.sh 1 FBYG15K norm 0.2 0 
# DBP15K
>> bash run_meaformer.sh 1 DBP15K zh_en 0.3 0 
>> bash run_meaformer.sh 1 DBP15K ja_en 0.3 0 
>> bash run_meaformer.sh 1 DBP15K fr_en 0.3 0
# ----  w/ surface  ---- 
# DBP15K
>> bash run_meaformer.sh 1 DBP15K zh_en 0.3 1 
>> bash run_meaformer.sh 1 DBP15K ja_en 0.3 1 
>> bash run_meaformer.sh 1 DBP15K fr_en 0.3 1
# -----------------------
# ------ iterative ------
# -----------------------
# ----  w/o surface  ---- 
# FBDB15K
>> bash run_meaformer_il.sh 1 FBDB15K norm 0.8 0 
>> bash run_meaformer_il.sh 1 FBDB15K norm 0.5 0 
>> bash run_meaformer_il.sh 1 FBDB15K norm 0.2 0 
# FBYG15K
>> bash run_meaformer_il.sh 1 FBYG15K norm 0.8 0 
>> bash run_meaformer_il.sh 1 FBYG15K norm 0.5 0 
>> bash run_meaformer_il.sh 1 FBYG15K norm 0.2 0 
# DBP15K
>> bash run_meaformer_il.sh 1 DBP15K zh_en 0.3 0 
>> bash run_meaformer_il.sh 1 DBP15K ja_en 0.3 0 
>> bash run_meaformer_il.sh 1 DBP15K fr_en 0.3 0
# ----  w/ surface  ---- 
# DBP15K
>> bash run_meaformer_il.sh 1 DBP15K zh_en 0.3 1 
>> bash run_meaformer_il.sh 1 DBP15K ja_en 0.3 1 
>> bash run_meaformer_il.sh 1 DBP15K fr_en 0.3 1

❗Tips: you can open the run_meaformer.sh or run_meaformer_il.sh file for parameter or training target modification.

🎯 Results

$\bf{H@1}$ Performance with the Settings: w/o surface & Non-iterative in UMAEA. We modified part of the MSNEA to involve not using the content of attribute values but only the attribute types themselves (See issues for details):

Method	$\bf{DBP15K_{ZH-EN}}$	$\bf{DBP15K_{JA-EN}}$	$\bf{DBP15K_{FR-EN}}$
MSNEA	.609	.541	.557
EVA	.683	.669	.686
MCLEA	.726	.719	.719
MEAformer	.772	.764	.771
UMAEA	.800	.801	.818

📚 Dataset

• ❗NOTE: Download from GoogleDrive (1.26G) and unzip it to make those files satisfy the following file hierarchy:

ROOT
├── data
│   └── mmkg
└── code
    └── MEAformer

• Case analysis Jupyter script: GoogleDrive (180M) base on the raw images of entities (need to be unzip). I hope this gives you a good understanding of this dataset.
• [ Option ] The raw Relations & Attributes appeared in DBP15k and case from MEAformer can be downloaded from Huggingface (150M).
• [ Option ] The raw images of entities appeared in DBP15k can be downloaded from Baidu Cloud Drive (50GB) with the pass code mmea. All images are saved as title-image pairs in dictionaries and can be accessed with the following code :

import pickle
zh_images = pickle.load(open("eva_image_resources/dbp15k/zh_dbp15k_link_img_dict_full.pkl",'rb'))
print(en_images["http://zh.dbpedia.org/resource/香港有線電視"].size)

Code Path

👈 🔎 Click

MEAformer
├── config.py
├── main.py
├── requirement.txt
├── run_meaformer.sh
├── run_meaformer_il.sh
├── run.sh
├── model
│   ├── __init__.py
│   ├── layers.py
│   ├── MEAformer_loss.py
│   ├── MEAformer.py
│   ├── MEAformer_tools.py
│   └── Tool_model.py
├── src
│   ├── __init__.py
│   ├── distributed_utils.py
│   ├── data.py
│   └── utils.py
└── torchlight
    ├── __init__.py
    ├── logger.py
    ├── metric.py
    └── utils.py

Data Path

👈 🔎 Click

mmkg
├── DBP15K
│   ├── fr_en
│   │   ├── ent_ids_1
│   │   ├── ent_ids_2
│   │   ├── ill_ent_ids
│   │   ├── training_attrs_1
│   │   ├── training_attrs_2
│   │   ├── triples_1
│   │   └── triples_2
│   ├── ja_en
│   │   ├── ent_ids_1
│   │   ├── ent_ids_2
│   │   ├── ill_ent_ids
│   │   ├── training_attrs_1
│   │   ├── training_attrs_2
│   │   ├── triples_1
│   │   └── triples_2
│   ├── translated_ent_name
│   │   ├── dbp_fr_en.json
│   │   ├── dbp_ja_en.json
│   │   └── dbp_zh_en.json
│   └── zh_en
│       ├── ent_ids_1
│       ├── ent_ids_2
│       ├── ill_ent_ids
│       ├── training_attrs_1
│       ├── training_attrs_2
│       ├── triples_1
│       └── triples_2
├── FBDB15K
│   └── norm
│       ├── ent_ids_1
│       ├── ent_ids_2
│       ├── ill_ent_ids
│       ├── training_attrs_1
│       ├── training_attrs_2
│       ├── triples_1
│       └── triples_2
├── FBYG15K
│   └── norm
│       ├── ent_ids_1
│       ├── ent_ids_2
│       ├── ill_ent_ids
│       ├── training_attrs_1
│       ├── training_attrs_2
│       ├── triples_1
│       └── triples_2
├── embedding
│   └── glove.6B.300d.txt
├── pkls
│   ├── dbpedia_wikidata_15k_dense_GA_id_img_feature_dict.pkl
│   ├── dbpedia_wikidata_15k_norm_GA_id_img_feature_dict.pkl
│   ├── FBDB15K_id_img_feature_dict.pkl
│   ├── FBYG15K_id_img_feature_dict.pkl
│   ├── fr_en_GA_id_img_feature_dict.pkl
│   ├── ja_en_GA_id_img_feature_dict.pkl
│   └── zh_en_GA_id_img_feature_dict.pkl
├── MEAformer
└── dump

🤝 Cite:

Please condiser citing this paper if you use the code or data from our work. Thanks a lot :)

@inproceedings{DBLP:conf/mm/ChenCZGFHZGPSC23,
  author       = {Zhuo Chen and
                  Jiaoyan Chen and
                  Wen Zhang and
                  Lingbing Guo and
                  Yin Fang and
                  Yufeng Huang and
                  Yichi Zhang and
                  Yuxia Geng and
                  Jeff Z. Pan and
                  Wenting Song and
                  Huajun Chen},
  title        = {MEAformer: Multi-modal Entity Alignment Transformer for Meta Modality
                  Hybrid},
  booktitle    = {{ACM} Multimedia},
  pages        = {3317--3327},
  publisher    = {{ACM}},
  year         = {2023}
}

💡 Acknowledgement

We appreciate MCLEA, MSNEA, EVA, MMEA and many other related works for their open-source contributions.

meaformer's Issues

thank you for your work

Hello author, first of all thank you very much for your work, I would like to ask you about how long it takes to train the DBP15K dataset, you said in the literature that there is no parallelism, what should the batch_size be set to if only a single card 3090 is used, you provided 3000 in your code

数据集问题

作者，您好， DBP15K 这个数据集是否有关系对应id的文件，我想在关系上继续做研究，奈何找了很多论文，并没有与这套映射配套的关系名

Could you share the code that replicates the results of the MSNEA model?

Hi friends,

I can't reproduce the results of the MSNEA model. Could you share the code that replicates the results of the MSNEA model?

Missing dataset DWY

It seems the dataset allowed as per the configuration file is not available in the google drive as of 14/09/2023. Anyone has that data in a compatible format? Or is there any instructions on how to download and prepare that dataset. I am interested in a mono-lingual dataset.

模型的test问题

您好，作者，首先非常感谢您的工作！！
我在代码训练最后一轮进行的test结果如下图所示，与论文中展示的数值差不多

但是我将训练好的模型单独进行test的结果会有较大差异呢，下图是修改run_meaformer.sh文件中only_test=1，并在main.py中_load_model方法中加载训练好的模型，test的结果

请问一下作者，是我进行test的方法不对吗？

w/o sf 是指name特征是随机初始化吗