zjunlp / knn-kg

[NLPCC 2023] Reasoning Through Memorization: Nearest Neighbor Knowledge Graph Embeddings with Language Models

KNN-KG

Code for the NLPCC2023 paper "Reasoning Through Memorization: Nearest Neighbor Knowledge Graph Embeddings with Language Models".

Requirements

To install requirements:

pip install -r requirements.txt

Run the experiments

Training

Entity Embedding Initialization

Use the command below to add entities to BERT and train the entity embedding layer to use in the later training. For another dataset WN18RR just replacing the dataset name will be fine.

./scripts/pretrain_fb15k.sh

The parameters of Entity Embedding Layer trained will be used in the next Entity prediction task.

Entity Prediction Task

Use the command below to train the model to predict the correct entity in the masked position.

./scripts/fb15k-237/fb15k.sh

Consturct Knowledge Store

After training the model in Entity prediction task, we use the model to get the knowledge store built from triples and descriptions.

./scripts/fb15k-237/get_knowledge_store.sh

Inference

Here we have a trained model and our knowledge store (e.g., faiss.dump file), use the command below to inference in the test set.

./scripts/fb15k-237/inference.sh

And for inductive setting, the command is similar to the transductive setting (just replace the dataset with inductive dataset), the code will automatically handle the differences.

knn-kg's Issues

Some questions about the reasoning output

Dear professor,
Thank you for reading my message.

I have read your paper, which name is Reasoning Through Memorization: Nearest Neighbor Knowledge Graph Embeddings, and I already run your code, and got result similar with yours.

My question is about how to get the output like table 4 of the part 3.4 analysis. I do not know how to output entities with their probability like table 4 shown.

Could your provide some ideas and method about my question?

If you could provide more specific information, I would be well grateful. Looking forward to your reply.

Kind regards,
Doupi

gpu调用问题

pytorch_lightning.utilities.exceptions.MisconfigurationException: You requested GPUs: [1]
But your machine only has: []
这是什么导致的？

TypeError: forward() got an unexpected keyword argument 'en'

Traceback (most recent call last):
File "main.py", line 141, in
main()
File "main.py", line 123, in main
trainer.fit(lit_model, datamodule=data)
File "/rainbow/miniconda3/envs/knnkg/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 458, in fit
self._run(model)
File "/rainbow/miniconda3/envs/knnkg/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 756, in _run
self.dispatch()
File "/rainbow/miniconda3/envs/knnkg/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 797, in dispatch
self.accelerator.start_training(self)
File "/rainbow/miniconda3/envs/knnkg/lib/python3.7/site-packages/pytorch_lightning/accelerators/accelerator.py", line 96, in start_training
self.training_type_plugin.start_training(trainer)
File "/rainbow/miniconda3/envs/knnkg/lib/python3.7/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 144, in start_training
self._results = trainer.run_stage()
File "/rainbow/miniconda3/envs/knnkg/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 807, in run_stage
return self.run_train()
File "/rainbow/miniconda3/envs/knnkg/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 869, in run_train
self.train_loop.run_training_epoch()
File "/rainbow/miniconda3/envs/knnkg/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 490, in run_training_epoch
batch_output = self.run_training_batch(batch, batch_idx, dataloader_idx)
File "/rainbow/miniconda3/envs/knnkg/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 731, in run_training_batch
self.optimizer_step(optimizer, opt_idx, batch_idx, train_step_and_backward_closure)
File "/rainbow/miniconda3/envs/knnkg/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 432, in optimizer_step
using_lbfgs=is_lbfgs,
File "/rainbow/miniconda3/envs/knnkg/lib/python3.7/site-packages/pytorch_lightning/core/lightning.py", line 1403, in optimizer_step
optimizer.step(closure=optimizer_closure)
File "/rainbow/miniconda3/envs/knnkg/lib/python3.7/site-packages/pytorch_lightning/core/optimizer.py", line 214, in step
self.__optimizer_step(*args, closure=closure, profiler_name=profiler_name, **kwargs)
File "/rainbow/miniconda3/envs/knnkg/lib/python3.7/site-packages/pytorch_lightning/core/optimizer.py", line 134, in __optimizer_step
trainer.accelerator.optimizer_step(optimizer, self._optimizer_idx, lambda_closure=closure, **kwargs)
File "/rainbow/miniconda3/envs/knnkg/lib/python3.7/site-packages/pytorch_lightning/accelerators/accelerator.py", line 329, in optimizer_step
self.run_optimizer_step(optimizer, opt_idx, lambda_closure, **kwargs)
File "/rainbow/miniconda3/envs/knnkg/lib/python3.7/site-packages/pytorch_lightning/accelerators/accelerator.py", line 336, in run_optimizer_step
self.training_type_plugin.optimizer_step(optimizer, lambda_closure=lambda_closure, **kwargs)
File "/rainbow/miniconda3/envs/knnkg/lib/python3.7/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 193, in optimizer_step
optimizer.step(closure=lambda_closure, **kwargs)
File "/rainbow/miniconda3/envs/knnkg/lib/python3.7/site-packages/torch/optim/lr_scheduler.py", line 65, in wrapper
return wrapped(*args, **kwargs)
File "/rainbow/miniconda3/envs/knnkg/lib/python3.7/site-packages/torch/optim/optimizer.py", line 89, in wrapper
return func(*args, **kwargs)
File "/rainbow/miniconda3/envs/knnkg/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/rainbow/miniconda3/envs/knnkg/lib/python3.7/site-packages/torch/optim/adamw.py", line 65, in step
loss = closure()
File "/rainbow/miniconda3/envs/knnkg/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 726, in train_step_and_backward_closure
split_batch, batch_idx, opt_idx, optimizer, self.trainer.hiddens
File "/rainbow/miniconda3/envs/knnkg/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 814, in training_step_and_backward
result = self.training_step(split_batch, batch_idx, opt_idx, hiddens)
File "/rainbow/miniconda3/envs/knnkg/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 280, in training_step
training_step_output = self.trainer.accelerator.training_step(args)
File "/rainbow/miniconda3/envs/knnkg/lib/python3.7/site-packages/pytorch_lightning/accelerators/accelerator.py", line 204, in training_step
return self.training_type_plugin.training_step(*args)
File "/rainbow/miniconda3/envs/knnkg/lib/python3.7/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 155, in training_step
return self.lightning_module.training_step(*args, **kwargs)
File "/rainbow/weiguoying/code/KNN-KG/lit_models/transformer.py", line 74, in training_step
logits = self.model(**batch, return_dict=True).logits
File "/rainbow/miniconda3/envs/knnkg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
TypeError: forward() got an unexpected keyword argument 'en'

一个关于features的问题

您好，最近在阅读您的论文，运行代码时发现在进行Entity Embedding Initialization时，Epoch 0 进行到50%的时候，出现KeyError: 'label'，查看了代码后发现是data_module.py的DataCollatorForSeq2Seq类的__init__方法中features处理问题，当lables不存在的时候会产生错误，我尝试了多种方法，依然无法运行，想询问您如何去解决这个问题

No such file or directory: 'dataset/FB15k-237/relation2text.txt'

您好，可以麻烦您补充一下文件吗

How to get the description file of entities

Could you please tell me how to obtain the description file of entities? It seems that the download FB15k-237 zip file does not have this description file.

Can you share the plot code ?

Can you share the plot code ?

Would you mind open-sourcing the code drawn by Figure 6?

Would you mind open-sourcing the code drawn by Figure 6?

No such file or directory: 'dataset/FB15k-237/train.tsv'

您好，又缺文件了
我感觉可能test的也没有
还有WN18RR可能文件也不全
谢谢

第三阶段如何跑通

你好，我在复现您KNN代码中遇到一些问题，麻烦您看一下可以吗。就是第三阶段跑的是training里面version几 ，我这里有好几个version 而且每个version下还有多个文件