dianbowork / spn4re Goto Github PK

我按论文中写的设置，并不能得到论文中的结果……请问是存在什么问题吗？
webNLG_partial

Best result on test set is 0.930512 achieving at epoch 8
python -m main --bert_directory /data/home/wuyuming/wxl/pretrained_models/bert-base-cased \
--batch_size 4 \
--num_generated_triples 10 \
--na_rel_coef 0.25 \
--max_grad_norm 20 \
--max_epoch 100 \
--encoder_lr 0.00002 \
--decoder_lr 0.00005 \
--num_decoder_layers 3 \
--max_span_length 10 \
--weight_decay 0.000001 \
--lr_decay 0.02 \
--refresh True \
--visible_gpu 1

webNLG_exact:
同样设置下：Best result on test set is 0.837710 achieving at epoch 98.
nyt_exact:在epoch7达到 0.8536778597111341后，recall迅速下降，f1变为0.1……
nyt_partial:在epoch8达到0.8662156396659961后，recall迅速下降，f1变为0.1……

ython -m main --bert_directory /data/home/wuyuming/wxl/pretrained_models/bert-base-cased \
--batch_size 4 \
--num_generated_triples 10 \
--na_rel_coef 0.25 \
--max_grad_norm 20 \
--max_epoch 100 \
--encoder_lr 0.00002 \
--decoder_lr 0.00005 \
--num_decoder_layers 3 \
--max_span_length 10 \
--weight_decay 0.000001 \
--lr_decay 0.02 \
--refresh True \
--visible_gpu 0 \
--train_file ./data/NYT/casrel_data/new_train.json \
--valid_file ./data/NYT/casrel_data/new_valid.json \
--test_file ./data/NYT/casrel_data/new_test.json

Hi, thank you for this fascinating research. I want to ask how to predict set triplets from text or sentences. Could you clarify the input and output required to predict a single text with this model? and could you provide any code for predicting from a text? Thank you so much!

这个错误如何解决？

您好，看了您的论文，很喜欢您的工作。不过命令行里似乎写成了num_generated_triplets，而main函数似乎少了个t导致该参数始终是10。

特别是几个超参的取值应该怎么设，谢谢~

我使用您readme里的WebNLG partial match启动命令，得到结果是Best result on test set is 0.928166 achieving at epoch 64，这和论文中的93.4有一点距离，请问93.4是不是您得到的最好结果，而不是平均结果呢？
另外在metric.py中似乎都是exact matching，因为他要求了实体得头尾都相同，如果我把代码改成partial matching 是否可能进一步达到93.4呢

您好！
请问您在使用其他GPU的时候是直接指定--visible_gpu这个参数的吗？我指定这个参数之后模型并没有在指定的GPU上运行，请问您有遇到这个问题吗？您是怎么解决的？谢谢。

想问下，针对没有关系和实体的样本，怎么加入到训练过程中。

使用当前模型加载会出现miss position id embedding这样的错误

请问这个错误应该怎么解决啊？
Traceback (most recent call last):
File "/root/miniconda3/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/root/miniconda3/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/root/autodl-tmp/SPN4RE/Nr_Partial_ch_SPN4RE-main/main.py", line 97, in
data = build_data(args)
File "/root/autodl-tmp/SPN4RE/Nr_Partial_ch_SPN4RE-main/utils/data.py", line 47, in build_data
data.generate_instance(args, data_process)
File "/root/autodl-tmp/SPN4RE/Nr_Partial_ch_SPN4RE-main/utils/data.py", line 31, in generate_instance
self.train_loader = data_process(args.train_file, self.relational_alphabet, tokenizer)
File "/root/autodl-tmp/SPN4RE/Nr_Partial_ch_SPN4RE-main/utils/functions.py", line 77, in data_process
tail_start_index, tail_end_index = list_index(tail_token, token_sent)
File "/root/autodl-tmp/SPN4RE/Nr_Partial_ch_SPN4RE-main/utils/functions.py", line 16, in list_index
return index[0], index[1]
UnboundLocalError: local variable 'index' referenced before assignment

Hi,

Thanks for your paper, it's really impressive. I wonder if it's possible to use SP4RE model using huggingface transformers.
If possible, could you please provide an inference example on raw example/data using Jupyter notebook (link)?

Thanks in advance. Best wishes.

这里load_data文件中，查找实体index，但是如果实体是bert词典没有的词，对应的它就是UNK，list_index方法使得它的index成为句子中最后一个UNK的index,这样的index不是错的吗？

请问，训练集数据大约有17万多条，epoch要设置成多少才可以充分发挥模型多性能，batch_size又是多少合适呢？

如题

I am trying to apply this model to some other datasets. I found that at times the model gives no predictions.
I found that it might be happening due to setting the relation logit output shape to num_classes+1
self.decoder2class = nn.Linear(config.hidden_size, num_classes + 1)

and then not creating triples when relation predicted was equal to num_classes
def generate_strategy(pred_rel, pred_head, pred_tail, num_classes, _Pred_Triple):
if pred_rel.pred_rel != num_classes:

Can somebody help me understand why this is done this way?

I directly run the script following README, but obtained abnormal results for NYT Exact Match setting, with the F1 = 0.667201.

My command is
python -m main --bert_directory ../pretrained_model/bert_base_uncased_huggingface/ --num_generated_triplets 15 --max_grad_norm 1 --na_rel_coef 0.5 --max_epoch 100 --max_span_length 10

Here is some excerpted logs which might be attributed to:

• My config

Unparsed args: ['--num_generated_triplets', '15']
dataset_name : NYT-exact
train_file : ./data/NYT/exact_data/train.json
valid_file : ./data/NYT/exact_data/valid.json
test_file : ./data/NYT/exact_data/test.json
generated_data_directory : ./data/generated_data/
generated_param_directory : ./data/generated_data/model_param/
bert_directory : ../pretrained_model/bert_base_uncased_huggingface/
partial : False
model_name : Set-Prediction-Networks
num_generated_triples : 10
num_decoder_layers : 3
matcher : avg
na_rel_coef : 0.5
rel_loss_weight : 1
head_ent_loss_weight : 2
tail_ent_loss_weight : 2
fix_bert_embeddings : True
batch_size : 8
max_epoch : 100
gradient_accumulation_steps : 1
decoder_lr : 2e-05
encoder_lr : 1e-05
lr_decay : 0.01
weight_decay : 1e-05
max_grad_norm : 1.0
optimizer : AdamW
n_best_size : 100
max_span_length : 10
refresh : False
use_gpu : True
visible_gpu : 1
random_seed : 1
DATA SUMMARY START:
Relation Alphabet Size: 24
Train Instance Number: 56196
Valid Instance Number: 5000
Test Instance Number: 5000
DATA SUMMARY END.
Data setting is saved to file: ./data/generated_data/NYT-exact_Set-Prediction-Networks_data.pickle

• And the results:

=== Epoch 99 train ===
Instance: 800; loss: 0.0072
Instance: 1600; loss: 0.0061
Instance: 2400; loss: 0.0053
Instance: 3200; loss: 0.0054
Instance: 4000; loss: 0.0055
Instance: 4800; loss: 0.0049
Instance: 5600; loss: 0.0045
Instance: 6400; loss: 0.0046
Instance: 7200; loss: 0.0042
Instance: 8000; loss: 0.0041
Instance: 8800; loss: 0.0044
Instance: 9600; loss: 0.0043
Instance: 10400; loss: 0.0046
Instance: 11200; loss: 0.0046
Instance: 12000; loss: 0.0046
Instance: 12800; loss: 0.0043
Instance: 13600; loss: 0.0042
Instance: 14400; loss: 0.0040
Instance: 15200; loss: 0.0040
Instance: 16000; loss: 0.0039
Instance: 16800; loss: 0.0038
Instance: 17600; loss: 0.0037
Instance: 18400; loss: 0.0036
Instance: 19200; loss: 0.0035
Instance: 20000; loss: 0.0035
Instance: 20800; loss: 0.0035
Instance: 21600; loss: 0.0034
Instance: 22400; loss: 0.0034
Instance: 23200; loss: 0.0033
Instance: 24000; loss: 0.0034
Instance: 24800; loss: 0.0033
Instance: 25600; loss: 0.0033
Instance: 26400; loss: 0.0033
Instance: 27200; loss: 0.0034
Instance: 28000; loss: 0.0034
Instance: 28800; loss: 0.0034
Instance: 29600; loss: 0.0033
Instance: 30400; loss: 0.0035
Instance: 31200; loss: 0.0038
Instance: 32000; loss: 0.0038
Instance: 32800; loss: 0.0038
Instance: 33600; loss: 0.0037
Instance: 34400; loss: 0.0039
Instance: 35200; loss: 0.0039
Instance: 36000; loss: 0.0039
Instance: 36800; loss: 0.0038
Instance: 37600; loss: 0.0038
Instance: 38400; loss: 0.0039
Instance: 39200; loss: 0.0039
Instance: 40000; loss: 0.0039
Instance: 40800; loss: 0.0039
Instance: 41600; loss: 0.0038
Instance: 42400; loss: 0.0038
Instance: 43200; loss: 0.0038
Instance: 44000; loss: 0.0039
Instance: 44800; loss: 0.0039
Instance: 45600; loss: 0.0041
Instance: 46400; loss: 0.0041
Instance: 47200; loss: 0.0040
Instance: 48000; loss: 0.0040
Instance: 48800; loss: 0.0039
Instance: 49600; loss: 0.0039
Instance: 50400; loss: 0.0039
Instance: 51200; loss: 0.0039
Instance: 52000; loss: 0.0039
Instance: 52800; loss: 0.0039
Instance: 53600; loss: 0.0039
Instance: 54400; loss: 0.0039
Instance: 55200; loss: 0.0039
Instance: 56000; loss: 0.0039
=== Epoch 99 Test ===
------Num of Gold Triplet is 1------
gold_num = 3240 pred_num = 3732 right_num = 2409 relation_right_num = 2826 entity_right_num = 2864
precision = 0.6454983922829582 recall = 0.7435185185185185 f1_value = 0.6910499139414802
rel_precision = 0.7572347266881029 rel_recall = 0.8722222222222222 rel_f1_value = 0.810671256454389
ent_precision = 0.767416934619507 ent_recall = 0.8839506172839506 ent_f1_value = 0.8215720022948939
------Num of Gold Triplet is 2------
gold_num = 2094 pred_num = 1675 right_num = 1309 relation_right_num = 1425 entity_right_num = 1476
precision = 0.7814925373134328 recall = 0.625119388729704 f1_value = 0.6946139559564872
rel_precision = 0.8507462686567164 rel_recall = 0.6805157593123209 rel_f1_value = 0.7561687450252057
ent_precision = 0.8811940298507462 ent_recall = 0.7048710601719198 ent_f1_value = 0.7832316264261078
------Num of Gold Triplet is 3------
gold_num = 942 pred_num = 590 right_num = 501 relation_right_num = 537 entity_right_num = 542
precision = 0.8491525423728814 recall = 0.5318471337579618 f1_value = 0.6540469973890339
rel_precision = 0.9101694915254237 rel_recall = 0.5700636942675159 rel_f1_value = 0.7010443864229765
ent_precision = 0.9186440677966101 ent_recall = 0.5753715498938429 ent_f1_value = 0.7075718015665795
------Num of Gold Triplet is 4------
gold_num = 1160 pred_num = 619 right_num = 530 relation_right_num = 576 entity_right_num = 559
precision = 0.8562197092084006 recall = 0.45689655172413796 f1_value = 0.59584035975267
rel_precision = 0.9305331179321487 rel_recall = 0.496551724137931 rel_f1_value = 0.6475548060708264
ent_precision = 0.9030694668820679 ent_recall = 0.4818965517241379 ent_f1_value = 0.628442945474986
------Num of Gold Triplet is greater than or equal to 5------
gold_num = 684 pred_num = 266 right_num = 241 relation_right_num = 254 entity_right_num = 251
precision = 0.9060150375939849 recall = 0.35233918128654973 f1_value = 0.5073684210526316
rel_precision = 0.9548872180451128 rel_recall = 0.3713450292397661 rel_f1_value = 0.5347368421052632
ent_precision = 0.943609022556391 ent_recall = 0.3669590643274854 ent_f1_value = 0.5284210526315789
------Normal Triplets------
gold_num = 2028 pred_num = 2259 right_num = 1414 relation_right_num = 1725 entity_right_num = 1604
precision = 0.6259406817175741 recall = 0.6972386587771203 f1_value = 0.6596687660368556
rel_precision = 0.7636122177954847 rel_recall = 0.8505917159763313 rel_f1_value = 0.8047585724282714
ent_precision = 0.7100486941124391 ent_recall = 0.7909270216962525 ent_f1_value = 0.748308840681129
------Multiply label Triplets------
gold_num = 4079 pred_num = 2511 right_num = 2052 relation_right_num = 2202 entity_right_num = 2255
precision = 0.8172043010752689 recall = 0.5030644765873988 f1_value = 0.6227617602427922
rel_precision = 0.8769414575866189 rel_recall = 0.5398381956361853 rel_f1_value = 0.6682852807283762
ent_precision = 0.8980485862206292 ent_recall = 0.5528315763667565 ent_f1_value = 0.684370257966616
------Overlapping Triplets------
gold_num = 5530 pred_num = 4243 right_num = 3319 relation_right_num = 3608 entity_right_num = 3791
precision = 0.7822295545604525 recall = 0.6001808318264015 f1_value = 0.6792182543742965
rel_precision = 0.8503417393353759 rel_recall = 0.6524412296564195 rel_f1_value = 0.7383607899314436
ent_precision = 0.8934716002828188 ent_recall = 0.6855334538878842 ent_f1_value = 0.7758109076025785
gold_num = 8120 pred_num = 6882 right_num = 4990 relation_right_num = 5618 entity_right_num = 5692
precision = 0.7250799186283057 recall = 0.6145320197044335 f1_value = 0.6652446340487934
rel_precision = 0.8163324614937518 rel_recall = 0.691871921182266 rel_f1_value = 0.7489668044260765
ent_precision = 0.8270851496657948 ent_recall = 0.7009852216748769 ent_f1_value = 0.7588321557125718
Best result on test set is 0.667201 achieving at epoch 90.
/pytorch/torch/csrc/utils/python_arg_parser.cpp:756: UserWarning: This overload of add_ is deprecated:
add_(Number alpha, Tensor other)
Consider using one of the following signatures instead:
add_(Tensor other, *, Number alpha)

Right now I'm guessing two possible reasons:
1.I see that the README has just updated recently, so I have not set --num_generated_triplets correctly, but I'm not sure this would lead to dramaticly performance drop from expected 90 to 66.
2.The script and model default to set fix_bert_embeddings=True, as can be refered to here, which is not the usual case where people finetune BERT on downstream tasks instead of freeze it.

Great thanks for your attention and help!

IndexError: index out of range in self

why the paper does not match the code

举个例子：
pred: [(0, 1, 2, 3, 4), (2, 1, 2,3,4)]
gold: [(0, 1, 2, 3, 4)]

按照代码里面的结果：

rel_recall = 1/1 = 100%
ent_recall = 2/1 = 200% (因为ent_num = 2)

这样会导致，当gold有很多组（假设N）， 而pred只预测对一组实体，但是这组实体对应了不同关系，所以重复了多次（假设M），最后也会被认为ent_num+=1，真实召回应该是1/N，按照代码计算可能是 M/N

您好，感谢分享。有一个问题，论文table 4的实验结果：

我根据您提供源代码和命令行，效果只到0.928841, 请问，可以分享更详细的参数，或者训练过程用了哪些技巧么 ？感谢您的回复！
python -m main --bert_directory BERT_DIR --batch_size 4 --num_generated_triplets 10 --na_rel_coef 0.25 --max_grad_norm 20 --max_epoch 100 --encoder_lr 0.00002 --decoder_lr 0.00005 --num_decoder_layers 4 --max_span_length 10 --weight_decay 0.000001 --lr_decay 0.02

=== Epoch 73 Test ===
precision = 0.930599, recall = 0.927090, f1_value = 0.928841
rel_precision = 0.955205, rel_recall = 0.951603, rel_f1_value = 0.953401
ent_precision = 0.950158, ent_recall = 0.946574, ent_f1_value = 0.948363
Achieving Best Result on Test Set

谢谢您的分享。
README部分信息太少了，能否增加一下说明信息。
比如，模型的表现，与其他方法的对比，是否支持重叠实体的关系抽取等，谢谢！

您好，在看了您的论文之后我有一点疑惑，模型是如何让decoder端的m个learnable embedding是如何学习到不同的三元组信息的？

您好，想问一下models/set_decoder.py中第124行一直报错，缺少BertAttention的attention_mask，应该如何修正

首先感谢您的论文和代码，然后想问一下，二部匹配之后使用的不还是交叉熵损失嘛，这一点没看懂，还有后边的对比实验里用的交叉熵又是啥？

请问如何生成模型推断结果文件？以及何如加载已经训练好的模型做推断？
Readme貌似都没有提及，请问能解答一下吗，谢谢！