Confusion about the experimental setup about re-gcn

[Liyyy2122]

以下是我在YAGO上进行实验设置和实验结果
训练阶段设置：
python main.py -d YAGO --train-history-len 1 --test-history-len 1 --dilate-len 1 --lr 0.001 --n-layers 1 --evaluate-every 1 --gpu=0 --n-hidden 200 --self-loop --decoder convtranse --encoder uvrgcn --layer-norm --weight 0.5 --entity-prediction --relation-prediction --angle 10 --discount 1 --task-weight 0.7 --gpu 0

测试阶段设置：
python main.py -d YAGO --train-history-len 1 --test-history-len 1 --dilate-len 1 --lr 0.001 --n-layers 1 --evaluate-every 1 --gpu=0 --n-hidden 200 --self-loop --decoder convtranse --encoder uvrgcn --layer-norm --weight 0.5 --entity-prediction --relation-prediction --angle 10 --discount 1 --task-weight 0.7 --gpu 0 --test

测试阶段结果：
Using backend: pytorch
Namespace(add_rel_word=False, add_static_graph=False, aggregation='none', angle=10, batch_size=1, dataset='YAGO', decoder='convtranse', dilate_len=1, discount=1.0, dropout=0.2, encoder='uvrgcn', entity_prediction=True, evaluate_every=1, feat_dropout=0.2, gpu=0, grad_norm=1.0, grid_search=False, hidden_dropout=0.2, input_dropout=0.2, layer_norm=True, lr=0.001, multi_step=False, n_bases=100, n_basis=100, n_epochs=500, n_hidden=200, n_layers=1, num_k=500, opn='sub', relation_evaluation=False, relation_prediction=True, run_analysis=False, run_statistic=False, self_loop=True, skip_connect=False, split_by_relation=False, task_weight=0.7, test=True, test_history_len=1, topk=10, train_history_len=1, tune='n_hidden,n_layers,dropout,n_bases', weight=0.5)
loading graph data
../data/YAGO
Sanity Check: entities: 10623
Sanity Check: relations: 10
Sanity Check: edges: 161540
Sanity Check: ave node num : 1057.247191, ave rel num : 18.662921, snapshots num: 0178, max edges num: 3694, min edges num: 0198, max union rate: 1.0000, min union rate: 1.0000
Sanity Check: ave node num : 3656.400000, ave rel num : 18.800000, snapshots num: 0005, max edges num: 4024, min edges num: 3740, max union rate: 1.0000, min union rate: 1.0000
Sanity Check: ave node num : 3095.666667, ave rel num : 14.000000, snapshots num: 0006, max edges num: 4072, min edges num: 0001, max union rate: 1.0000, min union rate: 1.0000
Sanity Check: ave node num : 3095.666667, ave rel num : 14.000000, snapshots num: 0006, max edges num: 4072, min edges num: 0001, max union rate: 1.0000, min union rate: 1.0000
Sanity Check: ave node num : 3095.666667, ave rel num : 14.000000, snapshots num: 0006, max edges num: 4072, min edges num: 0001, max union rate: 1.0000, min union rate: 1.0000
Sanity Check: ave node num : 3656.400000, ave rel num : 18.800000, snapshots num: 0005, max edges num: 4024, min edges num: 3740, max union rate: 1.0000, min union rate: 1.0000
Sanity Check: ave node num : 3656.400000, ave rel num : 18.800000, snapshots num: 0005, max edges num: 4024, min edges num: 3740, max union rate: 1.0000, min union rate: 1.0000
Sanity Check: stat name : ../models/YAGO-uvrgcn-convtranse-ly1-dilate1-his1-weight:0.5-discount:1.0-angle:10-dp0.2|0.2|0.2|0.2-gpu0
Sanity Check: Is cuda available ? True
use layer :uvrgcn
activate function: <function rrelu at 0x7efc4a2329e0>
Load Model name: ../models/YAGO-uvrgcn-convtranse-ly1-dilate1-his1-weight:0.5-discount:1.0-angle:10-dp0.2|0.2|0.2|0.2-gpu0. Using best epoch : 14

----------start testing----------

0% 0/6 [00:00<?, ?it/s]/usr/local/lib/python3.7/dist-packages/dgl/base.py:45: DGLWarning: Recommend creating graphs by dgl.graph(data) instead of dgl.DGLGraph(data).
return warnings.warn(message, category=category, stacklevel=1)
../rgcn/utils.py:110: UserWarning: This overload of nonzero is deprecated:
nonzero(Tensor input, *, Tensor out)
Consider using one of the following signatures instead:
nonzero(Tensor input, *, bool as_tuple) (Triggered internally at /pytorch/torch/csrc/utils/python_arg_parser.cpp:766.)
in_deg[torch.nonzero(in_deg == 0).view(-1)] = 1
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py:1625: UserWarning: nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.
warnings.warn("nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.")
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py:1614: UserWarning: nn.functional.tanh is deprecated. Use torch.tanh instead.
warnings.warn("nn.functional.tanh is deprecated. Use torch.tanh instead.")
100% 6/6 [00:05<00:00, 1.11it/s]
MRR (raw_ent): 0.629544
Hits (raw_ent) @ 1: 0.519200
Hits (raw_ent) @ 3: 0.708953
Hits (raw_ent) @ 10: 0.819734
MRR (filter_ent): 0.819819
Hits (filter_ent) @ 1: 0.785329
Hits (filter_ent) @ 3: 0.839484
Hits (filter_ent) @ 10: 0.882078
MRR (raw_rel): 0.937934
Hits (raw_rel) @ 1: 0.882228
Hits (raw_rel) @ 3: 0.991811
Hits (raw_rel) @ 10: 0.998327
MRR (filter_rel): 0.987296
Hits (filter_rel) @ 1: 0.980575
Hits (filter_rel) @ 3: 0.992610
Hits (filter_rel) @ 10: 0.998452

[Lee-zix]

It is very convenient for you to get the results of RE-GCN without ground truth using the command "--multi-step". Sorry for ignoring it in the readme. I have updated the readme and added the commands for getting results without ground truth (under multi-step).

[Liyyy2122]

Thank you for your reply. Do you mean it only needs to train once to test with and without ground truth? If so, I used multi-step in the training phase earlier, which may be the reason why I can't get the same experimental results in the paper.

[Lee-zix]

Yes, it only needs to train once.

[Liyyy2122]

Thank you for your patient reply. According to what you said, I can’t get the experimental results of the paper on YAGO, WIKI and ICEWS05-15 without ground truth. Can you give me some details on these datasets, such as the value of topK...
按照您所说的，我进行了一系列实验，但是我仍然没法得到论文中在YAGO，WIKI和ICEWS05-15三个数据集上的实验结果（结果相差较大，例如我训练出的模型在带有ground truth的结果上已经符合您论文中列出的结果，但是我在不带ground truth的结果上MRR只有52.0-53.0，而您在论文中YAGO 没有ground truth的MRR是58.27）。此外，YAGO的关系预测结果也与我进行的实验结果相差较大。您能否给出一些得到最终结果的例如multi-step中topk值的设置细节，这将对我用您的方法与我的方法进行比较非常有帮助。

[Lee-zix]

To get the optimal result reported in the paper, change the hyperparameters and other experiment set up according to section 5.1.4 in the paper (https://arxiv.org/abs/2104.10353). Could you give your parameter settings or the running command for YAGO??

[Liyyy2122]

没有GT的结果在topk设置为0时已解决，但是关系预测的结果与论文差距较大。

[lihuiliullh]

what are raw, filter, and RE-GCN w. GT