Cannot reproduce the results even with the same random seed about graphless-neural-networks

[ShichangZh]

Dear meettyj,

Thank you for your interest in our work. What you pointed out is indeed a concern but I don't think it is a bug in our code and it doesn't affect the conclusion of our paper.

We have tried our best to guarantee reproducibility by fixing random seeds. However, there are certain parts of the code that are random but beyond our control. Our code is based on the DGL platform, and the SAGE implementation is adapted from the DGL official examples, which unfortunately involves sampling steps that produce different results even when we fix all the seeds (you can read more here https://github.com/dmlc/dgl/blob/master/examples/pytorch/ogb/ogbn-products/graphsage/main.py). Other models, like GCN, behave better and provide consistent results over runs when the seeds are fixed. However, we have to choose SAGE as the base GNN for our study because the other models won't scale on large graphs. Therefore, we repeat each experiment multiple times and report average performance with std to rule out this kind of concern.

I just reran the first line of code you showed on my end, i.e.
python train_teacher.py --exp_setting tran --teacher SAGE --dataset cora
Unfortunately, this gave me 0.8351 and 0.8230.

However, when I replaced SAGE with GCN, GAT, APPNP, and MLP. I got consistent results over multiple runs as below
GCN 0.8244
GAT 0.8370
APPNP 0.8412
MLP 0.5705

You are welcome to try our method with these models on your end to see if you get the same numbers, or you may include other models implemented by yourself.

Nevertheless, studying different GNN architectures is not the focus of our paper. The results you provided are already very close to ours. All of them only differ by less than 1% and the differences are within 0.5 std. This level of inconsistency shouldn't cause too much concern and doesn't affect our conclusion at all.

Please let me know whether your concern is addressed or not, and I am happy to discuss if you have further questions.

Best,
Shichang

[meettyj]

Dear Shichang,

Thank you so much for your detailed answer and the well-written code! This makes sense to me if the randomness is coming from the sampling of SAGE. I will close this issue accordingly :)