yandex-research / heterophilous-graphs

A Critical Look at the Evaluation of GNNs under Heterophily: Are We Really Making Progress?

A Critical Look at the Evaluation of GNNs under Heterophily: Are We Really Making Progress?

This is the official repository for the paper "A Critical Look at the Evaluation of GNNs under Heterophily: Are We Really Making Progress?" (ICLR 2023).

Datasets

In the paper we introduce 5 new heterophilous graph datasets: roman-empire, amazon-ratings, minesweeper, tolokers, and questions. You can find these datasets in the data folder. Note: the datasets are stored as undirected, that is, each edge is stored only once. If you load the edges into DGL or PyG, which treat all graphs as directed, do not forget to call dgl.to_bidirected or pyg.transforms.ToUndirected to double the edges.

Roman-empire is a word dependency graph based on the Roman Empire article from the English Wikipedia.

Amazon-ratings is a product co-purchasing network based on data from SNAP Datasets.

Minesweeper is a synthetic graph emulating the eponymous game.

Tolokers is a crowdsourcing platform workers network based on data provided by Toloka.

Questions is an interaction graph of users of a question-answering website based on data provided by Yandex Q.

Our datasets come from different domains and exhibit a wide range of structual properties. We provide some statistics of our datasets in the table below:

	roman-empire	amazon-ratings	minesweeper	tolokers	questions
nodes	22662	24492	10000	11758	48921
edges	32927	93050	39402	519000	153540
avg degree	2.91	7.60	7.88	88.28	6.28
global clustering	0.29	0.32	0.43	0.23	0.02
avg local clustering	0.39	0.58	0.44	0.53	0.03
diameter	6824	46	99	11	16
node features	300	300	7	10	301
classes	18	5	2	2	2
edge homophily	0.05	0.38	0.68	0.59	0.84
adjusted homophily	-0.05	0.14	0.01	0.09	0.02
label informativeness	0.11	0.04	0.00	0.01	0.00

More details on our datasets can be found in the paper.

Reproducing experiments

In our paper we show that standard GNNs augmented with skip connections and layer normalization achieve strong results on heterophilous graphs and almost always outperform specialized models.

To reproduce results of our baseline models (ResNet and standard GNNs), you need to install PyTorch and DGL (see the full list of requirements in enivronment.yml). Then you can run scripts/run_all_experiments.sh (there are a lot of experiments, so you might want to comment out some of the lines). After that you can view a table with results in notebooks/results.ipynb.

To reproduce results of specialized models, see this repository.

Citation

If you found our datasets or our code helpful, please cite our paper:

@inproceedings{platonov2023critical,
  title={A critical look at evaluation of GNNs under heterophily: Are we really making progress?},
  author={Platonov, Oleg and Kuznedelev, Denis and Diskin, Michael and Babenko, Artem and Prokhorenkova, Liudmila},
  booktitle={The Eleventh International Conference on Learning Representations},
  year={2023}
}

heterophilous-graphs's Issues

Question: JacobiConv Best Hyperparameters

Can you share the best hyperparameters of JacobiConv for Roman-empire data?

I have tried some default parameters but can only get 56.71 acc. It's much lower than the 71.14 you listed in the paper.

Thank you !

Nan in roman_empire

Hi, I find that the node features of the roman-empire dataset have a lot of nan numbers which would cause loss becoming nan in training. Currently, I just replace the nan with 0. Is there any official way to process this kind of nan feature? Thanks!

Question: Are these datasets available on https://paperswithcode.com/datasets

Hi, I just have a question: Are these datasets available on https://paperswithcode.com/datasets? I did some search on paperswithcode but it seems you haven't put your datasets on it. Are you planning to do that? Thanks.

Raw text for Roman-empire, Amazon-ratings and Questions

Hi,

Thank you for your wonderful work, I wonder whether the raw text of dataset Roman-empire, Amazon-ratings and Questions is available? I am conducting research on LLM and graph, so the raw text information will be important. Is it possible to share the raw text of these datasets publicly?

Thanks!