mtang724 / nwr-gae

NWR-GAE

An implementation of ICLR 2022 Paper: [Graph Auto-Encoder via Neighborhood Wasserstein Reconstruction] (https://openreview.net/forum?id=ATUh28lnSuW)

Required Packages

Before to execute NWR-GAE, it is necessary to install the following packages using requirments.txt:

pip install -r requirements.txt

Overall Structure

The repository is organised as follows:

• data/ contains the necessary python files for generating synthetic data;
• datasets/ contains the necessary dataset files for real-world datasets;
• edgelists/ contains the necessary dataset files for real-world datasets in edgelist format;
• src/ contains the implementation of the NW-GAE pipeline (model.py) and our training file (train.py);
• src/layers contains the network layers, such as MLP, PairNorm (layers.py).
• src/utils contains the necessary processing subroutines (utils.py).

Basic Usage

Support Datasets

Proximity: cora, citeseer, pubmed

Structure: cornell, texas, wisconsin

Mixed: chameleon, squirrel, film (actor)

Synthetic: generated from https://github.com/snap-stanford/graphwave

Support Parameters

--dataset, supported datasets above, default: texas

--lr, learning rate for neighborhood reconstructor, default: 5e-5

--epoch_num, training epoch size, default: 100

--lambda_loss1, balance weights for degree and neighborhood information decoder, default: 1e-2

--lambda_loss2, balance weights for feature decoder, default: 1e-2

--sample_size, size of neighborhood down sampling, default: 5

--dimension, dimension of final output embeddings, default: 1500

Example

cd src
python train.py --dataset texas # real-world datasets
python train.py --dataset_type synthetic # Synthetic datasets

, the default setting can run most of the state-of-art results (especially on structure-oriented/mixed datasets, i.e. cornell, texas, wisconsin).

nwr-gae's Issues

How to use the model for Graph-Level Classification/Clustering.

So far, all that I've found the method for Graph Clustering is actually for node clustering or not a fully unsupervised learning method. It means that they eventually need their label to train at the downstream task.

For example, it is;

• pretraining some datasets with the unlabeled using the contrastive learning method or others. (No output channel or number of classes to be clustered is required)
• then finetuning the dataset with its labels using the supervised learning method.

I'm finding a fully unsupervised learning method for Graph-Level Clustering/Classification. Or, I'd like to know how to employ your model in a fully unsupervised learning method to classify lots of graph data that cannot be labeled by human efforts.

Question 1. Is your model only for Node-Level Classification?
Question 2. If not, could you show me a little direction to use the embedding from your model for the Graph-Level Classification/Clustering without its labels such as KMeans at the downstream task?

Could you please list the hyper-parameters for Squirrel & Chameleon?

Thanks for your excellent work, it inspires me a lot.
However , when I use the default hyper-parameters to run the dataset Chameleon & Squirrel,
I can not reproduce the result shown in the original paper, which is above 70% and 60% respectively.
So could you please list the hyper-parameters chosen for Squirrel & Chameleon?
Thanks again.

A quick question about choosing decoder

Hi, your paper is very interesting. Could I ask one question related to decoding degree and initial feature? I think it is not inituitive to choose these two approaches, could you please talk more about motivation? Moreover, why you choose to decode the initial feature rather than the original input x_v? I think the later choice is more common used in auto-encoder area. Thanks a lot.

Code logic problem

###The inner for loop is make no sense, and the out for loop is only excute once.

def reconstruction_neighbors(self, FNN_generator, neighbor_indexes, neighbor_dict, from_layer, to_layer, device):
        '''
         Reconstruction Neighbors
         INPUT:
         -----------------------
         FNN_generator    :    FNN decoder
         neighbor_indexes     :   new neighbor indexes after hungarian matching
         neighbor_dict    :    specific neighbors a node have
         from_layer     :    from which layer K
         to_layer     :   decode to which layer K-1
         device    :    CPU or GPU
         OUTPUT:
         -----------------------
         loss   :   reconstruction loss
         new index    :   new indexes after hungarian matching
        '''
        local_index_loss = 0
        sampled_embeddings_list, mark_len_list = self.sample_neighbors(neighbor_indexes, neighbor_dict, to_layer)
        for i, neighbor_embeddings1 in enumerate(sampled_embeddings_list):
            # Generating h^k_v, reparameterization trick
            index = neighbor_indexes[i]
            mask_len1 = mark_len_list[i]
            mean = from_layer[index].repeat(self.sample_size, 1)
            mean = self.mlp_mean(mean)
            sigma = from_layer[index].repeat(self.sample_size, 1)
            sigma = self.mlp_sigma(sigma)
            std_z = self.m.sample().to(device)
            var = mean + sigma.exp() * std_z
            nhij = FNN_generator(var, device)
            generated_neighbors = nhij
            # Caculate 2-Wasserstein distance
            sum_neighbor_norm = 0
            for indexi, generated_neighbor in enumerate(generated_neighbors):
                sum_neighbor_norm += torch.norm(generated_neighbor) / math.sqrt(self.out_dim)
            generated_neighbors = torch.unsqueeze(generated_neighbors, dim=0).to(device)
            target_neighbors = torch.unsqueeze(torch.FloatTensor(neighbor_embeddings1), dim=0).to(device)
            hun_loss, new_index = hungarian_loss(generated_neighbors, target_neighbors, mask_len1, self.pool)
            local_index_loss += hun_loss
            return local_index_loss, new_index

Please give some explanation about these two problems?

Thx a lot~

How to use for graph-level.

Is your model can be used for graph-level?
If yes, how can it be used? Average for all node embeddings?

Issues with model.py?

Hello,

I have some questions regarding your model implementation from src/model.py. Starting off with the forward encoder:

    def forward_encoder(self, g, h):
        # K-layer Encoder
        # Apply graph convolution and activation, pair-norm to avoid trivial solution
        h0 = h
        l1 = self.graphconv1(g, h0)
        l1_norm = torch.relu(self.norm(l1))
        l2 = self.graphconv2(g, l1_norm)
        l2_norm = torch.relu(self.norm(l2))
        l3 = self.graphconv3(g, l2)
        l3_norm = torch.relu(l3)
        l4 = self.graphconv4(g, l1_norm) # 5 layers
        return l4, l3_norm, l2_norm, l1_norm, h0

This code seems to implement the forward encoding with fixed k=4. However I do not understand why l4 is not based on l3_norm but on l1_norm. Why does l3_norm not call self.norm, but just ReLU? Why is l3 calculated using l2, but not l2_norm?

Next, the function returns the tuple l4, l3_norm, l2_norm, l1_norm, h0, but it is assigned in forward() as:

gij, l2, l3, l1, h0 = self.forward_encoder(g, h)

Are l3 and l2 accidentally swapped?

Finally, self.layer1_generator is initialized but appears to be never used. Isn't the model lacking the neighborhood decoder for the first convolution then?

Thank you for considering my questions,
Kind regards,
Maximilian Schier