Hi!
I've run your example on classification of MNIST data. Now I'm randomly changing the adjacency matrix by swapping a large number of random rows and columns and even replacing the matrix of adjacency with a zero matrix of the same size, but the accuracy of classification on test data is not affected. I've also tested this on my own dataset and the result is the same. I couldn't figure out what the problem is. Does it mean that the structure of the data graph has no effect on classification? So what is the point of using the adjacency matrix as an input? I'm not sure if I can ask you, but do you have any idea?
Thanks.

Hi,

1. I have a graph where I have data on only part of the nodes(always on single a node, meaning, there are no 2 neighboring nodes with known data).
   However, I must keep the adjacency of the full graph, since otherwise the hierarchy between the nodes won't be specified.
   Does Spektral provides any suitable platform for this kind of problem?
2. When identifing for example F=2 (dimension of the node attributes=2), the first 2 columns of the training data would be linked to the first node in the graph, and so on?
3. Say that the answer to Q.1 is positive, how are the columns of the data will be attributed to the right nodes in the graph?

Thank you!

I believe that the way single and disjoint mode are handled in the documentation is potentially confusing.

For example, SortPool works in single mode but does not work in disjoint mode.

Meanwhile if we look at GraphConv, the documentation says that it works for single mode but does not mention disjoint mode. This gives the impression that maybe this layer is also incompatible with disjoint mode. However, unless I am misunderstanding something, GraphConv works fine with disjoint graphs.

Therefore, I would suggest to exclude disjoint mode only if the layer will not work properly on disjoint graphs.

Best,
Levi

Hi, Thanks for your great works, I am a beginner of the GNN, they are really useful for me.

In Kipf and Welling GCN, the adjacency matrix needs to add a self-loop, and be renormalized then. In your work, the localpooling_filter function is designed to this operation. And it is used in "node_classification_gcn.py" demo, but not in the "regression_molecules_graph_batch.py". But they both use the GraphConv function. So, why the "regression_molecules_graph_batch.py" doesn't perform the localpooling_filter? And some other demos only use normalized_adjacency function, without adding self-loop "In".

And I am trying use GCN to classify some vibration signal. The performance is really good when the localpooling_filter function do not involved in the code just like the "regression_molecules_graph_batch.py" demo, but it becomes untrainable when I use tlideA =localpooling_filter(A) to feed the network.

Looking forward for your reply~:）

Hi Daniele,

Fantastic work!

Can you add those two pooling methods please ?

https://arxiv.org/pdf/1905.11577.pdf
&
https://arxiv.org/pdf/1904.13107.pdf (I have a source code but not very functional)

thanks in advance,

Guillaume

Hi @danielegrattarola, and thanks for this wonderful piece of work. It is really helpful to people like me who are taking their first steps towards a career in research. I have a minor doubt regarding the implementation of the ChebConv layer. According to equation 5 in the paper:

Where x is the n-dimensional feature vector, where n specifies the number of vertices in the graph. So even if we assume i and j to be just 1 (a simple case of single input feature vector and single feature output vector), our output should be an n-dimensional vector as well, as stated above.

Now coming to the code, we have:

self.kernel = self.add_weight(shape=(self.K, input_dim, self.channels), ...

and

T_0 = features
output = ops.dot(T_0, self.kernel[0])

Here, I'm assuming features is an n-dimensional input feature map, just like the x above. So wouldn't the output in this case be a vector with the dimensionality of the number of channels? Am I missing something here?

On Colab (tf 2.2.0) the installation is ok but the import gives an error:

/usr/local/lib/python3.6/dist-packages/spektral/layers/base.py in ()
4 from tensorflow.keras import backend as K
5 from tensorflow.keras.layers import Layer
----> 6 from tensorflow_core.python.keras.utils import tf_utils

ModuleNotFoundError: No module named 'tensorflow_core'

Hi!
Is possibile to use an adjacency matrix of float values? Several studies uses weighted adjacency matrix (i.e. 0<=Adj<=1 instead of Adj in {0,1} values). Is spektral already ready for this kind of adjacency matrix or using float Adj matrix could lead to some run (or logical) error?

Suppose I have a bunch of points on an image that has properties like its position and color.
I want to connect every point by a line and then the model should classify which line is appropriate.

For example, if a line connects between red and green point then it should be labeled 1, otherwise 0.

Which algorithm from this repo should I use?

Hi,

I'm trying to use some of the available pooling layers with a GCN in mixed mode. But there are some errors after the definition of the model, in the shape of the computed gradient. I can reproduce the same error with the example graph_signal_classification_mnist.py as by adding a DiffPool layer :

graph_conv = GraphConv(32,
                       activation='elu',
                       kernel_regularizer=l2(l2_reg),
                       use_bias=True)([X_in, A_in])
graph_conv,A_in2,_ = DiffPool(k=128)([graph_conv,A_in])
graph_conv = GraphConv(32,
                       activation='elu',
                       kernel_regularizer=l2(l2_reg),
                       use_bias=True)([graph_conv, A_in2])
flatten = Flatten()(graph_conv)

And the following error occurs:

Traceback (most recent call last):
 File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 655, in set_shape
   unknown_shape)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Shapes must be equal rank, but are 2 and 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
 File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gradients_util.py", line 697, in _GradientsHelper
   in_grad.set_shape(t_in.get_shape())
 File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 658, in set_shape
   raise ValueError(str(e))
ValueError: Shapes must be equal rank, but are 2 and 1

During handling of the above exception, another exception occurred:

ValueError: Incompatible shapes between op input and calculated input gradient.  Forward operation: diff_pool/SparseTensorDenseAdd.  Input index: 1. Original input shape: (7180,).  Calculated input gradient shape: (7180, 784)

Can you help me with this?

BTW I also was wondering: Is there any example of code using MNIST in single mode, and then compatible with the other pooling layers as TopKPool?

Best

Hi,
I would really appreciate it if you could provide a CORA example of the ARMAConv.

Hi @danielegrattarola ,
first of all great work. Do you have graculus pooling implemented (the one from Defferard's original ChebNet paper)? I could not find it, but it might be under different name.

Hi!

I would be really grateful if you could show me an example how to use pooling in graph neural networks, for example the topKpool or the MinCutPool. I have just made a good network with spektral, but to make steps forward I would need to do some pooling.

After some experiment I found that the MinCutPool does not provide better score (it is working really bad) and the topKpool is not working for me due to rank problems.

Thank you for your help in advance!

András

Hi,

Is there a way to use batch mode in topk pool? Right now it only supports single and disjoint mode.

My requirement is:
I was trying to implement graph unet using your topkpool layer. To train, I have to do backprop on a batch size (of say 50) (each example is a graph instance with same Adjacency matrix).

Issue:
Batch mode is not supported in topkpool and looping over each example is not possible. So, I was wondering if there is a way to do it.

Same is the case with SAGPool layer too.

Any suggestions?

Cheers!

Here's the link to he notebook
https://colab.research.google.com/drive/1s2Mr7SG2UG8lVbk81r94_4koTjy3ajsF

The error is
Building wheels for collected packages: pygraphviz
Building wheel for pygraphviz (setup.py) ... error
ERROR: Failed building wheel for pygraphviz
Running setup.py clean for pygraphviz
Failed to build pygraphviz
Installing collected packages: pygraphviz, spektral
Running setup.py install for pygraphviz ... error
ERROR: Command errored out with exit status 1: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-3huozh2y/pygraphviz/setup.py'"'"'; file='"'"'/tmp/pip-install-3huozh2y/pygraphviz/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-81ncw97y/install-record.txt --single-version-externally-managed --compile Check the logs for full command output.

Thank you for providing a geometric learning package in Kera.

In our recent paper (https://arxiv.org/pdf/1902.07153.pdf), we show that a linear model (Simple Graph Convolution, SGC) that is as effective as GCN while saving a lot of training time. We have our official implementation at https://github.com/Tiiiger/SGC. I would like to ask if there will be interests of including SGC as an example in spektral. I am happy to help with the implementation.

The huge computational benefit of SGC comes from precomputing the message passing step. I'd like to discuss with you how best/cleanest to implement this through spektral. Look forward to your comments.

Hi Daniele,
I tried running the examples on node classification using GAT and I got following error:
TypeError: unsupported operand type(s) for -: 'float' and 'SparseTensor'
on layers/convolutional/gat.py:179 call * mask = -10e9 * (1.0 - A)

Do you have any idea what caused this?
Thanks!

Hey,
I have implementations of the following few graphical neural network components:

Convolutions:

• Diffusion based graph convolution (as part of my GARNN implementation)
• Graph convolutions in the sense of M. Zhang et al., 2018 (as part of my DGCNN implementation)

Pooling:

• SortPooling (Also as part my DGCNN implementation)

I would love to add these to your framework however, I am a bit lost with all the requirements that my implementations need to fulfil in order to integrate seamlessly. Is there any guide for what tests, features or properties layers need to have?

Best,
Levi

May I ask why the output of TopKpooling does have the dimension of [batch]? like "Reduced node features of shape ([batch], K, channels);"

Hi,

It seems that this is now incompatible with the latest Keras release (2.2.5) from August 22, 2019. Here are the details:

Environment

Python 3.7
spektral==0.0.12
tensorflow==1.14.0
keras==2.2.5

How to reproduce

Run a python script that imports spektral:

import spektral

Expected behaviour

Everything runs smoothly and spektral is imported correctly.

Observed behaviour

We get an import error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<path_to_virtualenv>/lib/python3.7/site-packages/spektral/__init__.py", line 5, in <module>
    from . import layers
  File "<path_to_virtualenv>/lib/python3.7/site-packages/spektral/layers/__init__.py", line 4, in <module>
    from .convolutional import *
  File "<path_to_virtualenv>/lib/python3.7/site-packages/spektral/layers/convolutional.py", line 5, in <module>
    from keras.backend import tf
ImportError: cannot import name 'tf' from 'keras.backend' (<path_to_virtualenv>/lib/python3.7/site-packages/keras/backend/__init__.py)

Suggested change

On the short run, I think it would suffice to pin the keras version in setup.py to 2.2.4 (This fixed the issue in my project for now)

Thanks,
Tudor.

I am training a Graph neural network with an auxiliary input layer. I am concatenating the layers. The model compiles perfectly. But when fitting the data into the model I am getting the following error.

ValueError: No data provided for "input_10". Need data for each key in: ['input_10', 'input_12']

I have used an example from the example folder and modified it for multi input.

X_in = Input(shape=(1375, 3))
A_in = Input(tensor=sp_matrix_to_sp_tensor(adj_mat))

Feat_input = Input(shape=(55,8))

Feat_layer = Bidirectional(LSTM(32, return_sequences=True,),name='lstm_input')(Feat_input)
Feat_layer = Dense(512,activation='relu')(Feat_layer)
Feat_layer = Flatten()(Feat_layer)

graph_conv = GraphConvSkip(64, activation='relu', kernel_regularizer=l2(l2_reg),name='graph_input')([X_in, A_in])

graph_conv = Dropout(0.5)(graph_conv)
graph_conv = ChebConv(32, activation='relu', kernel_regularizer=l2(l2_reg))([graph_conv, A_in])
graph_conv = Dropout(0.5)(graph_conv)
graph_conv = GraphConvSkip(64, activation='relu', kernel_regularizer=l2(l2_reg))([graph_conv, A_in])
graph_conv = Dropout(0.5)(graph_conv)
graph_conv = ChebConv(32, activation='relu', kernel_regularizer=l2(l2_reg))([graph_conv, A_in])

flatten = Flatten()(graph_conv)

concatenated = concatenate([flatten, Feat_layer])

fc = Dense(512, activation='relu')(concatenated)
fc = Dense(256, activation='relu')(fc)
output = Dense(n_out, activation='softmax')(fc)

model = Model(inputs={'graph_input':[X_in, A_in], 'lstm_input':Feat_input}, outputs=output)
optimizer = RMSprop(lr=learning_rate)
model.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=['acc'])
model.summary()
history = model.fit({'graph_input': [X_train], 'lstm_input': x_train_feat }, y_train, batch_size=28, epochs=250,steps_per_epoch=10)

Hi, @danielegrattarola , thanks for your work. I'm wondering that how to port it to tensorflow.keras. Because my project needs a GNN model in tf.keras api.

I have made a script below, but there is a error. Could you take your time to look at this?

Issue

1. Your project seems to be not compatible with tensorflow 1.14.0, which would error

Traceback (most recent call last):
  File "/home/noone/Documents/New_TF/spektral-master/examples/node_classification_gat.py", line 81, in <module>
    shuffle=False,  # Shuffling data means shuffling the whole graph
  File "/home/noone/anaconda3/envs/lab/lib/python3.6/site-packages/keras/engine/training.py", line 1039, in fit
    validation_steps=validation_steps)
  File "/home/noone/anaconda3/envs/lab/lib/python3.6/site-packages/keras/engine/training_arrays.py", line 199, in fit_loop
    outs = f(ins_batch)
  File "/home/noone/anaconda3/envs/lab/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2697, in __call__
    if hasattr(get_session(), '_make_callable_from_options'):
  File "/home/noone/anaconda3/envs/lab/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 186, in get_session
    _SESSION = tf.Session(config=config)
  File "/home/noone/anaconda3/envs/lab/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1570, in __init__
    super(Session, self).__init__(target, graph, config=config)
  File "/home/noone/anaconda3/envs/lab/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 693, in __init__
    self._session = tf_session.TF_NewSessionRef(self._graph._c_graph, opts)
tensorflow.python.framework.errors_impl.InternalError: cudaGetDevice() failed. Status: cudaGetErrorString symbol not found.

This error is strange, since my cuda is 10.0, cudnn is 7.6.2

2. Then I change tf to 1.11.0, your project can work. But when I adapt them to the tf.keras interface (Code is shown below), another error would occur.

/home/noone/anaconda3/envs/tf_ray/bin/python /home/noone/Documents/New_TF/ray-autoregressive/python/ray/rllib/examples/GAT_model_keras.py
Tensor("model/graph_attention_1/truediv:0", shape=(1, 15, 7), dtype=float32)
2019-08-01 17:35:17.775631: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-08-01 17:35:17.836503: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-08-01 17:35:17.836854: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1411] Found device 0 with properties: 
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:01:00.0
totalMemory: 10.91GiB freeMemory: 8.45GiB
2019-08-01 17:35:17.836865: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1490] Adding visible gpu devices: 0
2019-08-01 17:35:18.133926: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-08-01 17:35:18.133948: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977]      0 
2019-08-01 17:35:18.133953: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0:   N 
2019-08-01 17:35:18.134015: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1103] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 8159 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
Traceback (most recent call last):
  File "/home/noone/anaconda3/envs/tf_ray/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1292, in _do_call
    return fn(*args)
  File "/home/noone/anaconda3/envs/tf_ray/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1277, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/home/noone/anaconda3/envs/tf_ray/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1367, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.FailedPreconditionError: Error while reading resource variable graph_attention_1/attn_kernel_self_0 from Container: localhost. This could mean that the variable was uninitialized. Not found: Container localhost does not exist. (Could not find resource: localhost/graph_attention_1/attn_kernel_self_0)
	 [[{{node model/graph_attention_1/transpose_1/ReadVariableOp}} = ReadVariableOp[dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](graph_attention_1/attn_kernel_self_0)]]
	 [[{{node model/graph_attention_1/truediv/_3}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_655_model/graph_attention_1/truediv", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/noone/Documents/New_TF/ray-autoregressive/python/ray/rllib/examples/GAT_model_keras.py", line 334, in <module>
    print(sess.run(emb_node))
  File "/home/noone/anaconda3/envs/tf_ray/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 887, in run
    run_metadata_ptr)
  File "/home/noone/anaconda3/envs/tf_ray/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1110, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/noone/anaconda3/envs/tf_ray/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1286, in _do_run
    run_metadata)
  File "/home/noone/anaconda3/envs/tf_ray/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1308, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.FailedPreconditionError: Error while reading resource variable graph_attention_1/attn_kernel_self_0 from Container: localhost. This could mean that the variable was uninitialized. Not found: Container localhost does not exist. (Could not find resource: localhost/graph_attention_1/attn_kernel_self_0)
	 [[{{node model/graph_attention_1/transpose_1/ReadVariableOp}} = ReadVariableOp[dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](graph_attention_1/attn_kernel_self_0)]]
	 [[{{node model/graph_attention_1/truediv/_3}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_655_model/graph_attention_1/truediv", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Caused by op 'model/graph_attention_1/transpose_1/ReadVariableOp', defined at:
  File "/home/noone/Documents/New_TF/ray-autoregressive/python/ray/rllib/examples/GAT_model_keras.py", line 330, in <module>
    emb_node = model([np.expand_dims(feat, 0), np.expand_dims(adj, 0)])
  File "/home/noone/anaconda3/envs/tf_ray/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 769, in __call__
    outputs = self.call(inputs, *args, **kwargs)
  File "/home/noone/anaconda3/envs/tf_ray/lib/python3.6/site-packages/tensorflow/python/keras/engine/network.py", line 845, in call
    mask=masks)
  File "/home/noone/anaconda3/envs/tf_ray/lib/python3.6/site-packages/tensorflow/python/keras/engine/network.py", line 1052, in _run_internal_graph
    output_tensors = layer.call(computed_tensors, **kwargs)
  File "/home/noone/Documents/New_TF/ray-autoregressive/python/ray/rllib/examples/GAT_model_keras.py", line 217, in call
    attn_for_self = K.dot(features, attention_kernel[0])  # (N x 1), [a_1]^T [Wh_i]
  File "/home/noone/anaconda3/envs/tf_ray/lib/python3.6/site-packages/tensorflow/python/keras/backend.py", line 1414, in dot
    array_ops.transpose(y, perm=y_permute_dim), [y_shape[-2], -1])
  File "/home/noone/anaconda3/envs/tf_ray/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 1419, in transpose
    ret = transpose_fn(a, perm, name=name)
  File "/home/noone/anaconda3/envs/tf_ray/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 8741, in transpose
    "Transpose", x=x, perm=perm, name=name)
  File "/home/noone/anaconda3/envs/tf_ray/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 510, in _apply_op_helper
    preferred_dtype=default_dtype)
  File "/home/noone/anaconda3/envs/tf_ray/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1144, in internal_convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/home/noone/anaconda3/envs/tf_ray/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py", line 1286, in _dense_var_to_tensor
    return var._dense_var_to_tensor(dtype=dtype, name=name, as_ref=as_ref)  # pylint: disable=protected-access
  File "/home/noone/anaconda3/envs/tf_ray/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py", line 1241, in _dense_var_to_tensor
    return self.value()
  File "/home/noone/anaconda3/envs/tf_ray/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py", line 672, in value
    return self._read_variable_op()
  File "/home/noone/anaconda3/envs/tf_ray/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py", line 755, in _read_variable_op
    self._dtype)
  File "/home/noone/anaconda3/envs/tf_ray/lib/python3.6/site-packages/tensorflow/python/ops/gen_resource_variable_ops.py", line 508, in read_variable_op
    "ReadVariableOp", resource=resource, dtype=dtype, name=name)
  File "/home/noone/anaconda3/envs/tf_ray/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/home/noone/anaconda3/envs/tf_ray/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "/home/noone/anaconda3/envs/tf_ray/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3272, in create_op
    op_def=op_def)
  File "/home/noone/anaconda3/envs/tf_ray/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1768, in __init__
    self._traceback = tf_stack.extract_stack()

FailedPreconditionError (see above for traceback): Error while reading resource variable graph_attention_1/attn_kernel_self_0 from Container: localhost. This could mean that the variable was uninitialized. Not found: Container localhost does not exist. (Could not find resource: localhost/graph_attention_1/attn_kernel_self_0)
	 [[{{node model/graph_attention_1/transpose_1/ReadVariableOp}} = ReadVariableOp[dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](graph_attention_1/attn_kernel_self_0)]]
	 [[{{node model/graph_attention_1/truediv/_3}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_655_model/graph_attention_1/truediv", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Codes

I made a copy of GraphAttention in this file, and you can save this to run. This file requires o extra files from your project.

import tensorflow.keras.backend as K
import tensorflow.keras.layers as layers
from tensorflow.keras.regularizers import l2
from tensorflow.keras import activations, initializers, regularizers, constraints
import tensorflow as tf
import networkx as nx
import numpy as np


def filter_dot(fltr, features):
    """
    Performs the multiplication of a graph filter (N x N) with the node features,
    automatically dealing with single, mixed, and batch modes.
    :param fltr: the graph filter(s) (N x N in single and mixed mode,
    batch x N x N in batch mode).
    :param features: the node features (N x F in single mode, batch x N x F in
    mixed and batch mode).
    :return: the filtered features.
    """
    if len(K.int_shape(features)) == 2:
        # Single mode
        return K.dot(fltr, features)
    else:
        if len(K.int_shape(fltr)) == 3:
            # Batch mode
            return K.batch_dot(fltr, features)
        else:
            # Mixed mode
            return mixed_mode_dot(fltr, features)

def mixed_mode_dot(fltr, features):
    """
    Computes the equivalent of `tf.einsum('ij,bjk->bik', fltr, output)`, but
    works for both dense and sparse input filters.
    :param fltr: rank 2 tensor, the filter for convolution
    :param features: rank 3 tensor, the features of the input signals
    :return:
    """
    _, m_, f_ = K.int_shape(features)
    features = K.permute_dimensions(features, [1, 2, 0])
    features = K.reshape(features, (m_, -1))
    features = K.dot(fltr, features)
    features = K.reshape(features, (m_, f_, -1))
    features = K.permute_dimensions(features, [2, 0, 1])

    return features


class GraphAttention(layers.Layer):
    """
    A graph attention layer as presented by
    [Velickovic et al. (2017)](https://arxiv.org/abs/1710.10903).

    **Mode**: single, mixed, batch.

    This layer computes a convolution similar to `layers.GraphConv`, but
    uses the attention mechanism to weight the adjacency matrix instead of
    using the normalized Laplacian.

    **Input**

    - node features of shape `(n_nodes, n_features)` (with optional `batch`
    dimension);
    - adjacency matrices of shape `(n_nodes, n_nodes)` (with optional `batch`
    dimension);

    **Output**

    - node features with the same shape of the input, but the last dimension
    changed to `channels`.

    **Arguments**

    - `channels`: integer, number of output channels;
    - `attn_heads`: number of attention heads to use;
    - `attn_heads_reduction`: how to reduce the outputs of the attention heads
    (can be either 'concat' or 'average');
    - `dropout_rate`: internal dropout rate;
    - `activation`: activation function to use;
    - `use_bias`: boolean, whether to add a bias to the linear transformation;
    - `kernel_initializer`: initializer for the kernel matrix;
    - `attn_kernel_initializer`: initializer for the attention kernel matrices;
    - `bias_initializer`: initializer for the bias vector;
    - `kernel_regularizer`: regularization applied to the kernel matrix;
    - `attn_kernel_regularizer`: regularization applied to the attention kernel
    matrices;
    - `bias_regularizer`: regularization applied to the bias vector;
    - `activity_regularizer`: regularization applied to the output;
    - `kernel_constraint`: constraint applied to the kernel matrix;
    - `attn_kernel_constraint`: constraint applied to the attention kernel
    matrices;
    - `bias_constraint`: constraint applied to the bias vector.

    **Usage**

    ```py
    # Load data
    A, X, _, _, _, _, _, _ = citation.load_data('cora')

    # Preprocessing operations
    A = utils.add_eye(A).toarray()  # Add self-loops

    # Model definition
    X_in = Input(shape=(F, ))
    A_in = Input((N, ))
    output = GraphAttention(channels)([X_in, A_in])
    ```
    """

    def __init__(self,
                 channels,
                 attn_heads=1,
                 attn_heads_reduction='concat',  # {'concat', 'average'}
                 dropout_rate=0.5,
                 activation='relu',
                 use_bias=True,
                 kernel_initializer='glorot_uniform',
                 bias_initializer='zeros',
                 attn_kernel_initializer='glorot_uniform',
                 kernel_regularizer=None,
                 bias_regularizer=None,
                 attn_kernel_regularizer=None,
                 activity_regularizer=None,
                 kernel_constraint=None,
                 bias_constraint=None,
                 attn_kernel_constraint=None,
                 **kwargs):
        if attn_heads_reduction not in {'concat', 'average'}:
            raise ValueError('Possbile reduction methods: concat, average')

        self.channels = channels
        self.attn_heads = attn_heads
        self.attn_heads_reduction = attn_heads_reduction
        self.dropout_rate = dropout_rate


        self.activation = activations.get(activation)
        self.use_bias = use_bias
        self.kernel_initializer = initializers.get(kernel_initializer)
        self.bias_initializer = initializers.get(bias_initializer)
        self.attn_kernel_initializer = initializers.get(attn_kernel_initializer)
        self.kernel_regularizer = regularizers.get(kernel_regularizer)
        self.bias_regularizer = regularizers.get(bias_regularizer)
        self.attn_kernel_regularizer = regularizers.get(attn_kernel_regularizer)
        self.activity_regularizer = regularizers.get(activity_regularizer)
        self.kernel_constraint = constraints.get(kernel_constraint)
        self.bias_constraint = constraints.get(bias_constraint)
        self.attn_kernel_constraint = constraints.get(attn_kernel_constraint)

        self.supports_masking = False

        # Populated by build()
        self.kernels = []  # Layer kernels for attention heads
        self.biases = []  # Layer biases for attention heads
        self.attn_kernels = []  # Attention kernels for attention heads

        if attn_heads_reduction == 'concat':
            # Output will have shape (..., attention_heads * channels)
            self.output_dim = self.channels * self.attn_heads
        else:
            # Output will have shape (..., channels)
            self.output_dim = self.channels

        super(GraphAttention, self).__init__(**kwargs)

    def build(self, input_shape):
        assert len(input_shape) >= 2
        input_dim = int(input_shape[0][-1])

        # Initialize weights for each attention head
        for head in range(self.attn_heads):
            # Layer kernel
            kernel = self.add_weight(shape=(input_dim, self.channels),
                                     initializer=self.kernel_initializer,
                                     regularizer=self.kernel_regularizer,
                                     constraint=self.kernel_constraint,
                                     name='kernel_{}'.format(head))
            self.kernels.append(kernel)

            # Layer bias
            if self.use_bias:
                bias = self.add_weight(shape=(self.channels,),
                                       initializer=self.bias_initializer,
                                       regularizer=self.bias_regularizer,
                                       constraint=self.bias_constraint,
                                       name='bias_{}'.format(head))
                self.biases.append(bias)

            # Attention kernels
            attn_kernel_self = self.add_weight(shape=(self.channels, 1),
                                               initializer=self.attn_kernel_initializer,
                                               regularizer=self.attn_kernel_regularizer,
                                               constraint=self.attn_kernel_constraint,
                                               name='attn_kernel_self_{}'.format(head))
            attn_kernel_neighs = self.add_weight(shape=(self.channels, 1),
                                                 initializer=self.attn_kernel_initializer,
                                                 regularizer=self.attn_kernel_regularizer,
                                                 constraint=self.attn_kernel_constraint,
                                                 name='attn_kernel_neigh_{}'.format(head))
            self.attn_kernels.append([attn_kernel_self, attn_kernel_neighs])
        self.built = True

    def call(self, inputs):
        X = inputs[0]  # Node features (N x F)
        A = inputs[1]  # Adjacency matrix (N x N)

        outputs = []
        for head in range(self.attn_heads):
            kernel = self.kernels[head]  # W in the paper (F x F')
            attention_kernel = self.attn_kernels[head]  # Attention kernel a in the paper (2F' x 1)

            # Compute inputs to attention network
            features = K.dot(X, kernel)  # (N x F')

            # Compute feature combinations
            # Note: [[a_1], [a_2]]^T [[Wh_i], [Wh_2]] = [a_1]^T [Wh_i] + [a_2]^T [Wh_j]
            attn_for_self = K.dot(features, attention_kernel[0])  # (N x 1), [a_1]^T [Wh_i]
            attn_for_neighs = K.dot(features, attention_kernel[1])  # (N x 1), [a_2]^T [Wh_j]

            # Attention head a(Wh_i, Wh_j) = a^T [[Wh_i], [Wh_j]]
            if len(K.int_shape(features)) == 2:
                attn_for_neighs_T = K.transpose(attn_for_neighs)
            else:
                attn_for_neighs_T = K.permute_dimensions(attn_for_neighs, (0, 2, 1))
            dense = attn_for_self + attn_for_neighs_T

            # Add nonlinearity
            dense = tf.keras.layers.LeakyReLU(alpha=0.2)(dense)

            # Mask values before activation (Vaswani et al., 2017)
            mask = -10e9 * (1.0 - A)
            dense += mask

            # Apply softmax to get attention coefficients
            dense = K.softmax(dense)  # (N x N)

            # Apply dropout to features and attention coefficients
            dropout_attn = tf.keras.layers.Dropout(self.dropout_rate)(dense)  # (N x N)
            dropout_feat = tf.keras.layers.Dropout(self.dropout_rate)(features)  # (N x F')

            # Convolution
            node_features = filter_dot(dropout_attn, dropout_feat)

            if self.use_bias:
                node_features = K.bias_add(node_features, self.biases[head])

            # Add output of attention head to final output
            outputs.append(node_features)

        # Aggregate the heads' output according to the reduction method
        if self.attn_heads_reduction == 'concat':
            output = K.concatenate(outputs)  # (N x KF')
        else:
            output = K.mean(K.stack(outputs), axis=0)  # N x F')

        output = self.activation(output)
        return output

    def compute_output_shape(self, input_shape):
        output_shape = input_shape[0][:-1] + (self.output_dim,)
        return output_shape

    def get_config(self):
        config = {
            'channels': self.channels,
            'attn_heads': self.attn_heads,
            'attn_heads_reduction': self.attn_heads_reduction,
            'dropout_rate': self.dropout_rate,
            'activation': activations.serialize(self.activation),
            'use_bias': self.use_bias,
            'kernel_initializer': initializers.serialize(self.kernel_initializer),
            'bias_initializer': initializers.serialize(self.bias_initializer),
            'attn_kernel_initializer': initializers.serialize(self.attn_kernel_initializer),
            'kernel_regularizer': regularizers.serialize(self.kernel_regularizer),
            'bias_regularizer': regularizers.serialize(self.bias_regularizer),
            'attn_kernel_regularizer': regularizers.serialize(self.attn_kernel_regularizer),
            'activity_regularizer': regularizers.serialize(self.activity_regularizer),
            'kernel_constraint': constraints.serialize(self.kernel_constraint),
            'bias_constraint': constraints.serialize(self.bias_constraint),
            'attn_kernel_constraint': constraints.serialize(self.attn_kernel_constraint),
        }
        base_config = super(GraphAttention, self).get_config()
        return dict(list(base_config.items()) + list(config.items()))



# Parameters
gat_channels = 8              # Output size of first GraphAttention layer
n_attn_heads = 8              # Number of attention heads in first GAT layer
N = 15   # Number of nodes in the graph
F = 1   # Original feature dimensionality
n_classes = 7  # Number of classes
dropout_rate = 0.25           # Dropout rate applied to the input of GAT layers
l2_reg = 5e-4                 # Regularization rate for l2
learning_rate = 1e-2          # Learning rate for SGD
epochs = 20000                # Number of training epochs
es_patience = 200             # Patience fot early stopping

# tf.enable_eager_execution()

# Model definition
X_in = tf.keras.layers.Input(batch_shape=(None, N, F))
A_in = tf.keras.layers.Input(batch_shape=(None, N, N))

dropout_1 = tf.keras.layers.Dropout(dropout_rate)(X_in)
graph_attention_1 = GraphAttention(gat_channels,
                                   attn_heads=n_attn_heads,
                                   attn_heads_reduction='concat',
                                   dropout_rate=dropout_rate,
                                   activation='elu',
                                   kernel_regularizer=l2(l2_reg),
                                   attn_kernel_regularizer=l2(l2_reg))([dropout_1, A_in])
dropout_2 = tf.keras.layers.Dropout(dropout_rate)(graph_attention_1)
graph_attention_2 = GraphAttention(n_classes,
                                   attn_heads=1,
                                   attn_heads_reduction='average',
                                   dropout_rate=dropout_rate,
                                   activation='softmax',
                                   kernel_regularizer=l2(l2_reg),
                                   attn_kernel_regularizer=l2(l2_reg))([dropout_2, A_in])

# Build model
model = tf.keras.Model(inputs=[X_in, A_in], outputs=graph_attention_2)

# Test model

G = nx.barabasi_albert_graph(15, 3)
adj = nx.to_numpy_matrix(G , nodelist=range(G.number_of_nodes()), dtype=np.float32)
feat = np.sum(adj, axis=1)
emb_node = model([np.expand_dims(feat, 0), np.expand_dims(adj, 0)])
print(emb_node)

with tf.Session() as sess:
    print(sess.run(emb_node))

Thanks in advance

Shanchao

Which keras version we need to install to use spektral v0.3?

So my intuitive guess for how the MNIST dataset worked would be classifying the graph of just the positive pixels into 0-9. The example being a 2.

Is that an interesting problem to use as a benchmark for classification or is the fixed adjacency matrix and varying features better suited?

In original ReadMe: giving rise to the filed of relational representation learning (RRL).
I believe there might be a typo: field.

It seems that Mincut result is not reproducible. I don't know how to set the seed.

Hi Daniele,
Is R-GNN implemented in spektral?

Hi,

In this file, the edge features are being one-hotted but the one-hot version is not used later on. (Lines 38, 49, ...) Am I missing something?

Hello @danielegrattarola, may you please deliver in examples the use of GINConv layer? I have a problem during passing a tensor (output of Keras "Input Layer") to this layer (model definition). Its connected with propagate method in Message Passing class:

Model Structure:

X_in = Input(shape=(F, ))
A_in = Input(shape=(N, ), sparse=True)
gc1 = GINConv(channels=300, mlp_activation='relu',)([X_in, A_in])

Error relation:
self.index_i = A.indices[:, 0]

Error Type:
TypeError: 'SparseTensor' object is not subscriptable.

Hi, thanks for the great package!, I'm trying to train embedding of graph vertices through link prediction using GraphSage conv.
Usually training goes without problems, but once every few hundred batches this error appears:

0 1614 train_epoch_loss 0.12077645 train_epoch_acc 0.83266443
0 1615 train_epoch_loss 0.12074445 train_epoch_acc 0.8327161
0 1616 train_epoch_loss 0.120728664 train_epoch_acc 0.832744
0 1617 train_epoch_loss 0.1207294 train_epoch_acc 0.8327376
0 1618 train_epoch_loss 0.12071675 train_epoch_acc 0.83276695
0 1619 train_epoch_loss 0.120711125 train_epoch_acc 0.8327747

0 1620 train_epoch_loss 0.1206873 train_epoch_acc 0.8328143
ConcatOp : Dimensions of inputs should match: shape[0] = [94,200] vs. shape[1] = [93,200]
[[node graph_sage_conv_1/concat (defined at /home/user/miniconda3/envs/safe/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:1956) ]]
[[node graph_sage_conv_3/GatherV2 (defined at /home/user/miniconda3/envs/safe/lib/python3.7/site-packages/spektral/layers/convolutional.py:378) ]]

Caused by op 'graph_sage_conv_1/concat', defined at:
File "/home/user/miniconda3/envs/safe/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/home/user/miniconda3/envs/safe/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/ipykernel_launcher.py", line 16, in
app.launch_new_instance()
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/traitlets/config/application.py", line 658, in launch_instance
app.start()
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/ipykernel/kernelapp.py", line 505, in start
self.io_loop.start()
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/tornado/platform/asyncio.py", line 148, in start
self.asyncio_loop.run_forever()
File "/home/user/miniconda3/envs/safe/lib/python3.7/asyncio/base_events.py", line 539, in run_forever
self._run_once()
File "/home/user/miniconda3/envs/safe/lib/python3.7/asyncio/base_events.py", line 1775, in _run_once
handle._run()
File "/home/user/miniconda3/envs/safe/lib/python3.7/asyncio/events.py", line 88, in _run
self._context.run(self._callback, *self._args)
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/tornado/ioloop.py", line 690, in
lambda f: self._run_callback(functools.partial(callback, future))
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/tornado/ioloop.py", line 743, in _run_callback
ret = callback()
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/tornado/gen.py", line 781, in inner
self.run()
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.send(value)
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/ipykernel/kernelbase.py", line 365, in process_one
yield gen.maybe_future(dispatch(*args))
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/tornado/gen.py", line 209, in wrapper
yielded = next(result)
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/ipykernel/kernelbase.py", line 272, in dispatch_shell
yield gen.maybe_future(handler(stream, idents, msg))
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/tornado/gen.py", line 209, in wrapper
yielded = next(result)
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/ipykernel/kernelbase.py", line 542, in execute_request
user_expressions, allow_stdin,
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/tornado/gen.py", line 209, in wrapper
yielded = next(result)
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/ipykernel/ipkernel.py", line 294, in do_execute
res = shell.run_cell(code, store_history=store_history, silent=silent)
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/ipykernel/zmqshell.py", line 536, in run_cell
return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 2848, in run_cell
raw_cell, store_history, silent, shell_futures)
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 2874, in _run_cell
return runner(coro)
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/IPython/core/async_helpers.py", line 67, in _pseudo_sync_runner
coro.send(None)
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3049, in run_cell_async
interactivity=interactivity, compiler=compiler, result=result)
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3214, in run_ast_nodes
if (yield from self.run_code(code, result)):
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3296, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 30, in
x = GraphSageConv(bb_embed_size)([x, A_in])
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/keras/engine/base_layer.py", line 457, in call
output = self.call(inputs, **kwargs)
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/spektral/layers/convolutional.py", line 380, in call
output = K.concatenate([features, features_neigh])
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py", line 1956, in concatenate
return tf.concat([to_dense(x) for x in tensors], axis)
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py", line 180, in wrapper
return target(*args, **kwargs)
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py", line 1256, in concat
return gen_array_ops.concat_v2(values=values, axis=axis, name=name)
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 1149, in concat_v2
"ConcatV2", values=values, axis=axis, name=name)
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op
op_def=op_def)
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 1801, in init
self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): ConcatOp : Dimensions of inputs should match: shape[0] = [94,200] vs. shape[1] = [93,200]
[[node graph_sage_conv_1/concat (defined at /home/user/miniconda3/envs/safe/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:1956) ]]
[[node graph_sage_conv_3/GatherV2 (defined at /home/user/miniconda3/envs/safe/lib/python3.7/site-packages/spektral/layers/convolutional.py:378) ]]

0 1622 train_epoch_loss 0.12065845 train_epoch_acc 0.83286613
0 1623 train_epoch_loss 0.120651215 train_epoch_acc 0.83288383
0 1624 train_epoch_loss 0.12064571 train_epoch_acc 0.8328906
0 1625 train_epoch_loss 0.12062876 train_epoch_acc 0.83291864
ConcatOp : Dimensions of inputs should match: shape[0] = [159,200] vs. shape[1] = [158,200]
[[node graph_sage_conv_1/concat (defined at /home/user/miniconda3/envs/safe/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:1956) ]]
[[node graph_sage_conv_3/GatherV2 (defined at /home/user/miniconda3/envs/safe/lib/python3.7/site-packages/spektral/layers/convolutional.py:378) ]]

Caused by op 'graph_sage_conv_1/concat', defined at:
File "/home/user/miniconda3/envs/safe/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/home/user/miniconda3/envs/safe/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/ipykernel_launcher.py", line 16, in
app.launch_new_instance()
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/traitlets/config/application.py", line 658, in launch_instance
app.start()
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/ipykernel/kernelapp.py", line 505, in start
self.io_loop.start()
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/tornado/platform/asyncio.py", line 148, in start
self.asyncio_loop.run_forever()
File "/home/user/miniconda3/envs/safe/lib/python3.7/asyncio/base_events.py", line 539, in run_forever
self._run_once()
File "/home/user/miniconda3/envs/safe/lib/python3.7/asyncio/base_events.py", line 1775, in _run_once
handle._run()
File "/home/user/miniconda3/envs/safe/lib/python3.7/asyncio/events.py", line 88, in _run
self._context.run(self._callback, *self._args)
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/tornado/ioloop.py", line 690, in
lambda f: self._run_callback(functools.partial(callback, future))
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/tornado/ioloop.py", line 743, in _run_callback
ret = callback()
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/tornado/gen.py", line 781, in inner
self.run()
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.send(value)
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/ipykernel/kernelbase.py", line 365, in process_one
yield gen.maybe_future(dispatch(*args))
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/tornado/gen.py", line 209, in wrapper
yielded = next(result)
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/ipykernel/kernelbase.py", line 272, in dispatch_shell
yield gen.maybe_future(handler(stream, idents, msg))
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/tornado/gen.py", line 209, in wrapper
yielded = next(result)
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/ipykernel/kernelbase.py", line 542, in execute_request
user_expressions, allow_stdin,
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/tornado/gen.py", line 209, in wrapper
yielded = next(result)
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/ipykernel/ipkernel.py", line 294, in do_execute
res = shell.run_cell(code, store_history=store_history, silent=silent)
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/ipykernel/zmqshell.py", line 536, in run_cell
return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 2848, in run_cell
raw_cell, store_history, silent, shell_futures)
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 2874, in _run_cell
return runner(coro)
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/IPython/core/async_helpers.py", line 67, in _pseudo_sync_runner
coro.send(None)
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3049, in run_cell_async
interactivity=interactivity, compiler=compiler, result=result)
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3214, in run_ast_nodes
if (yield from self.run_code(code, result)):
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3296, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 30, in
x = GraphSageConv(bb_embed_size)([x, A_in])
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/keras/engine/base_layer.py", line 457, in call
output = self.call(inputs, **kwargs)
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/spektral/layers/convolutional.py", line 380, in call
output = K.concatenate([features, features_neigh])
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py", line 1956, in concatenate
return tf.concat([to_dense(x) for x in tensors], axis)
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py", line 180, in wrapper
return target(*args, **kwargs)
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py", line 1256, in concat
return gen_array_ops.concat_v2(values=values, axis=axis, name=name)
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 1149, in concat_v2
"ConcatV2", values=values, axis=axis, name=name)
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op
op_def=op_def)
File "/home/user/miniconda3/envs/safe/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 1801, in init
self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): ConcatOp : Dimensions of inputs should match: shape[0] = [159,200] vs. shape[1] = [158,200]
[[node graph_sage_conv_1/concat (defined at /home/user/miniconda3/envs/safe/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py:1956) ]]
[[node graph_sage_conv_3/GatherV2 (defined at /home/user/miniconda3/envs/safe/lib/python3.7/site-packages/spektral/layers/convolutional.py:378) ]]

0 1627 train_epoch_loss 0.12060276 train_epoch_acc 0.83297145
0 1628 train_epoch_loss 0.120608345 train_epoch_acc 0.83296674
0 1629 train_epoch_loss 0.12059478 train_epoch_acc 0.832987
0 1630 train_epoch_loss 0.1205696 train_epoch_acc 0.83303547
0 1631 train_epoch_loss 0.120563745 train_epoch_acc 0.83303756
0 1632 train_epoch_loss 0.12054803 train_epoch_acc 0.833069
0 1633 train_epoch_loss 0.12054562 train_epoch_acc 0.8330679`

Loss decreases, acc increases. It seems the network is learning.
I control all the dimensions of my data through assert.
Tell me what could be the problem?

Hi,

with the Disjoint2Batch layer committed, I was wondering if there is need for a Batch2Disjoint layer.
My idea is, to then fuse these two into a ModeMatch layer that can go between any two spektral layers and will (potentially automatically) convert the output of the first layer to match the needed mode of the second.

On the up side, this would allow all layers to communicate with each other and if build into the GraphConv/MessagePassing base class, it could remove some headaches from new users by allowing them to not care much about modes.

On the down side, this conversion would then be part of the model and not just the preprocessing, which could have noticeable effects on training and memory usage.

Do you think it is worth-while?

Best,
Levi

Hi!

I am very curious! Do you haven an opinion about GraphVAE?
https://hal.archives-ouvertes.fr/hal-01990381/file/icann_paper_long.pdf

Best,
Tristan

Hi Daniele

Thanks so much for creating spektral! A very useful tool for the community. I am running your spektral/examples/classification_delaunay.py example, and met with the following bug. Could you please take a look? Maybe I missed something? Everything before the "output..." line went ok.

Tao

>>> output = Dense(n_classes, activation='softmax')(pool)
Traceback (most recent call last):
  File "/home2/twang6/.conda/envs/spektral/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1659, in _create_c_op
    c_op = c_api.TF_FinishOperation(op_desc)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Dimensions must be equal, but are 128 and 32 for 'dense_1/MatMul' (op: 'MatMul') with input shapes: [?,128], [32,2].

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home2/twang6/.conda/envs/spektral/lib/python3.6/site-packages/keras/engine/base_layer.py", line 457, in __call__
    output = self.call(inputs, **kwargs)
  File "/home2/twang6/.conda/envs/spektral/lib/python3.6/site-packages/keras/layers/core.py", line 879, in call
    output = K.dot(inputs, self.kernel)
  File "/home2/twang6/.conda/envs/spektral/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 1085, in dot
    out = tf.matmul(x, y)
  File "/home2/twang6/.conda/envs/spektral/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py", line 2455, in matmul
    a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name)
  File "/home2/twang6/.conda/envs/spektral/lib/python3.6/site-packages/tensorflow/python/ops/gen_math_ops.py", line 5333, in mat_mul
    name=name)
  File "/home2/twang6/.conda/envs/spektral/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "/home2/twang6/.conda/envs/spektral/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/home2/twang6/.conda/envs/spektral/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op
    op_def=op_def)
  File "/home2/twang6/.conda/envs/spektral/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1823, in __init__
    control_input_ops)
  File "/home2/twang6/.conda/envs/spektral/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1662, in _create_c_op
    raise ValueError(str(e))
ValueError: Dimensions must be equal, but are 128 and 32 for 'dense_1/MatMul' (op: 'MatMul') with input shapes: [?,128], [32,2].

Namaste!

Wow! spekral looks really great! I am currently investing some time to dig through the repository. I am amazed. Good work!

One thing: GraphAttention, as other attention layers, computes attention coefficients. Usually, these are visualized to learn a little more about attention during predictions.

Would it be feasible to make those available? Maybe a init parameter yield_attention_coefficients. If true, it would return both the output activation and the attention coefficients.

Hi @danielegrattarola ,

I just found this library and am excited to start using it! I really appreciate the examples and great documentation. After reading, I still have one question: do you have any advice on modeling dynamically sized graphs (where the node count varies from graph to graph)?

My naive instinct is to train a network with a large N and zero-pad the unused elements when the number of nodes is < N. I am not sure if the zeros should be at the beginning, end, or just anywhere. I am interested to hear your input.

Thanks in advance for any advice!
-Jack

Hi Daniele,
is there a simple way to use pooling layers, other than the global ones, in case of a fixed graph topology (e.g. fixed node positions in a 3D space) and a set of graph signals? Furthermore i was wondering if it is possible to include the mixed mode in EdgeConditionedConv layers?
Thanks in advance.
Cheers

I am trying to solve the NFL Big Data Bowl 2020 Kaggle competitions using GNNs. Basically you get the position and velocity of each player on the field and you try to predict the number of yards the player with the ball will move.

I've read the following papers:

1. SEMI-SUPERVISED CLASSIFICATION WITH GRAPH CONVOLUTIONAL NETWORKS
2. Relational inductive biases, deep learning, and graph networks

The 2nd paper from Deep Mind gives a good overview of the field and its easy to understand, while the 1st paper goes more into the implementation details of a GraphConv layer.

In terms of Algorithm 1 from the Deep Mind paper, it seems that spektral supports:

• 3. Compute updated node attributes via Convolutional Layers
• 5. Aggregate node attributes globally via Pooling Layers
• 6. Compute updated global attribute via regular Keras (at this point there is a single vector per graph)

The key missing steps are:

• 1. Compute updated edge attributes
• 2. Aggregate edge attributes per node

The first basically calculates/learns edge features between each pair of nodes, while the second reduces this back into a computed feature for the node. In their physics analogy, 1 would calculate the forces on an object given its interactions with other objects and 2 would aggregate them into total force for each object.

I've been thinking that maybe convolutional layers in Spektral might be doing 1 and 2 implicitly, but 1 is very explicit in that you would have to input/concatenate pairs of node features which doesn't seem to happen.

I mostly just wanted to check if my rational is on the right track, if this is useful I might contribute a very naive implementation of the procedure.

Running the following code using the Spektral 0.5.0 throws an unexpected error:

from spektral.datasets import tud
from spektral.utils import numpy_to_batch

A_list, X_list, y = tud.load_data('MUTAG', clean=True)
numpy_to_batch(X_list, A_list)

The ValueError it throws is:
ValueError: operands could not be broadcast together with remapped shapes [original->remapped]: (2,2) and requested shape (0,2)

Try it out in in collab

Hello,
I'm trying to build a GraphSAGE model with spektral,I can find the graphsage conv layer amongst layers but I don't know how to use it, especially about the neighbor sampling part. And it seems that there is no graphsage example in examples folder either. Could you please provide an example using graphsage? Thanks!

As a very visual learner, I thought it might be useful to have some more notebook-style examples with figures showing topology and outputs of intermediate layers, but I can appreciate they can unnecessarily bulk up a repository and make testing and versioning more difficult. I have made one here, should I make a PR or a new repo?

Hi,
when i am executing the below code -

Model definition

X_in = Input(shape=(F, )) # Input layer for X
A_in = Input((N, ), sparse=True) # Input layer for A

graph_conv_1 = GraphConv(16, activation='relu')([X_in, A_in])
dropout = Dropout(0.5)(graph_conv_1)
graph_conv_2 = GraphConv(n_classes, activation='softmax')([dropout, A_in])

model = Model(inputs=[X_in, A_in], outputs=graph_conv_2)

Build model

I got the below error-
I have used colab and tried different versions of tensorflow and keras.
AttributeError Traceback (most recent call last)
in ()
2 A_in = Input((N, ), sparse=True) # Input layer for A
3
----> 4 graph_conv_1 = GraphConv(16, activation='relu')([X_in, A_in])
5 dropout = Dropout(0.5)(graph_conv_1)
6 graph_conv_2 = GraphConv(n_classes, activation='softmax')([dropout, A_in])

5 frames
/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py in (t)
2027 call method of the layer at the call that created the node.
2028 """
-> 2029 inbound_layers = nest.map_structure(lambda t: t._keras_history.layer,
2030 input_tensors)
2031 node_indices = nest.map_structure(lambda t: t._keras_history.node_index,

AttributeError: 'tuple' object has no attribute 'layer'

Please help me with this error asap.

Hi!
I'm trying to run the example on the "Getting started" page of Spektral website. I'm using the exact codes there. I'm receiving "ValueError:A SciPy sparse matrix was passed to a model that expects dense inputs. Please densify your inputs first, such as by calling `x.toarray()." When I densify X by X = X.todense(): "ValueError: The two structures don't have the same nested structure." And when I change the input by "X_in = Input(shape=(F, ), sparse=True) # Input layer for X" I receive this error:"TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'". Can you please help me to run this example?

Thank you very much for developing the graph neural network implemented by keras. Is GAT code used for node classification and graph classification stable? If I want to use your API, what is the limitation of keras version? Have you changed your code to tf.keras?

Can someone explain if thats make sense? My training for disjoint mode data is too slow.

Hi Daniele,
I am trying to use Graph Convolutional Networks to forecast traffic flows between origin and destination cities. I only have one graph so it should be a single mode.
I am wondering is it possible to get a matrix format output (N*N) using GraphConv layer and LSTM layer? Will this output considering both the interactions between origin cities and destination cities?
I will really appreciate your kind help.
Thanks!

Hi,
Thanks for your great work.

Could you please support Tensorflow 2 for Spektral?

This issue not occur with tensorflow 1

`

Model definition

from spektral.layers import GraphConv, GraphSageConv
from spektral.layers import *
import tensorflow as tf

from tensorflow.keras.layers import Convolution2D, Input
from tensorflow.keras.models import Model
from tensorflow.keras.layers import *
from tensorflow.keras.callbacks import *
from tensorflow.keras import backend as K
from spektral.layers.ops import sp_matrix_to_sp_tensor
from tensorflow.keras.metrics import binary_accuracy
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.initializers import RandomUniform

r = RandomUniform(minval=-1, maxval=1, seed=42)
activation='relu'

S = 2

X_in = Input(shape=(N, F))
A_in = Input(shape=(N, N))

M = 50
M2 = 10
M3 = 5

def access_column(arr,col):
return arr[:,:,col]

def access_column_dims(arr,col):
return K.expand_dims(arr[:,col],axis=1)

x_1 = Lambda(lambda x: access_column(x,0))(X_in)
e1 = Embedding(total_items, M2, embeddings_initializer=r)(x_1)
e1 = Reshape((M2,-1))(e1)

x_2 = Lambda(lambda x: access_column(x,1))(X_in)
e2 = Embedding(total_categories, M2, embeddings_initializer=r)(x_2)
e2 = Reshape((M2,-1))(e2)

e3 = Lambda(lambda x: access_column(x,2))(X_in) #items clicked
e3 = Dense(10, input_shape=(N,1))(e3)
e3 = Reshape((M2,-1))(e3)

e4 = Lambda(lambda x: access_column(x,3))(X_in) #dwelltime
e4 = Dense(10, input_shape=(N,1))(e4)
e4 = Reshape((M2,-1))(e4)

c = Concatenate(axis=-1)([e1, e2, e3, e4])

n_attn_heads = 8
n1 = 64
n2 = 64

l = GraphAttention(n1, activation='relu', attn_heads = n_attn_heads)([c, A_in]) # skip-connection
l = GraphAttention(n1, activation='relu', attn_heads = n_attn_heads)([l, A_in])
l = GlobalAttentionPool(n2)(l)

output = Dense(1, activation='sigmoid')(l)
model = Model(inputs=[X_in, A_in], outputs=output)

opt = Adam(lr=0.0001)

loss_f = 'binary_crossentropy'

model.compile(optimizer=opt, loss=loss_f, metrics=[f1_m, precision_m, recall_m])
model.summary()
`

Output:
`
/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/init_ops_v2.py in _compute_fans(shape)
733 receptive_field_size = 1.
734 for dim in shape[:-2]:
--> 735 receptive_field_size *= dim
736 fan_in = shape[-2] * receptive_field_size
737 fan_out = shape[-1] * receptive_field_size

TypeError: unsupported operand type(s) for *=: 'float' and 'NoneType'
`
Please help :(

import spektral
import numpy as np
labels = np.array([[1,2,3],[4,5,6]])
adj = np.array([[0,1],[1,0]])
spektral.utils.numpy_to_nx(adj,labels)

results in:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/thomas/Library/Python/2.7/lib/python/site-packages/spektral/utils/conversion.py", line 192, in numpy_to_nx
    nx.set_node_attributes(g, node_attrs, nf_name)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/networkx/classes/function.py", line 660, in set_node_attributes
    G.nodes[n].update(d)
TypeError: cannot convert dictionary update sequence element #0 to a sequence

Using Python 2.7.16. pip reports Keras 2.2.5, tensorflow 1.14.0, and spektral 0.0.13.

I was looking at the MultiHeadAttention implementation from the BERT repo on tensorflow/models and saw they implement this operation without branching + concatenation. They use 2 tricks:

• They create a multi-head version of Dense
• They use tf.einsum to do the varios dot products while taking into account the heads.

I can try to convert the current implementation to this possibly more efficient strategy on the tf2 branch.

Checkout this striped down implementation of MHA I created recently: code

References:

1. Attention Layer from BERT
2. Official Tensorflow Tutorial