tensorflow / neural-structured-learning

Training neural models with structured signals.

Neural Structured Learning in TensorFlow

Neural Structured Learning (NSL) is a new learning paradigm to train neural networks by leveraging structured signals in addition to feature inputs. Structure can be explicit as represented by a graph [1,2,5] or implicit as induced by adversarial perturbation [3,4].

Structured signals are commonly used to represent relations or similarity among samples that may be labeled or unlabeled. Leveraging these signals during neural network training harnesses both labeled and unlabeled data, which can improve model accuracy, particularly when the amount of labeled data is relatively small. Additionally, models trained with samples that are generated by adversarial perturbation have been shown to be robust against malicious attacks, which are designed to mislead a model's prediction or classification.

NSL generalizes to Neural Graph Learning [1] as well as to Adversarial Learning [3]. The NSL framework in TensorFlow provides the following easy-to-use APIs and tools for developers to train models with structured signals:

• Keras APIs to enable training with graphs (explicit structure) and adversarial perturbations (implicit structure).

• TF ops and functions to enable training with structure when using lower-level TensorFlow APIs

• Tools to build graphs and construct graph inputs for training

The NSL framework is designed to be flexible and can be used to train any kind of neural network. For example, feed-forward, convolution, and recurrent neural networks can all be trained using the NSL framework. In addition to supervised and semi-supervised learning (a low amount of supervision), NSL can in theory be generalized to unsupervised learning. Incorporating structured signals is done only during training, so the performance of the serving/inference workflow remains unchanged. Please check out our tutorials for a practical introduction to NSL.

Getting started

You can install the prebuilt NSL pip package by running:

pip install neural-structured-learning

For more detailed instructions on how to install NSL as a package or to build it from source in various environments, please see the installation guide

Note that NSL requires a TensorFlow version of 1.15 or higher. NSL also supports TensorFlow 2.x with the exception of v2.1, which contains a bug that is incompatible with NSL.

Videos and Colab Tutorials

Get a jump-start on NSL by watching our video series on YouTube! It gives a complete overview of the framework as well as discusses several aspects of learning with structured signals.

We've also created hands-on colab-based tutorials that will allow you to interactively explore NSL. Here are a few:

• training with natural graphs
• training with synthesized graphs
• adversarial learning

You can find more examples and tutorials under the examples directory.

Contributing to NSL

Contributions are welcome and highly appreciated - there are several ways to contribute to TF Neural Structured Learning:

• Case studies: If you are interested in applying NSL, consider wrapping up your usage as a tutorial, a new dataset, or an example model that others could use for experiments and/or development. The examples directory could be a good destination for such contributions.

• Product excellence: If you are interested in improving NSL's product excellence and developer experience, the best way is to clone this repo, make changes directly on the implementation in your local repo, and then send us pull request to integrate your changes.

• New algorithms: If you are interested in developing new algorithms for NSL, the best way is to study the implementations of NSL libraries, and to think of extensions to the existing implementation (or alternative approaches). If you have a proposal for a new algorithm, we recommend starting by staging your project in the research directory and including a colab notebook to showcase the new features. If you develop new algorithms in your own repository, we would be happy to feature pointers to academic publications and/or repositories using NSL from this repository.

Please be sure to review the contribution guidelines.

Research

See our research directory for research projects in Neural Structured Learning:

• Low-Dimensional Hyperbolic Knowledge Graph Embeddings
• A2N: Attending to Neighbors for Knowledge Graph Inference
• GAM: Graph Agreement Models for Semi-Supervised Learning
• Neural Clustering Processes
• CARLS: Cross-platform Asynchronous Representation Learning System
• Denoised Smoothing: A Provable Defense for Pretrained Classifiers

Featured Usage

Please see the usage page to learn more about how NSL is being discussed and used in the open source community.

Issues, Questions, and Feedback

Please use GitHub issues to file issues, bugs, and feature requests. For questions, please direct them to Stack Overflow with the "nsl" tag. For feedback, please fill this form; we would love to hear from you.

Release Notes

Please see the release notes for detailed version updates.

References

[1] T. Bui, S. Ravi and V. Ramavajjala. "Neural Graph Learning: Training Neural Networks Using Graphs." WSDM 2018

[2] T. Kipf and M. Welling. "Semi-supervised classification with graph convolutional networks." ICLR 2017

[3] I. Goodfellow, J. Shlens and C. Szegedy. "Explaining and harnessing adversarial examples." ICLR 2015

[4] T. Miyato, S. Maeda, M. Koyama and S. Ishii. "Virtual Adversarial Training: a Regularization Method for Supervised and Semi-supervised Learning." ICLR 2016

[5] D. Juan, C. Lu, Z. Li, F. Peng, A. Timofeev, Y. Chen, Y. Gao, T. Duerig, A. Tomkins and S. Ravi "Graph-RISE: Graph-Regularized Image Semantic Embedding." WSDM 2020

neural-structured-learning's Issues

KeyError: 'input_2' while running data generators in fit_generator

Creating a new issue for this comment #73 (comment)
Using fit generator and nsl to train an image multi-label classifier
I have already split full_df into train_df and valid_df dataframes.

IMG_SIZE = (128, 128)
core_idg = ImageDataGenerator(samplewise_center=True, 
                              samplewise_std_normalization=True, 
                              horizontal_flip = True, 
                              vertical_flip = False, 
                              height_shift_range= 0.05, 
                              width_shift_range=0.1, 
                              rotation_range=5, 
                              shear_range = 0.1,
                              fill_mode = 'reflect',
                              zoom_range=0.15)
# custom flow_from_dataframe
def flow_from_dataframe(img_data_gen, in_df, path_col, y_col, **dflow_args):
    base_dir = os.path.dirname(in_df[path_col].values[0])
    print('## Ignore next message from keras, values are replaced anyways')
    df_gen = img_data_gen.flow_from_directory(base_dir, 
                                     class_mode = 'sparse',
                                    **dflow_args)
    df_gen.filenames = in_df[path_col].values
    df_gen.classes = np.stack(in_df[y_col].values)
    df_gen.samples = in_df.shape[0]
    df_gen.n = in_df.shape[0]
    df_gen._set_index_array()
    df_gen.directory = '' # since we have the full path
    print('Reinserting dataframe: {} images'.format(in_df.shape[0]))
    return df_gen

def nsl_train_generator(datagen):
    train_gen = datagen.flow_from_dataframe(dataframe=train_df,
                                         directory=None,
                                         x_col = 'newpath',
                                         y_col = 'newLabel',
                                         class_mode = 'categorical',
                                         classes = all_labels,
                                         target_size = IMG_SIZE,
                                         color_mode = 'rgb',
                                         batch_size = 64)
    for x_batch, y_batch in train_gen:
         yield {'feature': x_batch, 'label': y_batch}

def nsl_valid_generator(datagen):
    valid_gen = datagen.flow_from_dataframe(dataframe=valid_df,
                                         directory=None,
                                         x_col = 'newpath',
                                         y_col = 'newLabel',
                                         class_mode = 'categorical',
                                         classes = all_labels,
                                         target_size = IMG_SIZE,
                                         color_mode = 'rgb',
                                         batch_size = 1024) 
    for x_batch, y_batch in valid_gen:
         yield {'feature': x_batch, 'label': y_batch}

the above functions nsl_train_generator and nsl_valid_generator, I referred #3 (comment)

train_generator = nsl_train_generator(core_idg)
valid_generator = nsl_valid_generator(core_idg)
hist = adv_model.fit_generator(train_gen, 
                              steps_per_epoch=STEP_SIZE_TRAIN,
                              validation_data = valid_gen, 
                              epochs = 1,
                              callbacks = callbacks_list)

I'm getting this error,

Found 42276 validated image filenames belonging to 13 classes.
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-38-6b51ad0b4644> in <module>()
      4                               validation_data = valid_gen,
      5                               epochs = 1,
----> 6                               callbacks = callbacks_list)
      7 

15 frames
/usr/local/lib/python3.6/dist-packages/neural_structured_learning/keras/adversarial_regularization.py in <listcomp>(.0)
    633         # Converts input dictionary to a list so it conforms with the model's
    634         # expected input.
--> 635         inputs = [inputs[name] for name in base_input_names]
    636     elif not self._base_with_labels_in_features:
    637       # Removes labels and sample weights from the input dictionary, since they

KeyError: 'input_2'

NSL for Regression

I was able to integrate adversarial loss to my TensorFlow model, My input data has combination of categorical and continuous attributes.
Is there any way to specify not to perturb categorical data while creating adversarial examples?

code:
x = tf.placeholder(tf.float32,shape=(None,20))
y = tf.placeholder(tf.float32,shape=(None,1))
#build a sample model
def model(x,y,is_training):
    with tf.variable_scope("regression_model",reuse=tf.AUTO_REUSE) as scope:
        layer0 = tf.layers.Dense(units=64,activation=tf.nn.relu)(x)
        layer1 = tf.layers.Dense(units=128,activation=tf.nn.relu)(layer0)
        output = tf.layers.Dense(units=1)(layer1)
        error = tf.subtract(y,output)
        print(error)
        loss = tf.reduce_mean(error,axis=0)
    return loss
#normal loss mean absolute error 
regular_loss = model(x,y,True)
adv_config = nsl.configs.AdvRegConfig()
adv_input,adv_weights = nsl.lib.gen_adv_neighbor(x,regular_loss,config=adv_config.adv_neighbor_config)
adv_loss = model(adv_input,y,True)
overall_loss = regular_loss + 0.2*adv_loss
train_step = optim.minimize(overall_loss)
tf.random.set_random_seed(100)
sess = tf.Session()
init_op = tf.global_variables_initializer()
sess.run(init_op)
X = np.random.random((32,20))#batch of 32
Y = np.random.random((32,1))
sess.run([train_step,adv_loss,regular_loss,overall_loss],feed_dict={x:X,y:Y})

Metric computation broken when metric objects are used instead of string identifiers

Example from https://www.tensorflow.org/neural_structured_learning:

import tensorflow as tf
import neural_structured_learning as nsl

# Prepare data.
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0

# Create a base model -- sequential, functional, or subclass.
model = tf.keras.Sequential([
    tf.keras.Input((28, 28), name='feature'),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(128, activation=tf.nn.relu),
    tf.keras.layers.Dense(10, activation=tf.nn.softmax)
])

# Wrap the model with adversarial regularization.
adv_config = nsl.configs.make_adv_reg_config(multiplier=0.2, adv_step_size=0.05)
adv_model = nsl.keras.AdversarialRegularization(model, adv_config=adv_config)

# Compile, train, and evaluate.
adv_model.compile(optimizer='adam',
                  loss='sparse_categorical_crossentropy',
                  metrics=[tf.keras.metrics.SparseCategoricalAccuracy()])
adv_model.fit({'feature': x_train, 'label': y_train}, batch_size=32, epochs=5)
adv_model.evaluate({'feature': x_test, 'label': y_test})

Instead of passing 'accuracy' (string) as metric, an instance of the underlying metric class is passed to the compile function. It seems like the metric never gets updated, since the accuracy value stays at 0.0 while at the same time the loss decreases. This also happens with metrics which are not part of the default keras metric set (e.g. from the tensorflow_addons repository).

Package versions:
tensorflow-gpu 2.0.0rc0
neural-structured-learning 1.0.0

Adverserial loss should call model with training argument

With regard to custom training loops, the current implementation of the adverserial loss does not call the model with a training argument. Thus when using BatchNormalization and custom training loops this leads to incorrect behaviour, since the batch statistics are not updated for the adverserial examples. Since the adverserial loss is only useful in training mode it should be safe to always call the model with training=True.

neural-structured-learning/neural_structured_learning/keras/adversarial_regularization.py

Line 136 in aad3635

adv_output = model(adv_input)

See also:
tensorflow/tensorflow#28028
https://www.tensorflow.org/guide/migrate#customize_the_training_step

The other option is to explicitly set the learning phase in the keras backend. However, this is also discussed in the issue to be not tf-2.0 idiomatic when writing custom training loops, since it uses a hidden global state.

FB237 ids

Hello,

I have a question that I hope you can answer. Why are the downloaded files for FB237 containing entity and relation ids that are different from the ones with the original dataset? I am trying to test a model trained on the data from kbc repository with some of the original data converted to ids and they are not the same. Could you please provide information on how you created the ids?

Loss defined by string works out-of-the-box but not tf.keras.losses.X

The following works: loss='binary_crossentropy', but stops working when replaced by: tf.keras.losses.BinaryCrossentropy()

Are we expected to define weights? Or are we maybe supposed to wrap our loss functions in nsl.keras.adversarial_loss?

Edit: Can define weight key with sample_weight_key=
Edit2: I realized that it's actually the shape of values that is incorrect. This seems like a bug then?

Traceback (most recent call last):
  File "test_nsl3.py", line 86, in <module>
    main()
  File "test_nsl3.py", line 78, in main
    adv_model, train_dataset, test_dataset)
  File "test_nsl3.py", line 67, in train_and_evaluate
    model.fit(train_dataset.batch(16), epochs=200, steps_per_epoch=100)
  File "/home/richard/Documents/cloud/venv/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py", line 728, in fit
    use_multiprocessing=use_multiprocessing)
  File "/home/richard/Documents/cloud/venv/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 224, in fit
    distribution_strategy=strategy)
  File "/home/richard/Documents/cloud/venv/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 547, in _process_training_inputs
    use_multiprocessing=use_multiprocessing)
  File "/home/richard/Documents/cloud/venv/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 594, in _process_inputs
    steps=steps)
  File "/home/richard/Documents/cloud/venv/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py", line 2419, in _standardize_user_data
    all_inputs, y_input, dict_inputs = self._build_model_with_inputs(x, y)
  File "/home/richard/Documents/cloud/venv/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py", line 2622, in _build_model_with_inputs
    self._set_inputs(cast_inputs)
  File "/home/richard/Documents/cloud/venv/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py", line 2709, in _set_inputs
    outputs = self(inputs, **kwargs)
  File "/home/richard/Documents/cloud/venv/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 842, in __call__
    outputs = call_fn(cast_inputs, *args, **kwargs)
  File "/home/richard/Documents/cloud/venv/lib/python3.6/site-packages/tensorflow_core/python/autograph/impl/api.py", line 237, in wrapper
    raise e.ag_error_metadata.to_exception(e)
ValueError: in converted code:
    relative to /home/richard/Documents/cloud/venv/lib/python3.6/site-packages:

    neural_structured_learning/keras/adversarial_regularization.py:623 call  *
        adv_loss = adversarial_loss(
    neural_structured_learning/keras/adversarial_regularization.py:139 adversarial_loss  *
        adv_loss = loss_fn(labels, adv_output, adv_sample_weights)
    neural_structured_learning/keras/adversarial_regularization.py:570 _compute_total_loss  *
        loss, _ = _compute_loss_and_metrics(self._labeled_losses, None, labels,
    neural_structured_learning/keras/adversarial_regularization.py:372 _compute_loss_and_metrics  *
        loss_value = loss(label, output, sample_weights)
    neural_structured_learning/keras/adversarial_regularization.py:175 __call__  *
        loss_value = super(_LossWrapper, self).__call__(*args, **kwargs)
    tensorflow_core/python/keras/losses.py:128 __call__
        losses, sample_weight, reduction=self._get_reduction())
    tensorflow_core/python/keras/utils/losses_utils.py:107 compute_weighted_loss
        losses, sample_weight)
    tensorflow_core/python/ops/losses/util.py:148 scale_losses_by_sample_weight
        sample_weight = weights_broadcast_ops.broadcast_weights(sample_weight, losses)
    tensorflow_core/python/ops/weights_broadcast_ops.py:167 broadcast_weights
        with ops.control_dependencies((assert_broadcastable(weights, values),)):
    tensorflow_core/python/ops/weights_broadcast_ops.py:103 assert_broadcastable
        weights_rank_static, values.shape, weights.shape))

    ValueError: weights can not be broadcast to values. values.rank=0. weights.rank=2. values.shape=(). weights.shape=(None, 1).

Here is code reproducing the error:

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import neural_structured_learning as nsl
import tensorflow as tf


def prepare_datasets():
  (x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()

  def make_dataset(x, y, shuffle=False):
    y = y.reshape((-1, 28, 28, 1)).astype('float32') / 255.0
    dataset = tf.data.Dataset.from_tensor_slices((x, y))
    if shuffle:
      dataset = dataset.repeat().shuffle(100)
    return dataset.map(to_dict)

  return make_dataset(y_train, x_train, True), make_dataset(y_test, x_test)


def to_dict(x, y):
    return {'feature': x, 'label': y}


def build_base_model():
  return tf.keras.Sequential([
    tf.keras.Input(shape=(1,), dtype=tf.float32, name='feature'),
    tf.keras.layers.Dense(28*28, activation='relu'),
    tf.keras.layers.Dense(28*28, activation='sigmoid'),
    tf.keras.layers.Lambda(lambda x:  tf.reshape(x, (-1, 28, 28, 1))),
  ])


def apply_adversarial_regularization(model):
  adv_config = nsl.configs.make_adv_reg_config(
      multiplier=0.5,
      adv_step_size=0.5,
      adv_grad_norm='infinity'
  )
  return nsl.keras.AdversarialRegularization(
      model, label_keys=['label'], adv_config=adv_config)


def build_adv_model():
  base_model = build_base_model()
  return apply_adversarial_regularization(base_model)


def train_and_evaluate(model, train_dataset, test_dataset):
  model.compile(
      optimizer='adam',
      loss=tf.keras.losses.BinaryCrossentropy(),
      # loss='binary_crossentropy',
      metrics=['accuracy']
  )
  model.fit(train_dataset.batch(16), epochs=200, steps_per_epoch=100)
  eval_result = model.evaluate(test_dataset)
  return list(zip(model.metrics_names, eval_result))


def main():

  train_dataset, test_dataset = prepare_datasets()

  adv_model = build_adv_model()
  adv_result = train_and_evaluate(
      adv_model, train_dataset, test_dataset)

  for metric_name, result in adv_result:
    print('Eval %s for adversarial model: %s' % (metric_name, result))



if __name__ == '__main__':
  main()

StagingError appears for TensorFlow 2.2.0-rc3

Hi

I am primarily trying to train an adversarial variant of a custom image classifier. The code works perfectly fine when I am on TensorFlow 2.1.0, 2.2.0-rc2. But the same code breaks on TensorFlow 2.2.0-rc3 producing:

StagingError: in user code:

    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:571 train_function  *
        outputs = self.distribute_strategy.run(
    /usr/local/lib/python3.6/dist-packages/neural_structured_learning/keras/adversarial_regularization.py:627 call  *
        outputs, labeled_loss, metrics, tape = self._forward_pass(
    /usr/local/lib/python3.6/dist-packages/neural_structured_learning/keras/adversarial_regularization.py:606 _forward_pass  *
        outputs = self.base_model(inputs, **base_model_kwargs)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer.py:927 __call__  **
        outputs = call_fn(cast_inputs, *args, **kwargs)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/network.py:719 call
        convert_kwargs_to_constants=base_layer_utils.call_context().saving)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/network.py:826 _run_internal_graph
        inputs = self._flatten_to_reference_inputs(inputs)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/network.py:926 _flatten_to_reference_inputs
        return [tensors[inp._keras_history.layer.name] for inp in ref_inputs]
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/network.py:926 <listcomp>
        return [tensors[inp._keras_history.layer.name] for inp in ref_inputs]

    KeyError: 'input_1'

Here's the Colab Gist that produces this issue. Here's the Colab Notebook that shows that the code works for TensorFlow 2.2.0-rc2.

Broadcasting error when using tf.keras.losses.Loss based losses instead of functions

Modified example from the introductory example (don't punish me for using a regression loss on a classification task, it is just for demonstration):

import tensorflow as tf
import neural_structured_learning as nsl

# Prepare data.
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0

# Create a base model -- sequential, functional, or subclass.
model = tf.keras.Sequential([
    tf.keras.Input((28, 28), name='feature'),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(128, activation=tf.nn.relu),
    tf.keras.layers.Dense(1)
])

# Wrap the model with adversarial regularization.
adv_config = nsl.configs.make_adv_reg_config(multiplier=0.2, adv_step_size=0.05)
adv_model = nsl.keras.AdversarialRegularization(model, adv_config=adv_config)

# Compile, train with base model.
model.compile(optimizer='adam',
                  loss=tf.keras.losses.mae)
model.fit(x_train, y_train, batch_size=32, epochs=1)

model.compile(optimizer='adam',
                  loss=tf.keras.losses.MeanAbsoluteError())
model.fit(x_train, y_train, batch_size=32, epochs=1)

# Compile, train with adverserial model.
adv_model.compile(optimizer='adam',
                  loss=tf.keras.losses.mae)
adv_model.fit({'feature': x_train, 'label': y_train}, batch_size=32, epochs=1)

adv_model.compile(optimizer='adam',
                  loss=tf.keras.losses.MeanAbsoluteError())
adv_model.fit({'feature': x_train, 'label': y_train}, batch_size=32, epochs=1)

The first three fit invocations run fine, however, the fourth fails with a broadcasting exception:

Train on 60000 samples
2019-10-05 18:14:52.119047: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
60000/60000 [==============================] - 2s 42us/sample - loss: 1.0129
Train on 60000 samples
60000/60000 [==============================] - 2s 38us/sample - loss: 0.6782
WARNING: Logging before flag parsing goes to stderr.
W1005 18:14:59.278777 140658904659776 training_utils.py:1348] Output output_1 missing from loss dictionary. We assume this was done on purpose. The fit and evaluate APIs will not be expecting any data to be passed to output_1.
Train on 60000 samples
60000/60000 [==============================] - 3s 55us/sample - loss: 0.7647 - mean_absolute_error: 0.6109 - adversarial_loss: 0.7689
W1005 18:15:02.706085 140658904659776 training_utils.py:1348] Output output_1 missing from loss dictionary. We assume this was done on purpose. The fit and evaluate APIs will not be expecting any data to be passed to output_1.
Train on 60000 samples
   32/60000 [..............................] - ETA: 2:31Traceback (most recent call last):
  File "/tmp/nsl_bug.py", line 36, in <module>
    adv_model.fit({'feature': x_train, 'label': y_train}, batch_size=32, epochs=1)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training.py", line 728, in fit
    use_multiprocessing=use_multiprocessing)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2.py", line 324, in fit
    total_epochs=epochs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2.py", line 123, in run_one_epoch
    batch_outs = execution_function(iterator)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2_utils.py", line 86, in execution_function
    distributed_function(input_fn))
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/def_function.py", line 457, in __call__
    result = self._call(*args, **kwds)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/def_function.py", line 503, in _call
    self._initialize(args, kwds, add_initializers_to=initializer_map)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/def_function.py", line 408, in _initialize
    *args, **kwds))
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 1848, in _get_concrete_function_internal_garbage_collected
    graph_function, _, _ = self._maybe_define_function(args, kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 2150, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 2041, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/func_graph.py", line 915, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/def_function.py", line 358, in wrapped_fn
    return weak_wrapped_fn().__wrapped__(*args, **kwds)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2_utils.py", line 73, in distributed_function
    per_replica_function, args=(model, x, y, sample_weights))
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/distribute/distribute_lib.py", line 760, in experimental_run_v2
    return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/distribute/distribute_lib.py", line 1787, in call_for_each_replica
    return self._call_for_each_replica(fn, args, kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/distribute/distribute_lib.py", line 2132, in _call_for_each_replica
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/impl/api.py", line 292, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2_utils.py", line 264, in train_on_batch
    output_loss_metrics=model._output_loss_metrics)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_eager.py", line 311, in train_on_batch
    output_loss_metrics=output_loss_metrics))
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_eager.py", line 252, in _process_single_batch
    training=training))
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_eager.py", line 127, in _model_loss
    outs = model(inputs, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py", line 847, in __call__
    outputs = call_fn(cast_inputs, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/impl/api.py", line 292, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/neural_structured_learning/keras/adversarial_regularization.py", line 631, in call
    gradient_tape=tape)
  File "/usr/local/lib/python3.6/dist-packages/neural_structured_learning/keras/adversarial_regularization.py", line 139, in adversarial_loss
    adv_loss = loss_fn(labels, adv_output, adv_sample_weights)
  File "/usr/local/lib/python3.6/dist-packages/neural_structured_learning/keras/adversarial_regularization.py", line 571, in _compute_total_loss
    outputs, sample_weights)
  File "/usr/local/lib/python3.6/dist-packages/neural_structured_learning/keras/adversarial_regularization.py", line 372, in _compute_loss_and_metrics
    loss_value = loss(label, output, sample_weights)
  File "/usr/local/lib/python3.6/dist-packages/neural_structured_learning/keras/adversarial_regularization.py", line 175, in __call__
    loss_value = super(_LossWrapper, self).__call__(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/losses.py", line 128, in __call__
    losses, sample_weight, reduction=self._get_reduction())
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/utils/losses_utils.py", line 107, in compute_weighted_loss
    losses, sample_weight)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/losses/util.py", line 148, in scale_losses_by_sample_weight
    sample_weight = weights_broadcast_ops.broadcast_weights(sample_weight, losses)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/weights_broadcast_ops.py", line 167, in broadcast_weights
    with ops.control_dependencies((assert_broadcastable(weights, values),)):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/weights_broadcast_ops.py", line 103, in assert_broadcastable
    weights_rank_static, values.shape, weights.shape))
ValueError: weights can not be broadcast to values. values.rank=0. weights.rank=2. values.shape=(). weights.shape=(32, 1).

This seems to happen so far only for losses subclassed from the tf.keras.losses.Loss base class.

Error for adversarial training with Sequential model

I have a base model using tf.feature_column:

def build_base_model(hparams):
  feature_cols = [tf.feature_column.numeric_column(colname) for colname in hparams.feature_names]
  feature_layer = tf.keras.layers.DenseFeatures(feature_cols)
  l = []
  l.append(feature_layer)
  for num_units in hparams.layers:
    l.append(tf.keras.layers.Dense(num_units, activation='relu'))
    
  pred = tf.keras.layers.Dense(hparams.num_classes, activation='sigmoid')
  l.append(pred)
  model = tf.keras.Sequential(l)
  return model

And I have parser function to feed data from TFRecords (there are 500 feature columns in the file):

def parse_fn2(serialized_example, primary_keys, features, label_type):
    feature_dict = {}

    for primary_key in primary_keys:
      feature_dict[primary_key] = tf.io.FixedLenFeature([], tf.string)
    for f in features:
      feature_dict[f] = tf.io.FixedLenFeature([], tf.float32, default_value=0.0)

    if label_type == "float":
       feature_dict["label"] = tf.io.FixedLenFeature([], tf.float32, default_value=0.0)
    else:  # int
       feature_dict["label"] = tf.io.FixedLenFeature([], tf.int64, default_value=0)
      
    parsed = tf.io.parse_example(serialized_example, features=feature_dict)
    for k in primary_keys:
       primary_k = parsed.pop(k)
      
    #target = parsed.pop("label")
    return parsed #, target

def tfrecords_input_fn2(
    files_name_pattern, primary_keys, features, label_type, batch_size=1024
):
    shuffle = True if label_type else False

    file_names = tf.io.matching_files(files_name_pattern)
    dataset = tf.data.TFRecordDataset(filenames=file_names)

    if shuffle:
        dataset = dataset.shuffle(buffer_size=2 * batch_size + 1)

    dataset = dataset.batch(batch_size)
    dataset = dataset.map(
        lambda tf_example: parse_fn2(
            tf_example, primary_keys, features, label_type
        )
    )

    dataset = dataset.repeat()
    return dataset

My base model can run successfully when uncommenting the target. When I tried to run the adversarial regularization:

adv_config = nsl.configs.make_adv_reg_config(
    multiplier=HPARAMS.adv_multiplier,
    adv_step_size=HPARAMS.adv_step_size,
    adv_grad_norm=HPARAMS.adv_grad_norm
)
base_adv_model = build_base_model(HPARAMS)
adv_model = nsl.keras.AdversarialRegularization(
    base_adv_model,
    label_keys=["label"],
    adv_config=adv_config
)

adv_model.compile(optimizer='adam', loss='binary_crossentropy',
                   metrics=[tf.keras.metrics.AUC(name="auc"), 
                            tf.keras.metrics.Precision(name="precision"),
                            tf.keras.metrics.Recall(name="recall"),])

adv_model.fit(tfrecords_input_fn2(
    files_name_pattern, ["user_id"], feature_names, "integer", batch_size=1024
), epochs=100, steps_per_epoch=300)

I got error:

TypeError: in user code:

    /databricks/python/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py:571 train_function  *
        outputs = self.distribute_strategy.run(
    /databricks/python/lib/python3.7/site-packages/neural_structured_learning/keras/adversarial_regularization.py:667 call  *
        outputs, labeled_loss, metrics, tape = self._forward_pass(
    /databricks/python/lib/python3.7/site-packages/neural_structured_learning/keras/adversarial_regularization.py:646 _forward_pass  *
        outputs = self._call_base_model(inputs, **base_model_kwargs)
    /databricks/python/lib/python3.7/site-packages/neural_structured_learning/keras/adversarial_regularization.py:621 _call_base_model  *
        if (isinstance(self.base_model, tf.keras.Sequential) and

    TypeError: 'NoneType' object is not iterable

I don't understand this error. Is it because my sequential model does not build until training is running? Any suggestions for workarounds?

NSL, errors when trying to build a mlp classifier with float features

Hi all,

I am trying to run nsl on some of my own data but getting some issues reading the training data into the mlp model in my code.

The code is basically a combination of the first two tutorials.

I have my own embedding values and then use those to define a graph using your build_graph tool, running:
python nsl_repo/neural_structured_learning/tools/build_graph.py train.tfr train_graph.tsv --similarity_threshold 0.90 -v 1

I am using the embedding float values as features, 128 dimensions, and then also have labels plus ids byte value. Then, I want to try to classify
11 node types using a sequential MLP as a model.

But I keep getting the following error:

InvalidArgumentError                      Traceback (most recent call last)
~/OneDrive - AZCollaboration/projects/nsl_onBIKG/train_with_NSL.py in <module>
    161     loss='sparse_categorical_crossentropy',
    162     metrics=['accuracy'])

--> 163 base_model.fit(train_dataset, epochs=HPARAMS.train_epochs, verbose=1)
    164
    165 # def make_feed_forward_model():

/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation
_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, u
se_multiprocessing, **kwargs)
    732         max_queue_size=max_queue_size,
    733         workers=workers,
--> 734         use_multiprocessing=use_multiprocessing)
    735
    736   def evaluate(self,

/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2.py in fit(self, model, x, y, batch_size, epochs, verbose, callbacks,
validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, **kwargs)
    322                 mode=ModeKeys.TRAIN,
    323                 training_context=training_context,
--> 324                 total_epochs=epochs)
    325             cbks.make_logs(model, epoch_logs, training_result, ModeKeys.TRAIN)
    326

/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2.py in run_one_epoch(model, iterator, execution_function, dataset_size
, batch_size, strategy, steps_per_epoch, num_samples, mode, training_context, total_epochs)
    121         step=step, mode=mode, size=current_batch_size) as batch_logs:
    122       try:
--> 123         batch_outs = execution_function(iterator)
    124       except (StopIteration, errors.OutOfRangeError):
    125         # TODO(kaftan): File bug about tf function and errors.OutOfRangeError?

/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2_utils.py in execution_function(input_fn)

     84     # `numpy` translates Tensors to values in Eager mode.
     85     return nest.map_structure(_non_none_constant_value,
---> 86                               distributed_function(input_fn))
     87
     88   return execution_function

/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py in __call__(self, *args, **kwds)
    437         # Lifting succeeded, so variables are initialized and we can run the
    438         # stateless function.
--> 439         return self._stateless_fn(*args, **kwds)
    440     else:
    441       canon_args, canon_kwds = \

/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py in __call__(self, *args, **kwargs)
   1820     """Calls a graph function specialized to the inputs."""
   1821     graph_function, args, kwargs = self._maybe_define_function(args, kwargs)
-> 1822     return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
   1823
   1824   @property

/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py in _filtered_call(self, args, kwargs)
   1139          if isinstance(t, (ops.Tensor,
   1140                            resource_variable_ops.BaseResourceVariable))),
-> 1141         self.captured_inputs)
   1142
   1143   def _call_flat(self, args, captured_inputs, cancellation_manager=None):

/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py in _call_flat(self, args, captured_inputs, cancellation_manager)
   1222     if executing_eagerly:
   1223       flat_outputs = forward_function.call(
-> 1224           ctx, args, cancellation_manager=cancellation_manager)
   1225     else:
   1226       gradient_name = self._delayed_rewrite_functions.register()

/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py in call(self, ctx, args, cancellation_manager)
    509               inputs=args,
    510               attrs=("executor_type", executor_type, "config_proto", config),
--> 511               ctx=ctx)
    512         else:

    513           outputs = execute.execute_with_cancellation(

/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/eager/execute.py in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
     65     else:
     66       message = e.message
---> 67     six.raise_from(core._status_to_exception(e.code, message), None)
     68   except TypeError as e:
     69     keras_symbolic_tensors = [

/anaconda3/lib/python3.7/site-packages/six.py in raise_from(value, from_value)

InvalidArgumentError:   Feature: NL_nbr_0_embedding (data type: float) is required but could not be found.
         [[{{node ParseSingleExample/ParseSingleExample}}]]
         [[IteratorGetNext]] [Op:__inference_distributed_function_57927]

Function call stack:
distributed_function -> distributed_function

The following code generates the training and test TFRecord files:

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import random
import time

from absl import app
from absl import flags
from absl import logging
import neural_structured_learning as nsl
from neural_structured_learning.tools import graph_utils
from neural_structured_learning.tools import build_graph
from neural_structured_learning.tools import pack_nbrs
import six
import tensorflow as tf
import sys
import pandas as pd
import numpy as np
import os

# try:
#     node_f = sys.argv[1]
#     edge_f = sys.argv[2]
# except:
#     print ('.py [input_file] ')
#     sys.argv[1]

def _int64_feature(*value):
  """Returns int64 tf.train.Feature from a bool / enum / int / uint."""
  return tf.train.Feature(int64_list=tf.train.Int64List(value=list(value)))


def _bytes_feature(value):
  """Returns bytes tf.train.Feature."""
  return tf.train.Feature(
      bytes_list=tf.train.BytesList(value=[value.encode('utf-8')]))


def _float_feature(value):
  """Returns float tf.train.Feature."""
  return tf.train.Feature(float_list=tf.train.FloatList(value=value.tolist()))

#def parse_bikg_content(in_file, train_percentage):

def parse_generate_bikg_training_data():
  """Converts the bikg content (in tsv) to `tf.train.Example` instances.

  This function parses bikg content (in tsv), converts gene-disease edges into
  positive examples and other edges as negative.  This is setup for a link
  prediction task.

  randomly splits the data into training and test sets, and returns
  the training and test sets as outputs.

  Args:
    in_file: A string indicating the input file path.
    train_percentage: A float indicating the percentage of training examples
      over the dataset.

  Returns:
    train_examples: A dict with keys being example IDs (string) and values being
    `tf.train.Example` instances.
    test_examples: A dict with keys being example IDs (string) and values being
    `tf.train.Example` instances.
  """

# Fixes the random seed so the train/test split can be reproduced.
random.seed(1)
train_examples = {}
test_examples = {}

train_percentage = 0.7

df = pd.read_csv('result_embeddings.tsv', sep='\t', header=None)

# need to find translation for the labels to node types
# try to predict gene -> disease edges for this examples
node_df = pd.read_csv( 'node_data.csv', sep='\t')

id_to_type = { row[1]:row[3]  for row in node_df.values }

#use types as labels in this case
df['type'] = df[0].apply(lambda x: id_to_type[x])

#codify unique label types to index
label_to_index = {typ:i for i,typ in  enumerate(df['type'].unique()) }

for i, entry in enumerate( df.values ):
  # entries contains [ID, embedding]
  #embedding = map(float, entry[1:-1])
  features = {
      'embedding': _float_feature(np.asarray(entry[1:-1])),
      'id': _bytes_feature(str(i)),
      'label': _int64_feature(np.asarray( [ label_to_index[entry[-1]] ] ) )
  }

  #print (words)
  #print (entries)

  example_features = tf.train.Example(
      features=tf.train.Features(feature=features))
  example_id = entry[0]
  if random.uniform(0, 1) <= train_percentage:  # for train/test split.
    train_examples[example_id] = example_features
  else:
    test_examples[example_id] = example_features

# Writes 'train_examples'
with tf.io.TFRecordWriter('train.tfr') as writer:
    for example in six.itervalues(train_examples):
        writer.write(example.SerializeToString())

# Writes 'test_examples'
with tf.io.TFRecordWriter('test.tfr') as writer:
    for example in six.itervalues(test_examples):
        writer.write(example.SerializeToString())

The following code is where the NN model is defined.

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import random
import time

from absl import app
from absl import flags
from absl import logging
import neural_structured_learning as nsl
from neural_structured_learning.tools import graph_utils
from neural_structured_learning.tools import pack_nbrs
import six
import tensorflow as tf
import sys
import pandas as pd

NBR_FEATURE_PREFIX = 'NL_nbr_'
NBR_WEIGHT_SUFFIX = '_weight'

class HParams(object):
  """Hyperparameters used for training."""
  def __init__(self):
    ### dataset parameters
    self.num_classes = 11
    #self.vocab_size = 128
    self.embedding_size = 128

    ### neural graph learning parameters
    self.distance_type = nsl.configs.DistanceType.L2
    self.graph_regularization_multiplier = 0.1
    self.num_neighbors = 3
    ### model architecture
    self.num_embedding_dims = 16
    self.num_lstm_dims = 64
    self.num_fc_units = 64
    ### training parameters
    self.train_epochs = 10
    self.batch_size = 128
    self.dropout_rate = 0.5
    ### eval parameters
    self.eval_steps = None  # All instances in the test set are evaluated.

def parse_example(example_proto):
  """Extracts relevant fields from the `example_proto`.

  Args:
    example_proto: An instance of `tf.train.Example`.

  Returns:
    A pair whose first value is a dictionary containing relevant features
    and whose second value contains the ground truth labels.
  """

  # The 'embedding' feature is a fix length float vector of 128.
  feature_spec = {
      'embedding': tf.io.FixedLenFeature([HPARAMS.embedding_size], tf.float32,
                        default_value=tf.constant(
                        0, dtype=tf.float32, shape=[HPARAMS.embedding_size])),
      'label': tf.io.FixedLenFeature((), tf.int64, default_value=-1),
  }
  # We also extract corresponding neighbor features in a similar manner to
  # the features above.
  for i in range(HPARAMS.num_neighbors):
    nbr_feature_key = '{}{}_{}'.format(NBR_FEATURE_PREFIX, i, 'embedding')
    nbr_weight_key = '{}{}{}'.format(NBR_FEATURE_PREFIX, i, NBR_WEIGHT_SUFFIX)
    feature_spec[nbr_feature_key] = tf.io.FixedLenFeature([HPARAMS.embedding_size], tf.float32)

    # We assign a default value of 0.0 for the neighbor weight so that
    # graph regularization is done on samples based on their exact number
    # of neighbors. In other words, non-existent neighbors are discounted.
    feature_spec[nbr_weight_key] = tf.io.FixedLenFeature(
        [1], tf.float32, default_value=tf.constant([0.0]))

  features = tf.io.parse_single_example(example_proto, feature_spec)

  #print ('features: ', features)

  labels = features.pop('label')
  return features, labels

def make_dataset(file_path, training=False):
  """Creates a `tf.data.TFRecordDataset`.

  Args:
    file_path: Name of the file in the `.tfrecord` format containing
      `tf.train.Example` objects.
    training: Boolean indicating if we are in training mode.

  Returns:
    An instance of `tf.data.TFRecordDataset` containing the `tf.train.Example`
    objects.
  """
  dataset = tf.data.TFRecordDataset([file_path])
  if training:
    dataset = dataset.shuffle(10000)
  dataset = dataset.map(parse_example)
  dataset = dataset.batch(HPARAMS.batch_size)
  return dataset

HPARAMS = HParams()

training_samples_count = sum(1 for record in tf.data.TFRecordDataset([
                            'nsl_train_data.tfr']))

train_dataset = make_dataset('./nsl_train_data.tfr', training=True)
test_dataset = make_dataset('./test_data.tfr')

def make_mlp_sequential_model(hparams):
  """Creates a sequential multi-layer perceptron model."""
  model = tf.keras.Sequential()
  model.add(
      tf.keras.layers.InputLayer(
          input_shape=(hparams.embedding_size,), name='embedding'))
  # Input is already one-hot encoded in the integer format. We cast it to
  # floating point format here.
  model.add(
      tf.keras.layers.Lambda(lambda x: tf.keras.backend.cast(x, tf.float32)))
  for num_units in range( hparams.num_fc_units ):
    model.add(tf.keras.layers.Dense(num_units, activation='relu'))
    # For sequential models, by default, Keras ensures that the 'dropout' layer
    # is invoked only during training.
    model.add(tf.keras.layers.Dropout(hparams.dropout_rate))
  model.add(tf.keras.layers.Dense(hparams.num_classes, activation='softmax'))
  return model

base_model_tag, base_model = 'SEQUENTIAL', make_mlp_sequential_model(HPARAMS)
base_model.summary()

# Compile and train the base MLP model
base_model.compile(
    optimizer='adam',
    loss='sparse_categorical_crossentropy',
    metrics=['accuracy'])
base_model.fit(train_dataset, epochs=HPARAMS.train_epochs, verbose=1)

Potential example tutorial for Adversarial Learning

Hi @arjung creating this issue based on our conversation earlier. Open to thoughts on if this might be a valid example to also showcase besides the existing ones we have here on NSL. Folks did seem to like this one and we plan to also showcase this and some more (maybe on NLP too) in some upcoming conferences\events. This can BTW be edited and modified based on what might be appropriate \ relevant to make it presentable as per Google \ TF standards based on your insights.

Following is a notebook showcasing applying adversarial attacks (FGSM) on the FashionMNIST benchmark dataset leveraging a pre-trained model (VGG) and also how we used NSL to create a robust model and did some benchmarking on the test dataset after perturbations. This was given as a talk in the TensorFlow User Group India Summit 2020 which happened recently.

• Colab Notebook Tutorial

• GitHub Repo

• TFUG 2020 Session Video

Adversarial compilation failing to infer inputs properly

Hi.

I am currently on TensorFlow 2.3 and I am using the latest version of nsl. I am trying to train an adversarially robust flower classifier with the flowers dataset. I am preparing the data in the following way -

train_ds, validation_ds = tfds.load(
    "tf_flowers",
    split=["train[:85%]", "train[85%:]"],
    as_supervised=True
)

def preprocess_image(image, label):
    image = tf.image.convert_image_dtype(image, tf.float32)
    image = tf.image.resize(image, SIZE)
    return {"image": image, "label": label}

# Construct TensorFlow dataset
train_ds = (
    train_ds
    .map(preprocess_image, num_parallel_calls=AUTO)
    .cache()
    .shuffle(1024)
    .batch(BATCH_SIZE)
    .prefetch(buffer_size=tf.data.experimental.AUTOTUNE)
)

validation_ds = (
    validation_ds
    .map(preprocess_image, num_parallel_calls=AUTO)
    .cache()
    .batch(BATCH_SIZE)
    .prefetch(buffer_size=tf.data.experimental.AUTOTUNE)
)

My base model is constructured like so -

def get_training_model(base_model):
    inputs = Input(shape=(224, 224, 3))
    x = base_model(inputs, training=False)
    x = GlobalAveragePooling2D()(x)
    x = Dense(5)(x)
    classifier = Model(inputs=inputs, outputs=x)
    
    return classifier

base_model = MobileNetV2(weights="imagenet", include_top=False,
        input_shape=(224, 224, 3))
base_model.trainable = False
base_adv_model = get_training_model(base_model)
base_adv_model.summary()

The adversarial-regularized model is prepared in the following way -

adv_model = nsl.keras.AdversarialRegularization(
    base_adv_model,
    adv_config=adv_config
)

When I start training I run into -

StagingError: in user code:

    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:806 train_function  *
        return step_function(self, iterator)
    /usr/local/lib/python3.6/dist-packages/neural_structured_learning/keras/adversarial_regularization.py:667 call  *
        outputs, labeled_loss, metrics, tape = self._forward_pass(
    /usr/local/lib/python3.6/dist-packages/neural_structured_learning/keras/adversarial_regularization.py:646 _forward_pass  *
        outputs = self._call_base_model(inputs, **base_model_kwargs)
    /usr/local/lib/python3.6/dist-packages/neural_structured_learning/keras/adversarial_regularization.py:635 _call_base_model  *
        inputs = [inputs[name] for name in base_input_names]

    KeyError: 'input_16'

Here's my Colab Notebook. Am I missing out on something?

Cc: @csferng

tf.keras.models.load_model does not work for nsl.keras.AdversarialRegularization

Is there another intended way of saving model + optimizer?

2019-10-01 02:23:51.464271: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7219 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1070 with Max-Q Design, pci bus id: 0000:01:00.0, compute capability: 6.1)
Traceback (most recent call last):
  File "/home/richard/Documents/cloud/venv/.guild/runs/bca2e48d11674499949ff661421a60b2/.guild/sourcecode/retrain.py", line 52, in <module>
    model = load_model(config, pretrained_config)
  File "/home/richard/Documents/cloud/venv/.guild/runs/bca2e48d11674499949ff661421a60b2/.guild/sourcecode/cloud/model/load_model.py", line 32, in load_model
    model = tf.keras.models.load_model(directory + best_checkpoint_path)
  File "/home/richard/Documents/cloud/venv/lib/python3.6/site-packages/tensorflow_core/python/keras/saving/save.py", line 150, in load_model
    return saved_model_load.load(filepath, compile)
  File "/home/richard/Documents/cloud/venv/lib/python3.6/site-packages/tensorflow_core/python/keras/saving/saved_model/load.py", line 86, in load
    model = tf_load.load_internal(path, loader_cls=KerasObjectLoader)
  File "/home/richard/Documents/cloud/venv/lib/python3.6/site-packages/tensorflow_core/python/saved_model/load.py", line 541, in load_internal
    export_dir)
  File "/home/richard/Documents/cloud/venv/lib/python3.6/site-packages/tensorflow_core/python/keras/saving/saved_model/load.py", line 103, in __init__
    self._finalize()
  File "/home/richard/Documents/cloud/venv/lib/python3.6/site-packages/tensorflow_core/python/keras/saving/saved_model/load.py", line 120, in _finalize
    inputs = infer_inputs_from_restored_call_function(call_fn)
  File "/home/richard/Documents/cloud/venv/lib/python3.6/site-packages/tensorflow_core/python/keras/saving/saved_model/load.py", line 279, in infer_inputs_from_restored_call_function
    spec = nest.map_structure(common_spec, spec, spec2)
  File "/home/richard/Documents/cloud/venv/lib/python3.6/site-packages/tensorflow_core/python/util/nest.py", line 529, in map_structure
    expand_composites=expand_composites)
  File "/home/richard/Documents/cloud/venv/lib/python3.6/site-packages/tensorflow_core/python/util/nest.py", line 325, in assert_same_structure
    % (str(e), str1, str2))
ValueError: The two structures don't have the same nested structure.

First structure: type=TensorSpec str=TensorSpec(shape=(None, 256, 384, 3), dtype=tf.float32, name='feature')

Second structure: type=dict str={'feature': TensorSpec(shape=(None, 256, 384, 3), dtype=tf.float32, name='inputs/feature')}

More specifically: Substructure "type=dict str={'feature': TensorSpec(shape=(None, 256, 384, 3), dtype=tf.float32, name='inputs/feature')}" is a sequence, while substructure "type=TensorSpec str=TensorSpec(shape=(None, 256, 384, 3), dtype=tf.float32, name='feature')" is not
Entire first structure:
.
Entire second structure:
{'feature': .}

minor issue

i saw some code mistake in the tensorflow blog of nsl. how can i correct it?

Can't implement GridSearchCV

Dear,
I am trying to implement GridSearchCV on EEG-based BCI dataset.
I am unable to implement GridSearchCV may be because of your model needs input in dict and GridSearchCV in not accepting it. Any suggestion?

My Colab notebook is: https://github.com/sagihaider/EEG_Deep/blob/master/main_Single_Trial_Classification_EEGNet_NSL_2A_Grid_Search.ipynb

Examples with genomic and medical data

Hello,

I'm really excited to try this library out. Particularly, I noticed this from the article:

The graphs can come from multiple sources such as Knowledge graphs, medical records, genomic data or multimodal relations (e.g., image-text pairs).

Do you know if there is any existing examples using medical records? Thanks!

Support for ImageDataGenerators

Hi,

I am currently putting together a talk on Image Adversaries 101 that would include the following:

• Adversaries as an optimization problem
• Creating perturbations that do not hurt the eyes
• Create end-to-end image adversaries
• Neural Structured Learning for creating robust models

I have put together a basic Colab Notebook that shows how to create basic adversaries by taking inspiration from here: https://adversarial-ml-tutorial.org/introduction. I am now preparing a notebook that would show the other half - using NSL for training adversarially robust models. The notebook is available here. For the first part, it trains a basic image classification model to distinguish between different flower species (with the Flowers-17 dataset) using the ImageDataGenerator class. So, to extend that example, I am wondering if NSL would support ImageDataGenerator class or I would need to create NumPy arrays and then convert them to tf.data.Dataset.from_tensor_slices.

I am looking forward to your views and any suggestions. All of the pointers gathered from here will be open-sourced and will be made available for the world to see :)

Edit:

I was able to turn the dataset into tf.data.Dataset objects as you'd see in the Colab Notebook. I just wanted to check in and see if you would have any suggestions for the talk proposal and its structure. I would be really happy to consider it.

general question: multi-modal dataset -> consider graphs not for all modalities

Hello,

merging data and it's graph (explicitly given or created through an embedding e. g. as in the offical example Graph regularization for sentiment classification using synthesized graphs) is as easy like that

graph_reg_model = nsl.keras.GraphRegularization(base_reg_model, graph_reg_config)

However, what if I'd like to apply the graph(-induced constraints) only partially? Example:

example 1:

Here's an example. The dataset has several modalities, e. g.

• modality1 comprises features like AGE, GENDER, REGION, INCOME, DISEASE
• modality2 is a DISEASE-network (like a symmetric matrix with values according to the degree of association strength)

Speaking TF: The base_reg_model and the graph_reg_config models could be defined like in step 1. and 2.. But obviously point 3 graph_reg_model is inconsistent because base_reg_model + graph_reg_config don't know how to talk to each other.

1. base_reg_model = tf.keras.Model(inputs=modality1, outputs=outputs)
2. graph_reg_config = nsl.configs.GraphRegConfig(derived from modality2)
3. graph_reg_model = nsl.keras.GraphRegularization(base_reg_model, graph_reg_config)

example 2:

What if

• modality1 would be like AGE, GENDER, REGION, INCOME, DISEASE_1, ..., DISEASE_N
• modality2 would be a DISEASE-Network of N diseases

Is it possible (how?) to expand the graph with a dummy graph? Maybe in the way that the 4 covariates AGE, ..., INCOME are considered like in 2.2?
2.1 graph_DISEASE= nsl.configs.GraphRegConfig(derived from modality2)
2.2 graph_dummy= nsl.configs.GraphRegConfig(a fully connected, unconstraint graph with 4 nodes)
3. graph_reg_model = nsl.keras.GraphRegularization(base_reg_model, [graph_dummy, graph_DISEASE])

Thanks in advance.

'nsl' tag in Stack Overflow does not yet exist

Despite the guideline "For questions, please direct them to Stack Overflow with the "nsl" tag" in README, the said tag does not yet exist in SO:

https://stackoverflow.com/questions/tagged/nsl

and we already have the first question:

https://stackoverflow.com/questions/57823554/invalidargumenterror-cannot-update-variable-with-shape-using-a-tensor-with-s

Regardless of Epochs some dataset's returning 'graph_loss: 0.0000e+00'

Hello,
I am a bit new to the world of deep NN so there may be a simple answer to this.
I am using two custom dataset's both (aprox. 2500 samples large, shape (700,1)). I have been using a graph neural network on both. One appears to be functioning normally, however, the other consistently has a graph_loss of 0.0000e+00 regardless of number of epochs.
Is this to be expected? Does this mean that the graph for this dataset are not contributing anything?
The code more-or-less follows the https://www.tensorflow.org/neural_structured_learning/tutorials/graph_keras_lstm_imdb tutorial exactly.

fit_generator issue

Hi, I tried to modify the logic according to this suggestion but haven't managed to run training procedure successfully.

Here is the code:

import os
os.environ['TF_KERAS'] = '1'

import tensorflow as tf
from tensorflow.keras.layers import *
from tensorflow.keras.callbacks import TensorBoard, ModelCheckpoint, Callback
import dataloader
from heatmaps import *
import numpy as np
from keras_radam import RAdam
import neural_structured_learning as nsl


image_size = 200
## output shape is the same as input
n = 32 * 5
nClasses = 6
nfmp_block1 = 64
nfmp_block2 = 128
batch_size = 64

IMAGE_ORDERING = "channels_last"

model = tf.keras.Sequential([
    tf.keras.Input(shape=(image_size, image_size, 3)),
    # Encoder Block 1
    Conv2D(nfmp_block1, (3, 3), activation='relu', padding='same', name='block1_conv1', data_format=IMAGE_ORDERING),
    Conv2D(nfmp_block1, (3, 3), activation='relu', padding='same', name='block1_conv2', data_format=IMAGE_ORDERING),
    MaxPooling2D((2, 2), strides=(2, 2), name='block1_pool', data_format=IMAGE_ORDERING),
    # Encoder Block 2
    Conv2D(nfmp_block2, (3, 3), activation='relu', padding='same', name='block2_conv1', data_format=IMAGE_ORDERING),
    Conv2D(nfmp_block2, (3, 3), activation='relu', padding='same', name='block2_conv2', data_format=IMAGE_ORDERING),
    MaxPooling2D((2, 2), strides=(2, 2), name='block2_pool', data_format=IMAGE_ORDERING),
    ## bottoleneck
    (Conv2D(n, (int(image_size / 4), int(image_size / 4)),
                activation='relu', padding='same', name="bottleneck_1", data_format=IMAGE_ORDERING)),
    (Conv2D(n, (1, 1), activation='relu', padding='same', name="bottleneck_2", data_format=IMAGE_ORDERING)),
    ## Decoder Block
    ## upsampling to bring the feature map size to be the same as the input image i.e., heatmap size
    Conv2DTranspose(nClasses, kernel_size=(4, 4), strides=(4, 4), use_bias=False, name='upsample_2',
                             data_format=IMAGE_ORDERING),
    ## Reshaping is necessary to use sample_weight_mode="temporal" which assumes 3 dimensional output shape
    ## See below for the discussion of weights
    Reshape((image_size * image_size * nClasses, 1))
])

# Wrap the model with adversarial regularization.
adv_config = nsl.configs.make_adv_reg_config(multiplier=0.2, adv_step_size=0.05)
adv_model = nsl.keras.AdversarialRegularization(model, adv_config=adv_config)

radam = RAdam(total_steps=10000, warmup_proportion=0.1, min_lr=1e-5)
adv_model.compile(optimizer=radam, loss='mse', sample_weight_mode="temporal")

data_folder = 'data'
id2filename, filename2id, annotated_images = dataloader.get_image_annotations(data_folder)
df = dataloader.get_annotation_dataframe(id2filename, annotated_images)
msk = np.random.rand(len(df)) < 0.8
train = df[msk]
test = df[~msk]
encoding = MultiPointHeatmapEncoding(image_size, df, batch_size=64)

model_name = 'stacked_hourglass_adv'
log_dir = "logs/{}".format(model_name)
model_filename = "saved-models/{}.h5".format(model_name)

train_gen = encoding.generator(train, batch_size)
test_gen = encoding.generator(test, batch_size, get_weights=True)

steps_per_epoch = len(train) // batch_size
validation_steps = len(test) // batch_size
if validation_steps == 0:
    validation_steps = 1
if steps_per_epoch == 0:
    steps_per_epoch = 1

cb_tensorboard = TensorBoard(log_dir=log_dir, histogram_freq=1, write_graph=True, write_images=False)
callback_save_images = CallbackHeatmapOutput(model, get_generator(test_gen), log_dir, encoding)
checkpoint = ModelCheckpoint(model_filename, monitor='val_loss', verbose=1, save_best_only=True, mode='min')


history = adv_model.fit_generator(
            get_generator_dict(train_gen),
            validation_data=get_generator_dict(test_gen),
            steps_per_epoch=steps_per_epoch,
            epochs=5000,
            validation_steps=validation_steps,
            verbose=2,
            use_multiprocessing=True,
            callbacks=[checkpoint, callback_save_images, cb_tensorboard]
        )

before trying to use Neural Structured Learning, I used get_generator; the code is the following:

def get_generator(in_gen, should_augment=True):
    if should_augment:
        image_gen = ImageDataGenerator(fill_mode='reflect',
                                       data_format='channels_last',
                                       brightness_range=[0.5, 1.5])
    else:
        image_gen = ImageDataGenerator(fill_mode='reflect',
                                       data_format='channels_last',
                                       brightness_range=[1, 1])
    for items in in_gen:
        if len(items) == 2:
            in_x, in_y = items
        else:
            in_x, in_y, weights = items
        g_x = image_gen.flow(255 * in_x, in_y, batch_size=in_x.shape[0])
        x, y = next(g_x)
        yield x / 255.0, y

Then, according to suggestion, I modified the above generator to output the dictionary:

def get_generator_dict(in_gen, should_augment=True):
    if should_augment:
        image_gen = ImageDataGenerator(fill_mode='reflect',
                                       data_format='channels_last',
                                       brightness_range=[0.5, 1.5])
    else:
        image_gen = ImageDataGenerator(fill_mode='reflect',
                                       data_format='channels_last',
                                       brightness_range=[1, 1])
    for items in in_gen:
        if len(items) == 2:
            in_x, in_y = items
        else:
            in_x, in_y, weights = items
        g_x = image_gen.flow(255 * in_x, in_y, batch_size=in_x.shape[0])
        x, y = next(g_x)

        yield {'feature': x / 255.0, 'label': y}

Nevertheless, I get the following error:

Traceback (most recent call last):
File "/home/niko/programs/pycharm-community-2019.2.1/helpers/pydev/pydevd.py", line 2060, in
main()
File "/home/niko/programs/pycharm-community-2019.2.1/helpers/pydev/pydevd.py", line 2054, in main
globals = debugger.run(setup['file'], None, None, is_module)
File "/home/niko/programs/pycharm-community-2019.2.1/helpers/pydev/pydevd.py", line 1405, in run
return self._exec(is_module, entry_point_fn, module_name, file, globals, locals)
File "/home/niko/programs/pycharm-community-2019.2.1/helpers/pydev/pydevd.py", line 1412, in _exec
pydev_imports.execfile(file, globals, locals) # execute the script
File "/home/niko/programs/pycharm-community-2019.2.1/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/home/niko/workspace/pupil-detection-v2/fcn-adv.py", line 92, in
callbacks=[checkpoint, callback_save_images, cb_tensorboard]
File "/home/niko/workspace/advanced-computer-vision-course/venv_python3/lib/python3.5/site-packages/tensorflow_core/python/keras/engine/training.py", line 1296, in fit_generator
steps_name='steps_per_epoch')
File "/home/niko/workspace/advanced-computer-vision-course/venv_python3/lib/python3.5/site-packages/tensorflow_core/python/keras/engine/training_generator.py", line 209, in model_iteration
callbacks.on_epoch_begin(epoch, epoch_logs)
File "/home/niko/workspace/advanced-computer-vision-course/venv_python3/lib/python3.5/site-packages/tensorflow_core/python/keras/callbacks.py", line 283, in on_epoch_begin
callback.on_epoch_begin(epoch, logs)
File "/home/niko/workspace/advanced-computer-vision-course/venv_python3/lib/python3.5/site-packages/tensorflow_core/python/keras/callbacks_v1.py", line 382, in on_epoch_begin
self.model._make_test_function()
File "/home/niko/workspace/advanced-computer-vision-course/venv_python3/lib/python3.5/site-packages/tensorflow_core/python/keras/engine/training.py", line 2145, in _make_test_function
self._feed_targets +
AttributeError: 'AdversarialRegularization' object has no attribute '_feed_inputs'

Should we expect NSL to work with 2.0?

Based on #9 I'm guessing maybe we should not expect it to work? Edit: #3 suggests that 2.0 should work however.

The example code on https://www.tensorflow.org/neural_structured_learning is not working with 2.0 but it works with 1.15

Note that when running tf-nightly-gpu you get the message:
TensorFlow's tf-nightlypackage will soon be updated to TensorFlow 2.0.

Graphnet, GraphSage comparison

Hi,

Any comment on how NSL positioned with respect to DeepMind's Graphnet, or Graphsage?

On adv. robust classifiers having better transfer performance

I am referring to this paper: Do Adversarially Robust ImageNet Models Transfer Better? (Salman et al., NeurIPS'20). The authors show that adversarially robust models have better transfer learning performance. They attribute this gain to the following factors:

• Adv. robust models have better-informed gradients.
• Adv. robust models can learn representations that have better and easier invertibility.

I think it might be interesting to use nsl to train models on ImageNet-1k and verify how well these observations hold.

Will NSL work with TensorFlow 1.x ?

I can see NSL work with TensorFlow 2.0, what about the earlier version of TensorFlow, for example the TensorFlow 1.4? Thanks.

link prediction?

Hi,

I have been thinking about using NSL in the context of link prediction. It can definitely be reframed as a classification problem I believe. The only thing I am wondering about is if anyone has thought of an elegant way to add the neighbours (I guess in this case of both nodes which form the link ). Has anyone been working on this?

I guess a decent way of going about it would be changing the parse_example function:

def parse_example(example_proto):
  """Extracts relevant fields from the `example_proto`.

  Args:
    example_proto: An instance of `tf.train.Example`.

  Returns:
    A pair whose first value is a dictionary containing relevant features
    and whose second value contains the ground truth labels.
  """

so, that it can take a pair of examples which are part of a link.

Thanks,
George

TypeError: argument of type 'Tensor' is not iterable

#load datasets
X = np.load("indiana_XIMAGE_VECTORS_train.npy")
X_t = np.load("indiana_XIMAGE_VECTORS_test.npy")
Y = to_categorical(np.array(list(report_df_train["BINARY"])))
Y_t = to_categorical(np.array(list(report_df_test["BINARY"])))

# Compile, train, and evaluate.
##merge_img_classify.fit(X, Y, epochs=15, batch_size = 16, verbose=1)
merge_img_classify = build_img_model_classify(X)
# Wrap the model with adversarial regularization.
adv_config = nsl.configs.make_adv_reg_config(multiplier=0.2, adv_step_size=0.05)
adv_model = nsl.keras.AdversarialRegularization(merge_img_classify, adv_config=adv_config)
# Compile, train, and evaluate.
adv_model.compile(optimizer='adam',
                  loss='sparse_categorical_crossentropy',
                  metrics=['accuracy'])
adv_model.fit(x=X, y=Y, batch_size=32, epochs=10)
adv_model.evaluate(x=X_t, y=Y_t, verbose=1)

Epoch 1/10

TypeError Traceback (most recent call last) in () 8 loss='sparse_categorical_crossentropy', 9 metrics=['accuracy']) ---> 10 adv_model.fit(x=X, y=Y, batch_size=32, epochs=10) 11 adv_model.evaluate(x=X_t, y=Y_t, verbose=1)

TypeError: argument of type 'Tensor' is not iterable

Can not invoke 'summary' on the instance of AdversarialRegularization instance

Hi,

nsl.keras.AdversarialRegularization is a subclass of tf.keras.Model

When I invoke summary() on the instance of AdversarialRegularization I get an error saying the model needs to be built

ValueError                                Traceback (most recent call last)
<ipython-input-47-d160d5b3365c> in <module>()
      1 
----> 2 adv_model.summary()

/tensorflow-2.0.0/python3.6/tensorflow_core/python/keras/engine/network.py in summary(self, line_length, positions, print_fn)
   1255     """
   1256     if not self.built:
-> 1257       raise ValueError('This model has not yet been built. '
   1258                        'Build the model first by calling `build()` or calling '
   1259                        '`fit()` with some data, or specify '

ValueError: This model has not yet been built. Build the model first by calling `build()` or calling `fit()` with some data, or specify an `input_shape` argument in the first layer(s) for automatic build.

This is happening because even though the 'base_model' is built the wrapper one i.e. instance of AdversarialRegularization is not.

NSL vs graphical neural network?

Thanks for open sourcing NSL! Recent contribution activity looks very promising for an open source project.

I am no expert in NSL. In the past, I see multiple existing projects trying to provide building blocks for graphical neural networks:

• GraphNets
• Deep Graph Library
• Torch-Geometric
• Euler
• Torch-BigGraph

Would you mind to explain a little bit more (preferably in the project description) what is the difference between NSL and (other) graphical neural network abstractions? I believe that it would be very helpful for many users to decide which library to start with for building deep models on graphs.

An example of similarity search

I was wondering if the team would be interested to work on a tutorial to show how NSL could be used for similarity and retrieval tasks using the graph APIs provided by NSL. Would be even interesting to benchmark the performance of that against other nearest neighbors' approaches.

I personally think it would be helpful for many developers working in this area.

ImportError: cannot import name 'configs' from 'neural_structured_learning'

After installing neural_structured_learning with pip, I can successfully do:

import neural_structured_learning as nsl

But when I try:

from neural_structured_learning import configs

I get the following error:
ImportError: cannot import name 'configs' from 'neural_structured_learning'

I'm using python 3.7

gnn model implementation

Hi,

I’m interested in neural structure learning(graph neural network) and want to apply some basic gnn models such as GCN, GAT using Tenserflow2.0. I have some ideas on how to implement it, but I am not sure whether such a model architecture is clear. Here is the code structure of gnn I roughly thought:

• Create a folder named ‘models’ to design various gnn models structure
• In ‘models’ folder, create a file named sublayers.py which defines a single graph layer as a class called: GraphConvLayer, GraphAttenLayer, etc…
• Design base node & edge models to let other gnn models inherit them (increase flexibility, readability)
• Bulid functions in utils.py(or a folder named utils) such as evaluate, inference, calculate_loss, etc… to make users more convenience

About the above structure, I have slightly referred to the repository of Deep Graph Library(dgl). As mentioned before, I want to apply some gnn model application but don’t know if my idea is appropriate. My initial thought is maybe we can create a folder just called gnn-survey-paper or somewhere else to put these implementations of gnn models. This is the first time I tried to post an issue and hope to contribute to the open-source code. If there is anything unclear above or there are some recommendations from your team, just feel free to let me know. Thanks:)

Best regards,
Josh

Api could be more flexible to entertain various common scenarios

Hi,

Here are some use cases to motivate and show the limitations happening because of current api -

In classification models, it is not uncommon to use labels as inputs. These labels are passed to custom layers that use them to store information and use them as keys and/or use them to compute loss.

INPUT_IMAGE_FEATURE_NAME = 'input_img'
INPUT_LABEL_FEATURE_NAME = 'input_label'

def _map_fn(features):
  image = features["image"]
  label = features["label"]

  # cast image
  image = tf.cast(image, tf.float32)
  image = image / 255

  input_features = {INPUT_IMAGE_FEATURE_NAME: image, INPUT_LABEL_FEATURE_NAME:label}
  return input_features, label

ds_train, ds_test = tfds.load(name="mnist", split=["train", "test"])

dataset = ds_train.repeat()
dataset = dataset.shuffle(10*100)
dataset = dataset.map(_map_fn)
dataset = dataset.batch(32)


def build_base_model():
  input_img = tf.keras.Input(
        shape=(28,28,1), dtype=tf.float32, name=INPUT_IMAGE_FEATURE_NAME)
  input_label = tf.keras.Input(shape=(10), dtype=tf.int64, name=INPUT_LABEL_FEATURE_NAME)
  x = tf.keras.layers.Flatten()(input_img)
  x = tf.keras.layers.Dense(100, activation='relu')(x)
  x = MyCenterLossLayer(...)[x, input_label]
  pred = tf.keras.layers.Dense(10, activation='softmax')(x)
  model = tf.keras.Model(inputs=[input_img,input_label], outputs=pred)
  return model

Above is a pseudo code to show one such example. Indeed in my set of experiments where I try many different loss functions, there are quite many of them that end up using custom layers that require labels as input.

NSL requires dictionaries and fortunately for me I already had the setup because most of my experiments do require label as the input.

Now when I try to use AdversarialRegularization this scenario causes issues. The reason it does not work is because of following -

• My base model requires 2 inputs and _split_inputs removes the 'label'

  neural-structured-learning/neural_structured_learning/keras/adversarial_regularization.py

  Line 586 in 30e357f

  def _split_inputs(self, inputs):

i.e. only input_img is passed to the base_model whereas it expects 2 of them (i.e. input_label as well).

I know your suggestion would be to have 3 keys in the dictionary so that _split_inputs does not filter it out. This would work however passing labels is a common use case (as explained above) and passing label twice would makes data pipeline bit inefficient.

I believe (not verified) that you have enough information thanks to self.base_model in AdversarialRegularization to check which 'inputs' of the base_model that you should preserve. If it is indeed the case then instead of filtering them out you could this approach.

If you do not have the information then may be you could ask for argument that specifies which entries of the input dictionary that you should pass to the base_model.

In order to try nsl further I did following -

class PatchedAdvReg(nsl.keras.AdversarialRegularization):
  def __init__(self, base_model, label_keys, adv_config):
    super(PatchedAdvReg, self).__init__(base_model, label_keys=label_keys, adv_config=adv_config)

  def _split_inputs(self, inputs):
    sample_weights = inputs.get(self.sample_weight_key, None)
    # Labels shouldn't be perturbed when generating adversarial examples.
    labels = [
        tf.stop_gradient(inputs[label_key]) for label_key in self.label_keys
    ]
    # Removes labels and sample weights from the input dictionary, since they
    # are only used in this class and base model does not need them as inputs.
    # non_feature_keys = set(self.label_keys).union([self.sample_weight_key])
    # inputs = {
    #     key: value
    #     for key, value in six.iteritems(inputs)
    #     if key not in non_feature_keys
    # }
    return inputs, labels, sample_weights

As should be clear from above, all I am doing is to not exclude the labels as they are required by my base_model.

This made it go further however I started to get many other warnings for e.g. -

"Could not perturb input_label"

I can understand why it is displayed as now the labels are part of the input. For me this shows that it may be good idea to have a parameter in the api where the caller can specify which inputs should be perturbed. There is a possibility that there are some other inputs to the model and the user of NSL would not like to have those perturbed.

--

Finally thanks for this great work.

NSL is going to be a great library and toolkit for many use cases.

Regards & thanks
Kapil

Add __version__ (PEP 396)

This project should expose its version as nsl.__version__ to respect PEP 396.

>>> import neural_structured_learning as nsl
>>> nsl.__version__
[...]
AttributeError: module 'neural_structured_learning' has no attribute '__version__'

Maximum number of neighbours in the imdb tutorial

There is an inconsistency regarding the maximum number of neighbours to use in the tutorial. In the Augment training data with graph neighbors section, a maximum of 1 neighbour is mentioned. However, in the following run of the pack_nbrs tool the parameter --max_nbrs=3 is used. The HParams object later specifies self.num_neighbors = 2.

If I had to guess, I would say it should be 2 neighbours everywhere. Please note that this also changes the output of the pack_nbrs tool.

Thanks for sharing this library!

Issue with fit() function - problem with passing model containing multiple inputs

While using standalone Keras my model worked fine but when I try adding adversarial regularisation I run into trouble with passing the inputs (I tried many ways but nothing works).

The details of my model are irrelevant but I must say that it has two separate inputs which are joined together later. The standard Keras model which worked looked like this:
input0 = Input(shape=(50,3), name='input_A')
input1 = Input(shape=(50,30),name='input_B')
(...intermediate-layers...)
output0=GlobalMaxPooling1D(name='output')(output0)
model=Model(inputs=[input0,input1],outputs=[output0])
I successfully trained this model by passing two separate dictionaries:
model.fit({'input_A':train[i][0] ,'input_B':train[i][1]} ,{'output':train[i][-1]})
Note that the first dimension of train (i) is just used to chunk up the training into smaller sets that will fit into memory (in this case 801838 data instances). So train[0][0] is shape (801838,50,3), while train[0][1] is shape (801838,50,30).

Then I tried adding adversarial regularisation, straight from the demo:
adv_model = nsl.keras.AdversarialRegularization(model, label_keys=['label'])
But when I try running fit() there are always problems when I attempt to pass the inputs:

1. Passing the two inputs in a list does not work:
   adv_model.fit(x={'features':[train[i][0] ,train[i][1]],'label':train[i][-1]})
   ValueError: could not broadcast input array from shape (801838,50,3) into shape (801838,50)
2. A single dictionary produces the following error:
   adv_model.fit(x={'input_A':train[i][0] ,'input_B':train[i][1],'label':train[i][-1]})
   ValueError: Layer model_1 was called with an input that isn't a symbolic tensor. Received type: <type 'dict'>. Full input: [{'input_B': <tf.Tensor 'input_B_1:0' shape=(?, 50, 30) dtype=float32>, 'input_A': <tf.Tensor 'input_A_1:0' shape=(?, 50, 3) dtype=float32>}]. All inputs to the layer should be tensors.
3. I also tried a nested dictionary:
   adv_model.fit(x={'features':{'input_A':train[i][0] ,'input_B':train[i][1]},'label':train[i][-1]})
   ValueError: Cannot create a tensor proto whose content is larger than 2GB.
4. Passing in the labels separately as before:
   adv_model.fit(x={'input_A':train[i][0] ,'input_B':train[i][1]},y={'label':train[i][-1]})
   ValueError: Labels are not in the input. For predicting examples without labels, please use the base model instead.

Can anyone help me?

Unexpected memory usage for efficientnetb1?

I can train much larger models than efficientnetb1 without nsl (moreover, only the decoder is trainable here) without using much RAM. NSL's memory usage seems to be growing without end. Is this expected behaviour to use 13+ GB of RAM for this?

Edit: Using nsl.keras.AdversarialRegularization

Using this implementation: https://github.com/qubvel/segmentation_models/tree/master/segmentation_models

Edit: Uses 2.6 GB of RAM when training with a single layer Conv2D (with adverserial model) and around 4-5 GB when running efficientnetb1 without nsl

Possible for NSL to be use in Seq2seq setting?

Hi Team,

I chanced upon NSL framework and would like to say that this is an amazing work. I am currently working on some Structured Prediction task and would like to explore this framework. But I am not too clear if is is possible for this framework can be extended to say a Seq2Seq problem? Based on your experience and familiarity is it possible to do so? At the moment, is it possible for me to pack more than 1 graph and augment it with the training set?

Out of Memory issue when building large graph

I have 800k instances with 200 dimensional embeddings. I am trying to build the graph using nsl.tools.build_graph with similarity threshold of 0.95. My driver type is r4.4xlarge. I keep having OOM error. Anyone knows how to estimate how much memory I need?

About the version of tensorflow

Hi,
I had a problem running your code.You can say which version of tensorflow you use?

thkans!

node classification [blogcatalog]

Hello,
Could you please share the experimental code of classification of nodes on blogcatalog dataset. It would be very helpful to understand the NSL framework.
Thanks.

Extending Graph regularization to images?

Hi folks.

I am willing to work on a tutorial that shows how to extend graph regularization example in the same way it's done for text-based problems. Is there a scope for this tutorial inside this repo?

Experimental details

Hello,

Thank you for the great work and making the implementation publicly available [1]. I do enjoy reading findings of your work. I was wondering whether you could share your opionon on the following matters:

1- Is there any plan to share pretrained models ?

2- Could you elaborate on runtimes ?

3- What are the ranges for hyperparameters of models ?

4- How many numbers of unique hyperparameter configurations are tested ?

I fail to find enough information to repeat experiments as Section A.3 reports only the number of epochs.

Cheers

[1] Low-Dimensional Hyperbolic Knowledge Graph Embeddings https://arxiv.org/pdf/2005.00545.pdf

ValueError in the tutorial on Colab environment

Hello

The ValueError error Insufficient elements in branch_graphs[0].outputs. is found in both Neural Graph Learning tutorials (sentiment and document classification).

Both happen during the training of graph_reg_model.fit() process.
The error messages are shown below, I think it is due to the same issue.

• Graph regularization for document classification using natural graphs

Epoch 1/100
      1/Unknown - 0s 303ms/step

---------------------------------------------------------------------------

ValueError                                Traceback (most recent call last)

<ipython-input-20-0c7d19de6181> in <module>()
     10     loss='sparse_categorical_crossentropy',
     11     metrics=['accuracy'])
---> 12 graph_reg_model.fit(train_dataset, epochs=HPARAMS.train_epochs, verbose=1)

25 frames

/tensorflow-2.1.0/python3.6/tensorflow_core/python/ops/cond_v2.py in _make_indexed_slices_indices_types_match(op_type, branch_graphs)
    650                      "Expected: %i\n"
    651                      "Actual: %i" %
--> 652                      (current_index, len(branch_graphs[0].outputs)))
    653 
    654   # Cast indices with mismatching types to int64.

ValueError: Insufficient elements in branch_graphs[0].outputs.
Expected: 11
Actual: 10

• Graph regularization for sentiment classification using synthesized graphs

Epoch 1/10
      1/Unknown - 2s 2s/step

---------------------------------------------------------------------------

ValueError                                Traceback (most recent call last)

<ipython-input-30-e49eed0ffe51> in <module>()
      3     validation_data=validation_dataset,
      4     epochs=HPARAMS.train_epochs,
----> 5     verbose=1)

25 frames

/tensorflow-2.1.0/python3.6/tensorflow_core/python/ops/cond_v2.py in _make_indexed_slices_indices_types_match(op_type, branch_graphs)
    650                      "Expected: %i\n"
    651                      "Actual: %i" %
--> 652                      (current_index, len(branch_graphs[0].outputs)))
    653 
    654   # Cast indices with mismatching types to int64.

ValueError: Insufficient elements in branch_graphs[0].outputs.
Expected: 18
Actual: 17

Thanks

This line should the other way around. If weights is None then don't unpacked from the input

neural-structured-learning/neural_structured_learning/keras/layers/pairwise_distance.py

Line 114 in c21dad4

if weights is None:

Generators

Is there a particular format when combining with ImageDataGenerator?
Error:
OperatorNotAllowedInGraphError: iterating over tf.Tensor is not allowed in Graph execution

Wrong input tensor to placeholder assignment

The following code shows the introductory example with a simple modification. It uses an additional input tensor with (random) meta information, aka wide&deep model. Both input layers are named, so the normal behaviour of the keras model would be to make a name lookup and assign the values based on the key in the input dictionary. However, with the adverserial regularized model, the inputs seem to be assigned by a different order.

import numpy as np
import tensorflow as tf
import neural_structured_learning as nsl

# Prepare data.
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()
x_train, x_test = x_train[:, :, :, np.newaxis] / 255.0, x_test[:, :, :, np.newaxis] / 255.0
m_train = np.random.normal(size=(x_train.shape[0], 128))
m_test = np.random.normal(size=(x_test.shape[0], 128))

# Create a base model -- sequential, functional, or subclass.
input_image = tf.keras.Input((28, 28, 1), name='image')
input_meta = tf.keras.Input((128), name='meta')
net = tf.keras.layers.Conv2D(16, 3)(input_image)
net = tf.keras.layers.Flatten()(net)
net = tf.keras.layers.concatenate([net, input_meta])
net = tf.keras.layers.Dense(128, activation=tf.nn.relu)(net)
net = tf.keras.layers.Dense(10, activation=tf.nn.softmax)(net)
model = tf.keras.Model([input_image, input_meta], [net])  # works
model = tf.keras.Model([input_meta, input_image], [net])  # doesn't work

# Wrap the model with adversarial regularization.
adv_config = nsl.configs.make_adv_reg_config(multiplier=0.2, adv_step_size=0.05)
adv_model = nsl.keras.AdversarialRegularization(model, adv_config=adv_config)

# Compile, train, and evaluate with base model.
model.compile(optimizer='adam',
                  loss='sparse_categorical_crossentropy',
                  metrics=['accuracy'])
model.fit({'image': x_train, 'meta': m_train}, y_train, batch_size=32, epochs=1)
model.evaluate({'image': x_test, 'meta': m_test}, y_test)

# Compile, train, and evaluate with adverserial regularized model.
adv_model.compile(optimizer='adam',
                  loss='sparse_categorical_crossentropy',
                  metrics=['accuracy'])
adv_model.fit({'image': x_train, 'meta': m_train, 'label': y_train}, batch_size=32, epochs=1)
adv_model.evaluate({'image': x_test, 'meta': m_test, 'label': y_test})

Here is the error log:

Traceback (most recent call last):
  File "/tmp/nsl_bug.py", line 29, in <module>
    adv_model.fit({'image': x_train, 'meta': m_train, 'label': y_train}, batch_size=32, epochs=5)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training.py", line 728, in fit
    use_multiprocessing=use_multiprocessing)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2.py", line 224, in fit
    distribution_strategy=strategy)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2.py", line 547, in _process_training_inputs
    use_multiprocessing=use_multiprocessing)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2.py", line 594, in _process_inputs
    steps=steps)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training.py", line 2419, in _standardize_user_data
    all_inputs, y_input, dict_inputs = self._build_model_with_inputs(x, y)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training.py", line 2622, in _build_model_with_inputs
    self._set_inputs(cast_inputs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training.py", line 2709, in _set_inputs
    outputs = self(inputs, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py", line 842, in __call__
    outputs = call_fn(cast_inputs, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/impl/api.py", line 237, in wrapper
    raise e.ag_error_metadata.to_exception(e)
ValueError: in converted code:
    relative to /usr/local/lib/python3.6/dist-packages:

    neural_structured_learning/keras/adversarial_regularization.py:615 call  *
        outputs, labeled_loss, metrics, tape = self._forward_pass(
    neural_structured_learning/keras/adversarial_regularization.py:594 _forward_pass  *
        outputs = self.base_model(inputs, **base_model_kwargs)
    tensorflow_core/python/keras/engine/base_layer.py:842 __call__
        outputs = call_fn(cast_inputs, *args, **kwargs)
    tensorflow_core/python/keras/engine/network.py:708 call
        convert_kwargs_to_constants=base_layer_utils.call_context().saving)
    tensorflow_core/python/keras/engine/network.py:860 _run_internal_graph
        output_tensors = layer(computed_tensors, **kwargs)
    tensorflow_core/python/keras/engine/base_layer.py:812 __call__
        self.name)
    tensorflow_core/python/keras/engine/input_spec.py:177 assert_input_compatibility
        str(x.shape.as_list()))

    ValueError: Input 0 of layer conv2d is incompatible with the layer: expected ndim=4, found ndim=2. Full shape received: [None, 128]

As you can see, the 2d input tensor with the meta information was assigned the 4d input placeholder and which was then passed to the 2d convolutional layer.

How to fit using validation_set?

Hey I am new to NSL, I want to know how to fit the nsl model using validation set. In the traditional keras model I used to do this -

history = model.fit_generator(
    train_datagen.flow(X_train, y_train, batch_size=batch_size),
    validation_data=(X_valid, y_valid),
    steps_per_epoch=len(X_train) / batch_size,
    epochs=epochs,
    callbacks=callbacks,
    use_multiprocessing=True
)

When I passed validation_set to NSL model it throwed me error. How to do this with NSL?

I want this because my model is terribly overfitting, 99.916% accuracy on train and just 79.737% on validation test when tested individually after training is over. I think it totally learned the train data.

Also can you please give me some guidance how to tweak multiplier and adv_step_size to get better generalized model as my model is very bad as compared to base model. I am doing facial emotion recognition on three different classes(happy, sad & neutral) using the FER dataset available at kaggle.

Introducing Val_data in Fit function

Hello guys,

I was trying NSL out, but I was unable to run .fit with my validation data. What is the correct way to do it?

The following seems to be working just fine:
adv_model.fit({'feature': Xtrain_array, 'label': Ytrain_array},epochs=20)

• Xtrain_array is a np.array (shape= (20410, 70, 7, 1))
• Ytrain_array is a np.array (shape= (20410,4))

I tried converting the training and test arrays to a Dataset class using the following code but I'm getting an error "KeyError: 'feature'":

x_train=Xtrain_array
y_train=Ytrain_array
x_val=Xtest_array
y_val=Ytest_array
batch_size=32
train_data = tf.data.Dataset.from_tensor_slices({'input': x_train, 'label': y_train}).batch(batch_size)
val_data = tf.data.Dataset.from_tensor_slices({'input': x_val, 'label': y_val}).batch(batch_size)
val_steps = x_val.shape[0] / batch_size
adv_model.fit(train_data, validation_data=val_data,validation_steps=val_steps, epochs=2, verbose=1)

Massive Label Leakage in GAM/GAM* Implementation