keisen / tf-keras-vis

Neural network visualization toolkit for tf.keras

tf-keras-vis

Web documents

https://keisen.github.io/tf-keras-vis-docs/

Overview

tf-keras-vis is a visualization toolkit for debugging tf.keras.Model in Tensorflow2.0+. Currently supported methods for visualization include:

• Feature Visualization
  • ActivationMaximization (web, github)
• Class Activation Maps
  • GradCAM (paper)
  • GradCAM++ (paper)
  • ScoreCAM (paper, github)
  • Faster-ScoreCAM (github)
  • LayerCAM (paper, github) 🆕⚡
• Saliency Maps
  • Vanilla Saliency (paper)
  • SmoothGrad (paper)

tf-keras-vis is designed to be light-weight, flexible and ease of use. All visualizations have the features as follows:

• Support N-dim image inputs, that's, not only support pictures but also such as 3D images.
• Support batch wise processing, so, be able to efficiently process multiple input images.
• Support the model that have either multiple inputs or multiple outputs, or both.
• Support the mixed-precision model.

And in ActivationMaximization,

• Support Optimizers that are built to tf.keras.

Visualizations

Dense Unit

Convolutional Filter

Class Activation Map

The images above are generated by GradCAM++.

Saliency Map

The images above are generated by SmoothGrad.

Usage

ActivationMaximization (Visualizing Convolutional Filter)

import tensorflow as tf
from tensorflow.keras.applications import VGG16
from matplotlib import pyplot as plt
from tf_keras_vis.activation_maximization import ActivationMaximization
from tf_keras_vis.activation_maximization.callbacks import Progress
from tf_keras_vis.activation_maximization.input_modifiers import Jitter, Rotate2D
from tf_keras_vis.activation_maximization.regularizers import TotalVariation2D, Norm
from tf_keras_vis.utils.model_modifiers import ExtractIntermediateLayer, ReplaceToLinear
from tf_keras_vis.utils.scores import CategoricalScore

# Create the visualization instance.
# All visualization classes accept a model and model-modifier, which, for example,
#     replaces the activation of last layer to linear function so on, in constructor.
activation_maximization = \
   ActivationMaximization(VGG16(),
                          model_modifier=[ExtractIntermediateLayer('block5_conv3'),
                                          ReplaceToLinear()],
                          clone=False)

# You can use Score class to specify visualizing target you want.
# And add regularizers or input-modifiers as needed.
activations = \
   activation_maximization(CategoricalScore(FILTER_INDEX),
                           steps=200,
                           input_modifiers=[Jitter(jitter=16), Rotate2D(degree=1)],
                           regularizers=[TotalVariation2D(weight=1.0),
                                         Norm(weight=0.3, p=1)],
                           optimizer=tf.keras.optimizers.RMSprop(1.0, 0.999),
                           callbacks=[Progress()])

## Since v0.6.0, calling `astype()` is NOT necessary.
# activations = activations[0].astype(np.uint8)

# Render
plt.imshow(activations[0])

Gradcam++

import numpy as np
from matplotlib import pyplot as plt
from matplotlib import cm
from tf_keras_vis.gradcam_plus_plus import GradcamPlusPlus
from tf_keras_vis.utils.model_modifiers import ReplaceToLinear
from tf_keras_vis.utils.scores import CategoricalScore

# Create GradCAM++ object
gradcam = GradcamPlusPlus(YOUR_MODEL_INSTANCE,
                          model_modifier=ReplaceToLinear(),
                          clone=True)

# Generate cam with GradCAM++
cam = gradcam(CategoricalScore(CATEGORICAL_INDEX),
              SEED_INPUT)

## Since v0.6.0, calling `normalize()` is NOT necessary.
# cam = normalize(cam)

plt.imshow(SEED_INPUT_IMAGE)
heatmap = np.uint8(cm.jet(cam[0])[..., :3] * 255)
plt.imshow(heatmap, cmap='jet', alpha=0.5) # overlay

Please see the guides below for more details:

Getting Started Guides

• Saliency and CAMs
• Visualize Dense Layer
• Visualize Convolutional Filer

[NOTES] If you have ever used keras-vis, you may feel that tf-keras-vis is similar with keras-vis. Actually tf-keras-vis derived from keras-vis, and both provided visualization methods are almost the same. But please notice that tf-keras-vis APIs does NOT have compatibility with keras-vis.

Requirements

• Python 3.7+
• Tensorflow 2.0+

Installation

• PyPI

$ pip install tf-keras-vis tensorflow

• Source (for development)

$ git clone https://github.com/keisen/tf-keras-vis.git
$ cd tf-keras-vis
$ pip install -e .[develop] tensorflow

Use Cases

• chitra
  • A Deep Learning Computer Vision library for easy data loading, model building and model interpretation with GradCAM/GradCAM++.

Known Issues

• With InceptionV3, ActivationMaximization doesn't work well, that's, it might generate meaninglessly blur image.
• With cascading model, Gradcam and Gradcam++ don't work well, that's, it might occur some error. So we recommend to use FasterScoreCAM in this case.
• channels-first models and data is unsupported.

ToDo

• Guides
  • Visualizing multiple attention or activation images at once utilizing batch-system of model
  • Define various score functions
  • Visualizing attentions with multiple inputs models
  • Visualizing attentions with multiple outputs models
  • Advanced score functions
  • Tuning Activation Maximization
  • Visualizing attentions for N-dim image inputs
• We're going to add some methods such as below
  • Deep Dream
  • Style transfer

tf-keras-vis's Issues

Is Vanilla Saliency Implemented? I could see only smooth-grad in saliency.py

Graph disconected: custom model with two inputs

Hi, I'm getting an error when I try to get the grad-cam visualizations for a custom model with two inputs. A minimal working example follows

img_width=224
img_height=224

model = EfficientNetB0(include_top=False, input_shape=(img_width, img_height, 3),
                   weights='imagenet', drop_connect_rate=0.2)

input1 = layers.Input(shape=(img_width, img_height, 3))
input2 = layers.Input(shape=(img_width, img_height, 3))

output1 = model(input1)
output2 = model(input2)

conc = layers.Concatenate(axis=-1, name='Concat_features')([output1, output2])
# Rebuild last layer
x = layers.GlobalAveragePooling2D(name="avg_pool")(conc)
x = layers.BatchNormalization()(x)

top_dropout_rate = 0.2
x = layers.Dropout(top_dropout_rate, name="top_dropout")(x)
outputs = layers.Dense(self.num_classes, activation="softmax", name="predictions")(x)

dual_model = Model([input1, input2], outputs)
optimizer = optimizers.Adam(learning_rate=1e-4)

dual_model.compile(optimizer=optimizer, loss="categorical_crossentropy",
              metrics=["accuracy"])    

Then when I try to call to GradCam I get:

ValueError: Graph disconnected: cannot obtain value for tensor Tensor("input_1:0", shape=(None, 224, 224, 3), dtype=float32) at layer "rescaling". The following previous layers were accessed without issue: []

Any idea how can i solve this?

Anaconda recipe for tf-keras-viz

Hello and many thanks for this library!

By any chance, could you set up an Anaconda recipe for installing tf-keras-viz, as an alternative to pip? People recommend against mixing pip and conda packages in the same environment and I use conda for managing all of my deep learning libraries.

Graph disconected: transfer-learning model with top-layer

I built the transfer-learning model below.
top_layer is used to resize images for utilizing more information of images .

conv_base = MobileNetV3Small(weights='imagenet',
include_top=False,
input_shape=(224, 224, 3))

input_width=450
input_height=450
inputs = tf.keras.Input(shape=(input_height, input_width, 3))

top_layer = layers.Conv2D(filters=3, kernel_size=3, strides=1)(inputs)
top_layer = layers.MaxPool2D(pool_size=(2, 2))(top_layer)

x = conv_base(top_layer, training=False)
x = layers.GlobalAveragePooling2D()(x)
#x = layers.Dropout(0.2)(x)

outputs = layers.Dense(1)(x)
outputs = layers.Activation("sigmoid")(outputs)

model = Model(inputs, outputs)

Then when I try to call to GradCam I get:
ValueError: Graph disconnected:

I reffered this site. But I am using top_layer, so I could not do same way.
https://stackoverflow.com/questions/60623869/gradcam-with-guided-backprop-for-transfer-learning-in-tensorflow-2-0

How can I solve this? Could you give me any idea?

Could you tell me the paper wrote about visualizing dense layer and filters?

Like grad-cam or Smooth grad, Please Let me know the papers wrote about them.

tf_keras_vis.utils.normalize() can't batchwisely do normalization

Gradcam does not work with keras.layers.convolutional.Conv layers

I have the following model created and trained with Keras:

Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d (Conv2D)              (None, 28, 28, 32)        320       
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 14, 14, 32)        0         
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 14, 14, 16)        4624      
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 7, 7, 16)          0         
_________________________________________________________________
flatten (Flatten)            (None, 784)               0         
_________________________________________________________________
dropout (Dropout)            (None, 784)               0         
_________________________________________________________________
y_pred (Dense)               (None, 5)                 3925      
=================================================================
Total params: 8,869
Trainable params: 8,869
Non-trainable params: 0
_________________________________________________________________

When I print out the layers, I get the following:

<keras.layers.convolutional.Conv2D object at 0x7f30f8d9e310>
<keras.layers.pooling.MaxPooling2D object at 0x7f30f8bbfdd0>
<keras.layers.convolutional.Conv2D object at 0x7f30f8f5edd0>
<keras.layers.pooling.MaxPooling2D object at 0x7f30f8bddf50>
<keras.layers.core.Flatten object at 0x7f30f8be2b90>
<keras.layers.core.Dropout object at 0x7f30f8be8250>
<keras.layers.core.Dense object at 0x7f30f8be8bd0>

I try to run the following:

def model_modifier(cloned_model):
  cloned_model.layers[-1].activation = activations.linear
  return cloned_model


gradcam = Gradcam(model, model_modifier=model_modifier, clone=False)
cams = gradcam(score, images, seek_penultimate_conv_layer=True)

I get the following error:

ValueError: ('Unable to determine penultimate `Conv` layer. `penultimate_layer`=', -1)

From what I can tell, Gradcam is attempting to look for tensorflow.python.keras.layers.convolutional.Conv layers as opposed to keras.layers.convolutional.Conv layers (these are apparently different types of objects from what I can tell).

Even if I call:

cams = gradcam(score, images, seek_penultimate_conv_layer=False, penultimate_layer=2)

The cams variable comes back as all 0s. This does not seem right. Any help would be appreciated.

activation_maximization with YOLOv3

Hi Keisen,

Thank you very much for sharing this great work. My question is not directly related to the implementation of tf-keras-vis but on its use with tf.keras models. Currently, I am working with YOLOv3 (implemented in tf2.0. Link: [(https://github.com/zzh8829/yolov3-tf2)]). I already have trained the model and everything is working as expected. Now, I am trying to use tf-keras-vis for activation_maximization but I am struggling to understand how can I actually use it with the kind of implementation given in the link above. Would you be please be kind and help me to understand how can I use tf-keras-vis with this trained model?

I would like to share the knowledge that I have of this implementation. Starting from the network architecture shown below.

And following is the summary of the same model that I extracted with model.summary().

This implementation of YOLOv3 is using Darknet-53 as its backbone, but I did not train the backbone of the network but its head.

The loss function that has been used in this implementation is described here (https://github.com/zzh8829/yolov3-tf2/blob/66529977b2816c6b914360c720e080eed9054482/yolov3_tf2/models.py#L259). And while training, it is used in this way (https://github.com/zzh8829/yolov3-tf2/blob/66529977b2816c6b914360c720e080eed9054482/train.py#L123). Furthermore, author of this implementation also used the skip connections as described in YOLOv3 paper.

Now, as I already have trained the model, I would like to apply activation_maximization with this trained model to visualize what the conv_filters are trying to search in the input image. For this, tf-keras-vis is a great tool that you provided and can help with my ongoing research big time. I already have tried to use the examples that you provided with this repo, but with no luck. I am not an expert with this stuff but I would like to show you what I have tried so far and the error that I got.

layer_name = 'conv2d_56' 
def model_modifier(current_model):
    target_layer = current_model.get_layer(name=layer_name)
    new_model = tf.keras.Model(inputs=current_model.inputs,
                               outputs=target_layer.output)
    new_model.layers[-1].activation = tf.keras.activations.linear
    new_model.summary()
    return new_model

from tf_keras_vis.activation_maximization import ActivationMaximization
activation_maximization = ActivationMaximization(yolo_conv_0_layer,
                                                 model_modifier,
                                                 clone=False)

from yolov3_tf2.models_detect import (
    YoloV3, YoloLoss,
    yolo_anchors, yolo_anchor_masks
)
anchors = yolo_anchors
anchor_masks = yolo_anchor_masks
loss = [YoloLoss(anchors[mask], classes=9)
       for mask in anchor_masks]
from tf_keras_vis.utils.callbacks import Print
activation = activation_maximization(loss,
                                     callbacks=[Print(interval=50)])

Following is the error log that I got after executing activation = activation_maximization(loss, callbacks=[Print(interval=50)]).

---

ValueError Traceback (most recent call last)
in
9 from tf_keras_vis.utils.callbacks import Print
10 activation = activation_maximization(loss,
---> 11 callbacks=[Print(interval=50)])

f:\yolov3-tf2-master\venv\lib\site-packages\tf_keras_vis\activation_maximization.py in call(self, loss, seed_input, input_range, input_modifiers, regularizers, steps, optimizer, normalize_gradient, gradient_modifier, callbacks)
62 """
63 # losses
---> 64 losses = self._get_losses_for_multiple_outputs(loss)
65
66 # Get initial seed-inputs

f:\yolov3-tf2-master\venv\lib\site-packages\tf_keras_vis_init_.py in _get_losses_for_multiple_outputs(self, loss)
47 raise ValueError(('The model has {} outputs, '
48 'but the number of loss-functions you passed is {}.').format(
---> 49 len(self.model.outputs), len(losses)))
50 return losses
51

ValueError: The model has 4 outputs, but the number of loss-functions you passed is 3.

In my opinion, I have to initialize the model differently then it already is. But, unfortunately I can not figure out how to do it. Would you be interested to help me? As this would help a lot of students like me to use this repo with state of the art Object Detection models.

Thank you in advance for taking the time read my post!

How can I make this work with a custom top layer?

Hello. I'm new to ML and newer still to this library. I want to use the gradcam on a vgg16 architecture with custom top layers that I created. If my model in this case consists of two separate models, what would I have to change, say, in the example case to make this work?

[Request] update required libraries

By the way, in 9b3d509 you pinned scipy==1.4.* - is that necessary? Because I am now getting

plotnine 0.8.0 requires scipy>=1.5.0, but you have scipy 1.4.1 which is incompatible.

Edit: Also, pillow==7.1.* is over a year old.

Both of these old packages are not available (prebuilt) for Python 3.9, so installing 280868e on Python 3.9 fails for me due to missing "lapack/blas resources" (when compiling scipy). See scipy/scipy#9005 (comment), but I do not have admin rights on my system to install the missing libraries. Note also that TF 2.5.0rc1 will support and is prebuilt for Python 3.9.

Originally posted by @bersbersbers in #39 (comment)

Allow passing arguments to model calls

Currently, the codebase does not allow to pass arguments to model calls, e.g.

tf-keras-vis/tf_keras_vis/gradcam.py

Line 58 in fdfb61b

outputs = model(seed_inputs)

This means one cannot influence whether a model is called in inference or training mode, as e.g. data augmentation layers are on by default (https://github.com/tensorflow/tensorflow/blob/v2.3.0/tensorflow/python/keras/layers/preprocessing/image_preprocessing.py#L396) and global setting of learning phase is deprecated (https://www.tensorflow.org/api_docs/python/tf/keras/backend/set_learning_phase). I suggest the training mode be passed as a parameter of e.g. Gradcam.call.

ActivationMaximization fails after tf.keras.models.load_model

hello,

I've manged to execute the Visualize Dense Layer example.
then tried the following change :

OriginalModel = Model(weights='imagenet', include_top=True)
OriginalModel.summary()
OriginalModel.save('SavedVgg',include_optimizer=True)
model = tf.keras.models.load_model('SavedVgg')
activation_maximization = ActivationMaximization(model, model_modifier, clone=False)

(model is saved in tensorflow checkpoint format )

this fails at runtime
File "DenseLayerVisualization2.py", line 60, in
main()
File "DenseLayerVisualization2.py", line 46, in main
activation = activation_maximization(loss,callbacks=[Print(interval=50)])
File "/usr/local/lib/python3.6/dist-packages/tf_keras_vis/activation_maximization.py", line 92, in call
outputs = self.model(seed_inputs, training=training)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py", line 891, in call
outputs = self.call(cast_inputs, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/saving/saved_model/utils.py", line 57, in return_outputs_and_add_losses
outputs, losses = fn(inputs, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/saving/saved_model/utils.py", line 111, in wrap_with_training_arg
lambda: replace_training_and_call(False))
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/utils/tf_utils.py", line 59, in smart_cond
pred, true_fn=true_fn, false_fn=false_fn, name=name)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/smart_cond.py", line 56, in smart_cond
return false_fn()
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/saving/saved_model/utils.py", line 111, in
lambda: replace_training_and_call(False))
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/saving/saved_model/utils.py", line 106, in replace_training_and_call
return wrapped_call(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/def_function.py", line 457, in call
result = self._call(*args, **kwds)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/def_function.py", line 494, in _call
results = self._stateful_fn(*args, **kwds)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 1822, in call
graph_function, args, kwargs = self._maybe_define_function(args, kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 2150, in _maybe_define_function
graph_function = self._create_graph_function(args, kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 2041, in _create_graph_function
capture_by_value=self._capture_by_value),
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/func_graph.py", line 915, in func_graph_from_py_func
func_outputs = python_func(*func_args, **func_kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/def_function.py", line 358, in wrapped_fn
return weak_wrapped_fn().wrapped(*args, **kwds)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/saved_model/function_deserialization.py", line 262, in restored_function_body
"\n\n".join(signature_descriptions)))
ValueError: Could not find matching function to call loaded from the SavedModel. Got:
Positional arguments (3 total):
* [<tf.Variable 'Variable:0' shape=(1, 224, 224, 3) dtype=float32>]
* False
* None
Keyword arguments: {}

Expected these arguments to match one of the following 4 option(s):.....

Can anyone help ?

thanks,
Omer.

Feature request: support mixed precision for GradCam(PlusPlus)

This short example shows that GradCam(PlusPlus) fails when the network is trained with mixed precision:

import numpy as np
import tensorflow.keras as keras
from tensorflow.keras.mixed_precision import experimental as mixed_precision
from tf_keras_vis.gradcam import Gradcam, GradcamPlusPlus
from tf_keras_vis.saliency import Saliency

policy = mixed_precision.Policy("mixed_float16")
mixed_precision.set_policy(policy)

model = keras.applications.MobileNet()
input_data = np.empty(model.input.shape[1:])


def model_modifier(model):
    model.layers[-1].activation = keras.activations.linear
    return model


def loss(output):
    return [o[0] for o in output]


saliency = Saliency(model, model_modifier, clone=False)

# works
saliency(loss, input_data)

# works
saliency(loss, input_data, smooth_samples=20)

# does not work: "array type dtype('float16') not supported"
gradcam = Gradcam(model, model_modifier, clone=False)
gradcam(loss, input_data, penultimate_layer=-1)

# does not work: "array type dtype('float16') not supported"
gradcampp = GradcamPlusPlus(model, model_modifier, clone=False)
gradcampp(loss, input_data, penultimate_layer=-1)

It works fine when you comment out mixed_precision.set_policy(policy).

Visualizing Effects from Previous Layers

Hello. I have already starred your package. It seems very usefull and accurate. I have a question, is it possible to select which layer of the model we want to visualize these explainability effects? Instead of just using the last one. This would be helpfull to understand what is learning each layer, e.g. layers with a lower number of filters tend to learn bigger features an viceversa. Does this make sense for you?

Thanks a lot and kind regards.
Borja

[Enhancement] Introduce express API

Thank you for maintaining ideas of keras-vis 🙏

I'm currently trying to inspect my CNN and it seems like here is the minimal amount of code it's needed to plot feature maps:

It seems to be possible to shortcut. I propose to introduce an express API for solving the most frequent use cases like:

• plot all/selected feature maps of a convolution layer
• plot all/selected filters of a convolution layer
• plot dense layer activation
• plot a class activation
• plot silency map

Sklearn and Yellowbrick have great examples of express API:

• https://github.com/scikit-learn/scikit-learn/blob/4afd4fba6/sklearn/metrics/_plot/confusion_matrix.py#L165
• https://www.scikit-yb.org/en/latest/oneliners.html

This would dramatically reduce learning curve and increase adoption of the library.

Write some user guides

• Visualizing multiple attention or activation images at once utilizing batch-system of model
• Define various loss functions
• Visualizing attentions with multiple inputs models
• Visualizing attentions with multiple outputs models
• Advanced loss functions
• Tuning Activation Maximization
• Visualizing attentions for N-dim image inputs

loss function for different labels

Dear Keisen,

Thank you for sharing with us this nice project. I really like it.

In attentions.ipynb, cell 4,

# Define loss function. 20 is the imagenet index corresponding to ouzel.
loss = lambda output: K.mean(output[:, 20])

This loss function is only for images with the same class (20). If I want to use images with different classes (for example, the first image is class 20, the second image is class 21), how should I modify the code?

Thank you for your help.

Best Wishes,

Alex

Error when using custom convolutional layer

Hello,

I have a CNN with custom convolutional layers and the name of those layers are not instances of tensorflow.python.keras.layers.convolutional

Therefore, the function _find_penultimate_output always returns None and raises en Error. I had to comment out a few lines in _find_penultimate_output to be able to obtain a grad-CAM, those lines:

if layer is not None:
            layer = find_layer(model, lambda l: isinstance(l, Conv), offset=layer)

Not a clean solution but it can help if someone else has this issue, and would be great if you could release an official fix for custom convolutional layers.

And by the way, would also be more than great to have an option to return the non-interpolated grad-CAM as for custom convolutional layers the interpolation may be special.

Thanks so much !

Implement Loss functions for Imagenet models

We need some basic loss libraries to be ease to use.

Implement KerasCallback for Attentions

https://www.tensorflow.org/guide/keras/custom_callback

Multi-GPU not working when using model(inputs) nor when computing the gradients

Hi,

I've been using tf-keras-vis for a while and wanted to speed things up by using more than one GPU to compute the saliency maps of multiple images. I noticed that when using the interpretability methods (the ones that use tape.gradients and model(inputs)) only one GPU is being used.

I don't know if this is normal behavior, a bug on TensorFlow's part, or a problem on my end, but I decided to open an issue, maybe someone has tried it.

Here are a couple of useful links:

• Stackoverflow: I opened this issue on StackOverflow and added a bounty to it. You can find a list of things that I've tried and also more information on this issue.
• Working Example: A trained CNN on the dogs_vs_cats dataset. This replicates the problem of not being able to distribute the performance between the GPUs. You can also download the models in HDF5 and Saved formats.

GradCam for custom models

I'm trying to use gradcam for my custom model with binary class output

I'm getting the following error, please let me know how to fix this:

---> 25     cam = gradcam(lambda x: x, ds[weights])
     26     cam = normalize(cam)
     27     print(cam.shape)

~/miniconda/envs/bioexp/lib/python3.6/site-packages/tf_keras_vis/gradcam.py in __call__(self, loss, seed_input, penultimate_layer, seek_penultimate_layer, activation_modifier, normalize_gradient)
     57             loss_values = [loss(y) for y, loss in zip(outputs[:-1], losses)]
     58             penultimate_outputs = outputs[-1]
---> 59         grads = tape.gradient(loss_values, penultimate_outputs)
     60         if normalize_gradient:
     61             grads = K.l2_normalize(grads)

~/miniconda/envs/bioexp/lib/python3.6/site-packages/tensorflow_core/python/eager/backprop.py in gradient(self, target, sources, output_gradients, unconnected_gradients)
   1013         output_gradients=output_gradients,
   1014         sources_raw=flat_sources_raw,
-> 1015         unconnected_gradients=unconnected_gradients)
   1016 
   1017     if not self._persistent:

~/miniconda/envs/bioexp/lib/python3.6/site-packages/tensorflow_core/python/eager/imperative_grad.py in imperative_grad(tape, target, sources, output_gradients, sources_raw, unconnected_gradients)
     74       output_gradients,
     75       sources_raw,
---> 76       compat.as_str(unconnected_gradients.value))

~/miniconda/envs/bioexp/lib/python3.6/site-packages/tensorflow_core/python/eager/backprop.py in _gradient_function(op_name, attr_tuple, num_inputs, inputs, outputs, out_grads, skip_input_indices)
    136     return [None] * num_inputs
    137 
--> 138   return grad_fn(mock_op, *out_grads)
    139 
    140 

~/miniconda/envs/bioexp/lib/python3.6/site-packages/tensorflow_core/python/ops/control_flow_grad.py in _SwitchGrad(op, *grad)
     42   graph = ops.get_default_graph()
     43   # pylint: disable=protected-access
---> 44   op_ctxt = op._get_control_flow_context()
     45   grad_ctxt = graph._get_control_flow_context()
     46   # pylint: enable=protected-access

~/miniconda/envs/bioexp/lib/python3.6/site-packages/tensorflow_core/python/eager/backprop.py in _get_control_flow_context(self)
    107   def _get_control_flow_context(self):
    108     raise NotImplementedError(
--> 109         "tf.GradientTape.gradients() does not support graph control flow "
    110         "operations like tf.cond or tf.while at this time. Use tf.gradients() "
    111         "instead. If you need this feature, please file a feature request at "

NotImplementedError: tf.GradientTape.gradients() does not support graph control flow operations like tf.cond or tf.while at this time. Use tf.gradients() instead. If you need this feature, please file a feature request at https://github.com/tensorflow/tensorflow/issues/new

Thank you
Avinash

Implement Score-CAM

https://arxiv.org/abs/1910.01279

How to select an specific output from a Multi Task Learning Model?

Hello Keisen,

First I'd like to thank you for sharing this project. I've already used the former keras-vis and I found your project exactly when I was developing a new code for TF 2.1.

I built a multi task model with 3 output layers and I'm not sure how to select each of this outputs before running the saliency and normalize methods.

Here is the outputs of my model (obtained via model.outputs command). The first and third outputs use binary_crossentropy loss while the second uses MSE loss.

[<tf.Tensor 'sex_3/Identity:0' shape=(None, 1) dtype=float32>,
 <tf.Tensor 'age_3/Identity:0' shape=(None, 1) dtype=float32>,
 <tf.Tensor 'autism_3/Identity:0' shape=(None, 1) dtype=float32>]

I had success running the code bellow (from your examples). But I failed when I needed to select a different output and loss than the last one.

def model_modifier(m):
    m.layers[-1].activation = tf.keras.activations.linear
saliency = Saliency(model, model_modifier)
loss = lambda output: K.mean(output[:, 0])

Thanks in advance!

Implement Gradcam++

https://arxiv.org/abs/1710.11063

Regression Task Scoring

Hi all, I'm currently wondering if there's a way to get Saliency maps on a regression task I'm attempting to visualize. I assumed that the "InactiveScore" class would be the way to go, but I get ValueErrors as such:

  File "train.nn.PYTHON3.py", line 108, in <module>
    sal_map = viz.gen_saliency(model=model, X=xtest_untransposed, score=score)
  File "/overflow/dschridelab/users/lswhiteh/saliency/pop_gen_cnn/theta/viz.py", line 13, in gen_saliency
    return saliency(score, X)  # , smooth_samples=20, smooth_noise=0.2)
  File "/nas/longleaf/home/lswhiteh/.conda/envs/saliency/lib/python3.8/site-packages/tf_keras_vis/saliency.py", line 109, in __call__
    grads = self._get_gradients(seed_inputs, scores, gradient_modifier, training,
  File "/nas/longleaf/home/lswhiteh/.conda/envs/saliency/lib/python3.8/site-packages/tf_keras_vis/saliency.py", line 131, in _get_gradients
    grads = [gradient_modifier(g) for g in grads]
  File "/nas/longleaf/home/lswhiteh/.conda/envs/saliency/lib/python3.8/site-packages/tf_keras_vis/saliency.py", line 131, in <listcomp>
    grads = [gradient_modifier(g) for g in grads]
  File "/nas/longleaf/home/lswhiteh/.conda/envs/saliency/lib/python3.8/site-packages/tf_keras_vis/saliency.py", line 30, in <lambda>
    gradient_modifier=lambda grads: K.abs(grads),
  File "/nas/longleaf/home/lswhiteh/.conda/envs/saliency/lib/python3.8/site-packages/tensorflow/python/util/dispatch.py", line 201, in wrapper
    return target(*args, **kwargs)
  File "/nas/longleaf/home/lswhiteh/.conda/envs/saliency/lib/python3.8/site-packages/tensorflow/python/keras/backend.py", line 2440, in abs
    return math_ops.abs(x)
  File "/nas/longleaf/home/lswhiteh/.conda/envs/saliency/lib/python3.8/site-packages/tensorflow/python/util/dispatch.py", line 201, in wrapper
    return target(*args, **kwargs)
  File "/nas/longleaf/home/lswhiteh/.conda/envs/saliency/lib/python3.8/site-packages/tensorflow/python/ops/math_ops.py", line 398, in abs
    x = ops.convert_to_tensor(x, name="x")
  File "/nas/longleaf/home/lswhiteh/.conda/envs/saliency/lib/python3.8/site-packages/tensorflow/python/profiler/trace.py", line 163, in wrapped
    return func(*args, **kwargs)
  File "/nas/longleaf/home/lswhiteh/.conda/envs/saliency/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 1540, in convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/nas/longleaf/home/lswhiteh/.conda/envs/saliency/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py", line 339, in _constant_tensor_conversion_function
    return constant(v, dtype=dtype, name=name)
  File "/nas/longleaf/home/lswhiteh/.conda/envs/saliency/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py", line 264, in constant
    return _constant_impl(value, dtype, shape, name, verify_shape=False,
  File "/nas/longleaf/home/lswhiteh/.conda/envs/saliency/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py", line 276, in _constant_impl
    return _constant_eager_impl(ctx, value, dtype, shape, verify_shape)
  File "/nas/longleaf/home/lswhiteh/.conda/envs/saliency/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py", line 301, in _constant_eager_impl
    t = convert_to_eager_tensor(value, ctx, dtype)
  File "/nas/longleaf/home/lswhiteh/.conda/envs/saliency/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py", line 98, in convert_to_eager_tensor
    return ops.EagerTensor(value, ctx.device_name, dtype)
ValueError: Attempt to convert a value (None) with an unsupported type (<class 'NoneType'>) to a Tensor. ```

I should mention that I'm also trying to do this with a 1DCNN architecture, maybe that's also causing issues?

Any help would be great. Thanks

EfficientNet GradCAM doesn't work

Could you advice on how to plot GradCAM for EfficientNet. The current strategy you mentioned on the tutorials won't work, specially my last activation is a sigmoid.

Model Visualization with Chitra

Check out Chitra which uses tf-keras-vis for GradCAM visualization.

You can train CNNs in few lines and also interpret your model with InterpretModel class.

have been unable to visualize dense layer on graphs/dataSets

hello,

i've managed to execute the visualize sense layer example,
but I have not manged to reproduce results using other graphs (like the current tensorflow classification tutorial graph ) on my dataset (9 classes , image size 50X50 , 2400 training , 100 validation).

training converges, I get decent results ( for example loss: 0.0714 - sparse_categorical_accuracy: 0.9766 - val_loss: 0.6758 - val_sparse_categorical_accuracy: 0.8304) ,

and yet the visuals generated by activation_maximization are meaningless ( attached a few examples )

( all images contain apples , I expected to see round shapes )

( I've tried both with and without image preprocessing/normalization ( i.e dividing input pixel data by 255 ) ,
I've tried visualizing the results directly after fitting ( without saving/loading the graph),
I've tried modifying the tutorial graph in several different ways )

can someone point me in the right direction ?

thanks
Omer.

GradCAM can't return non-interpolated cam values

It could be useful in some very specific cases to return the grad-CAM vectors before they undergo the interpolation step (Line 68 of gradcam.py).

Originally posted by @pir127 in #4 (comment)

Scipy.ndimage.interpolation raises exception on rotate in utils.input_modifiers

Hi there, thank you for this great library!

I've stepped over a problem on trying to "Visualizing Dense layer using ActivationMaximization" as in your perfect example.

I updated everything and the example worked as expected. When I run with my own fully dense model I get following error (model summary in front, exact same for the other example with conv!):

Model: "sequential_4"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
flatten_4 (Flatten)          (None, 128000)            0         
_________________________________________________________________
dense_16 (Dense)             (None, 256)               32768256  
_________________________________________________________________
dropout_12 (Dropout)         (None, 256)               0         
_________________________________________________________________
dense_17 (Dense)             (None, 128)               32896     
_________________________________________________________________
dropout_13 (Dropout)         (None, 128)               0         
_________________________________________________________________
dense_18 (Dense)             (None, 64)                8256      
_________________________________________________________________
dropout_14 (Dropout)         (None, 64)                0         
_________________________________________________________________
dense_19 (Dense)             (None, 2)                 130       
=================================================================
Total params: 32,809,538
Trainable params: 32,809,538
Non-trainable params: 0
_________________________________________________________________
Traceback (most recent call last):
  File "explain_dense.py", line 47, in <module>
    activations = activation_maximization(score, steps=256, callbacks=[PrintLogger(interval=50)])
  File "/home/oyster/.local/lib/python3.8/site-packages/tf_keras_vis/activation_maximization/__init__.py", line 111, in __call__
    seed_inputs[j] = modifier(seed_inputs[j])
  File "/home/oyster/.local/lib/python3.8/site-packages/tf_keras_vis/utils/input_modifiers.py", line 56, in __call__
    seed_input = rotate(seed_input,
  File "/home/oyster/.local/lib/python3.8/site-packages/scipy/ndimage/interpolation.py", line 686, in rotate
    raise ValueError('axes should contain exactly two values')
ValueError: axes should contain exactly two values

I debugged through the code but couldn't find any clues yet and would like to help to find out if this is a bug or a setup problem.
Is anything similar known? Any suggestions where to look?

Thanks!

Is it really N-dim ?

Hey, thanks for the great repository.
I have a NN that needs a 4 dimensional input because it uses a ConvLSTM2D, so the input shape is (Time,Row,Cols,Channels).
However, when using the activation maximization feature, it leads to a TF error : 'images' must be either 3 or 4-dimensional. I guess another dimension is added for the batch size of 1.
Can you please help me figure out how to fix this issue ?

Note : using the activation maximization for a 3 dimensional input (other model without ConvLSTM2D) works perfectly.

Problem in grads

Hi,

First of all, thank you for developing this library! I am trying to use the GradcamPlusPlus for a custom model with two inputs (image and a 1D-array with either a 1 or a 0 based on gender). The output is one single value (regression).

I experimented using only one instance:

X is a list of two numpy arrays (image input and gender input)

def loss(output):
return (output-ground_truth_value)

gradcam = GradcamPlusPlus(model)
cam = gradcam(loss, X, penultimate_layer=21) (This layer is the last convolutional layer before flattening and concatenating with gender input).

I get an error when grads.ndim is called, saying: 'Tensor' object has no attribute 'ndim'.
So for some reason 'grads' is a tensor where it should not be the case.

Could someone help me out with this one?

Thanks!

Graph disconected: transfer-learning model with custom dense layers

Hi, I'm getting an error when I try to get the grad-cam visualizations for a custom model as given below.

Then when I try to call to GradCam I get:
ValueError: Graph disconnected: cannot obtain value for tensor Tensor("input_1:0", shape=(None, 224, 224, 3), dtype=float32) at layer "rescaling". The following previous layers were accessed without issue: []

Python 3.8 support

user@host:~/pip_patches> python --version
Python 3.8.2
user@host:~/pip_patches> pip install tf-keras-vis==0.2.2
ERROR: Could not find a version that satisfies the requirement tf-keras-vis==0.2.2 (from versions: 0.0.1, 0.1.0)
ERROR: No matching distribution found for tf-keras-vis==0.2.2

Support for LayerCAM

Hi, @keisen ,
Our paper "LayerCAM: Exploring Hierarchical Class Activation Maps for Localization" is accepted by TIP recently, which can visualize the class activation maps from any CNN layer of an off-the-shelf network. Could you add our method to your popular repository for more people to try this method? Our method is a simple modification of Grad-CAM. It should easy to implement. Here is the paper and code. Hope for your reply.

Examples: ScoreCAM should not use the linear-activation model_modifier

https://arxiv.org/pdf/1910.01279.pdf says quite explicitly:

The relative output value (postsoftmax) after normalization is more reasonable to measure the relevance than absolute output value (pre-softmax). Thus, in Score-CAM, we represent weight as post-softmax value, so that the score can be rescaled into a fixed range. [...] Score-CAM with softmax can well distinguish two different categories, even though the prediction probability of ‘cat’ is lower than the probability of ‘dog’. Normalization operation equips Score-CAM with good class discrimination ability

So I'd say in https://github.com/keisen/tf-keras-vis/blob/master/examples/attentions.ipynb, don't suggest use of model_modifier in [11] and [12].

See also algorithm 1, third line from the bottom, which may or may not imply another softmax step.

Data shape for saliency map

First of all, thank you so much for sharing this package. I am trying to create a saliency map, but have been unable to, and I am hoping that you might be able to offer some advice.

Instead of using images as the input for classification, I am using 2-dimensional arrays of numbers. To create salience maps for 3 samples, my input data (X) with has the shape: (3, 1, 33, 128), where 3 is the number of samples, 33 is the number of rows, and 128 is the number of columns, because that is the necessary shape of the input data during model training. When I create a saliency map with the attentions example code and run print(saliency_map.shape) it shows (3, 1, 33), so it looks like the column information is lost. If I instead first transpose the input shape to (3, 33, 128, 1) to match the shape of the image data in the attentions example, I get the following error:

NotFoundError` Traceback (most recent call last)
<ipython-input-172-58f751d2ff4e> in <module>()
     20 from tf_keras_vis.saliency import Saliency
     21 from tf_keras_vis.utils import normalize
---> 22 saliency_map = saliency(loss, X_train)
     23 saliency_map = normalize(saliency_map)
13 frames
/usr/local/lib/python3.6/dist-packages/six.py in raise_from(value, from_value)
NotFoundError: No algorithm worked! `[Op:Conv2D]

My code is as follows:

from tensorflow.keras import backend as K
from tf_keras_vis.saliency import Saliency
from tf_keras_vis.utils import normalize
import tensorflow as tf
import keras

from tensorflow.keras.models import load_model
model = load_model('my_model.h5')

model.summary()

def model_modifier(m):
    m.layers[-1].activation = tf.keras.activations.linear
    return m

def loss(output):
    return (output[0][0], output[1][0], output[2][0])

saliency = Saliency(model,
                model_modifier=model_modifier)
from tensorflow.keras import backend as K
from tf_keras_vis.saliency import Saliency
from tf_keras_vis.utils import normalize
saliency_map = saliency(loss, X_train)
saliency_map = normalize(saliency_map)

And my model, using model.summary(), looks like:

Layer (type)                 Output Shape              Param #   
=================================================================
input_3 (InputLayer)         [(None, 1, 33, 128)]      0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 8, 33, 128)        512       
_________________________________________________________________
batch_normalization_6 (Batch (None, 8, 33, 128)        32        
_________________________________________________________________
depthwise_conv2d_2 (Depthwis (None, 16, 1, 128)        528       
_________________________________________________________________
batch_normalization_7 (Batch (None, 16, 1, 128)        64        
_________________________________________________________________
activation_4 (Activation)    (None, 16, 1, 128)        0         
_________________________________________________________________
average_pooling2d_4 (Average (None, 16, 1, 32)         0         
_________________________________________________________________
dropout_4 (Dropout)          (None, 16, 1, 32)         0         
_________________________________________________________________
separable_conv2d_2 (Separabl (None, 16, 1, 32)         512       
_________________________________________________________________
batch_normalization_8 (Batch (None, 16, 1, 32)         64        
_________________________________________________________________
activation_5 (Activation)    (None, 16, 1, 32)         0         
_________________________________________________________________
average_pooling2d_5 (Average (None, 16, 1, 4)          0         
_________________________________________________________________
dropout_5 (Dropout)          (None, 16, 1, 4)          0         
_________________________________________________________________
flatten (Flatten)            (None, 64)                0         
_________________________________________________________________
dense (Dense)                (None, 2)                 130       
_________________________________________________________________
softmax (Activation)         (None, 2)                 0         
=================================================================
Total params: 1,842
Trainable params: 1,762
Non-trainable params: 80

ResNet-50 visualization

Hi,

I used fine-tuned resnet-50 and added a new classifier on top. When I try to visualize with CAM methods, I get meaningless heatmaps. On the other hand, when visualizing with gradient, there is an attention in regions that should be highlighted. I know ResNet is hard to visualize due to its skip connections, but do you have any tips on what I could do to improve CAM visualizations, or if there is any good library that could help me (I tried keras-vis same outcome).

Thanks,
Tarik

'keras.backend' has no attribute 'identity'

Multi-output will only pass the first output to the score funuction

tf-keras-vis/tf_keras_vis/gradcam.py

Line 62 in ddd9513

loss_values = [loss(y) for y, loss in zip(outputs, losses)]

Say if I have an output of [tf.tensor , tf.tensor], after this line executed even I use my own scoring function, the obtained "output" is now only the first tf.tensor.

I don't think this is the expected behavior...

Sample model: https://github.com/duckietown/challenge-aido_LF-baseline-behavior-cloning/blob/master/duckieChallenger/frankModel.py

What is different between the models?

Hello, I have a suggestion for you. Since you have implement the methods so you understand very well the ideas behind these methods I would recommend you to explain what are the main difference between the methods on the examples notebook. Just a short detail, that would very great.

Thanks and kind regards,

Borja.

problem after updating to tensorflow 2.5 and keras 2.4 - Expected `model` argument

Hi,

just after updating tensorflow to 2.5.0 and keras to 2.4.3, I can not use tf_keras_vis anymore, when I pass my model to functions I get
('Expected modelargument to be aModel instance, got ', <keras.engine.functional.Functional object at 0x00000230BFDB2AC0>)
I also upgraded the tf-keras-vis package, but still I get the same error.

did anyone else face the same thing?

GradCam Issues with MobileNetV3

Hi @keisen, first off, thank you all so much for the amazing library.

I was recently using GradCam++ with MobileNetV3 and found that the heatmaps created were entirely blue despite the predictions being accurate. I am currently reading through the relevant literature to figure out whether it is a feature of the MobileNetV3 architecture or some lack of compatibility with the visualization code. I was just wondering if you guys would have any insight to perhaps resolve the issue more quickly.

Here is some code to recreate the issue (adapted mostly from your tutorial):

import tensorflow as tf
from tf_keras_vis.utils.scores import CategoricalScore
from matplotlib import cm
from tf_keras_vis.gradcam import GradcamPlusPlus
from tensorflow.keras.preprocessing.image import load_img

mobilenet = tf.keras.applications.MobileNetV3Large(
    input_shape=(224,224,3), include_top=True,
    weights='imagenet', classes=1000)

# Random goldfish or bear images
img1 = load_img("goldfish.jpeg", target_size=(224,224))
img2 = load_img("bear.jpeg", target_size=(224,224))

# Cast to tf.float32 to watch gradients
# No additional preprocessing is necessary for mobilenetv3 as the architecture takes care of it
# Source: https://www.tensorflow.org/api_docs/python/tf/keras/applications/mobilenet_v3/preprocess_input
images = tf.cast(np.asarray([np.array(img1), np.array(img2)]), tf.float32)

def model_modifier(cloned_model):
    cloned_model.layers[-1].activation = tf.keras.activations.linear
    return cloned_model

score = CategoricalScore([1, 294])

gradcam = GradcamPlusPlus(mobilenet,
                  model_modifier=model_modifier,
                  clone=False)

cam = gradcam(score, images, penultimate_layer=-1)

image_titles = ['Goldfish', 'Bear']
f, ax = plt.subplots(nrows=1, ncols=2, figsize=(12, 4))
for i, title in enumerate(image_titles):
    heatmap = np.uint8(cm.jet(cam[i])[..., :3] * 255)
    ax[i].set_title(title, fontsize=16)
    ax[i].imshow(images[i]/255)
    ax[i].imshow(heatmap, cmap='jet', alpha=0.5) # overlay
    ax[i].axis('off')
plt.tight_layout()
plt.show()

re-executions of activation_maximization generate different results for same graph/weights

hello,

I've made the following minor change to the visualize dense layer example:

for CurrentAttempt in range(0,4):
        print('attempt number {}'.format(CurrentAttempt))
        activation = activation_maximization(loss,callbacks=[Print(interval=50)])
        image = activation[0].astype(np.uint8)

        subplot_args = { 'nrows': 1, 'ncols': 1, 'figsize': (3, 3),
                         'subplot_kw': {'xticks': [], 'yticks': []} }
        f, ax = plt.subplots(**subplot_args)
        ax.imshow(image)
        ax.set_title('Ouzel', fontsize=14)
        plt.tight_layout()
        plt.show()

the generated images for ouzel class are different for each execution.
( i can upload the images if that will help ).

I'm not sure if this is an error of some kind or expected behavior .
can someone please shed some light on this ?
Omer.

don't clone model by default if modeifier is None

if you are not modifying the model i guess there should be no reason to clone it by default

Error occurred in GradCAM++ when the data type of model-outputs and gradients are different.

Here's a minimal example that is giving the above error:

import numpy as np
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import applications, layers
from tensorflow.keras.mixed_precision import experimental as mixed_precision
from tf_keras_vis.gradcam import GradcamPlusPlus

policy = mixed_precision.Policy("mixed_float16")
mixed_precision.set_policy(policy)

base_model = applications.MobileNet(
    input_shape=[224, 244, 3],
    include_top=False,
    weights=None,
)
layer = base_model.output
layer = layers.Flatten(name="flatten")(layer)
layer = layers.Dense(2, dtype=tf.float32)(layer)
model = keras.models.Model(inputs=base_model.input, outputs=layer)
data = np.empty(model.input.shape[1:])


def model_modifier(model):
    model.layers[-1].activation = keras.activations.linear
    return model


def loss(output):
    return [o[0] for o in output]


GradcamPlusPlus(model, model_modifier)(loss, data)
print("Done.")

Originally posted by @bersbersbers in #41 (comment)

"array type dtype('float16') not supported" in ScoreCAM with mixed_precision

This example fails when interpolating data deep down in scipy:

import numpy as np
import tensorflow.keras as keras
from tf_keras_vis.scorecam import ScoreCAM

policy = keras.mixed_precision.experimental.Policy("mixed_float16")
keras.mixed_precision.experimental.set_policy(policy)

ScoreCAM(model := keras.applications.MobileNet())(
    lambda output: [o[0] for o in output], np.empty(model.input.shape[1:])
)

Generating more than 4 Images at once

Hi,

I want to generate the activation maximization for all the neurons of my output layer (128). But when generating more than 4 at once, I get the following error:

Traceback (most recent call last):
  File "", line 49, in <module>
    activation = activation_maximization(loss, seed_input=seed_input, callbacks=[Print(interval=50)], steps=512)
  File "/home/stefan/anaconda3/envs/blender/lib/python3.7/site-packages/tf_keras_vis/activation_maximization.py", line 114, in __call__
    optimizer.apply_gradients(zip(grads, seed_inputs))
  File "/home/stefan/anaconda3/envs/blender/lib/python3.7/site-packages/tensorflow_core/python/keras/optimizer_v2/optimizer_v2.py", line 426, in apply_gradients
    grads_and_vars = _filter_grads(grads_and_vars)
  File "/home/stefan/anaconda3/envs/blender/lib/python3.7/site-packages/tensorflow_core/python/keras/optimizer_v2/optimizer_v2.py", line 1026, in _filter_grads
    grads_and_vars = tuple(grads_and_vars)
  File "/home/stefan/anaconda3/envs/blender/lib/python3.7/site-packages/tf_keras_vis/activation_maximization.py", line 113, in <genexpr>
    grads = (K.l2_normalize(g, axis=tuple(range(len(g))[1:])) for g in grads)
  File "/home/stefan/anaconda3/envs/blender/lib/python3.7/site-packages/tensorflow_core/python/keras/backend.py", line 4697, in l2_normalize
    return nn.l2_normalize(x, axis=axis)
  File "/home/stefan/anaconda3/envs/blender/lib/python3.7/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/home/stefan/anaconda3/envs/blender/lib/python3.7/site-packages/tensorflow_core/python/ops/nn_impl.py", line 615, in l2_normalize
    return l2_normalize_v2(x, axis, epsilon, name)
  File "/home/stefan/anaconda3/envs/blender/lib/python3.7/site-packages/tensorflow_core/python/ops/nn_impl.py", line 642, in l2_normalize_v2
    square_sum = math_ops.reduce_sum(math_ops.square(x), axis, keepdims=True)
  File "/home/stefan/anaconda3/envs/blender/lib/python3.7/site-packages/tensorflow_core/python/util/dispatch.py", line 180, in wrapper
    return target(*args, **kwargs)
  File "/home/stefan/anaconda3/envs/blender/lib/python3.7/site-packages/tensorflow_core/python/ops/math_ops.py", line 1595, in reduce_sum
    _ReductionDims(input_tensor, axis))
  File "/home/stefan/anaconda3/envs/blender/lib/python3.7/site-packages/tensorflow_core/python/ops/math_ops.py", line 1606, in reduce_sum_with_dims
    gen_math_ops._sum(input_tensor, dims, keepdims, name=name))
  File "/home/stefan/anaconda3/envs/blender/lib/python3.7/site-packages/tensorflow_core/python/ops/gen_math_ops.py", line 10163, in _sum
    input, axis, keep_dims=keep_dims, name=name, ctx=_ctx)
  File "/home/stefan/anaconda3/envs/blender/lib/python3.7/site-packages/tensorflow_core/python/ops/gen_math_ops.py", line 10197, in _sum_eager_fallback
    ctx=ctx, name=name)
  File "/home/stefan/anaconda3/envs/blender/lib/python3.7/site-packages/tensorflow_core/python/eager/execute.py", line 67, in quick_execute
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InvalidArgumentError: Invalid reduction dimension (4 for input with 4 dimension(s) [Op:Sum]

Here is my code:

#load model
model = tf.keras.models.load_model(path)

#count the neurons
neurons = 5

# Find the codelayer and set to linear
def model_modifier(m):
    code = m.get_layer('code')
    code.activation = tf.keras.activations.linear

def loss(output):
    ret = tuple(output[x, x] for x in range(0, neurons)) # (output[0, 0], output[1, 1], ... , output[neurons, neurons])
    return ret

#create isntance
activation_maximization = ActivationMaximization(model, model_modifier, clone=False)

#visualize
seed_input = tf.random.uniform((neurons, 224, 224, 3), 0, 255)
activation = activation_maximization(loss, seed_input=seed_input, callbacks=[Print(interval=50)], steps=512)

For values of neurons <= 4 it seems to work just fine.
Thanks for helping

Guided Backpropagation

Hi @keisen, thanks for putting this together.
Is there a way to generate a guided backpropagation saliency map with a custom function in gradient-modifier?