activation_maximization with YOLOv3 about tf-keras-vis

[keisen]

Thank you for the interesting question!
I'm going to examine the cause and find a solution.

But now, unfortunately, I don't have enough time, so it may take quite a while.
Could you please submit a code snippet to reproduce the error?
If you do so, we can respond relatively quickly.

Thanks!

[asquare92]

Thank you for the interesting question!
I'm going to examine the cause and find a solution.

But now, unfortunately, I don't have enough time, so it may take quite a while.
Could you please submit a code snippet to reproduce the error?
If you do so, we can respond relatively quickly.

Thanks!

Thank you for your quick response @keisen. Here is the gist that generates the above-mentioned error log. I hope it will help you to locate better and speed up the process :)

[keisen]

Thank you for sharing the code!
I'm sorry for the late reply, I didn't realize your comment updated.

I've made sure that your code and improved the direct cause of the ValueError.
(The cause was that the number of loss functions that need four in that situation was just one.)
However, we are facing other error that is more complicated.

https://gist.github.com/keisen/a3b492fef779974d3e8cc07d97189afd

I'm not sure the concrete causes because I'm not familiar with YOLO, but it seems to be NOT able to calculate gradients in yolo_nms Lambda layer.
I will find out this problem a little more, so please wait for a while.

[asquare92]

Thank you for sharing the code!
I'm sorry for the late reply, I didn't realize your comment updated.

I've made sure that your code and improved the direct cause of the ValueError.
(The cause was that the number of loss functions that need four in that situation was just one.)
However, we are facing other error that is more complicated.

https://gist.github.com/keisen/a3b492fef779974d3e8cc07d97189afd

I'm not sure the concrete causes because I'm not familiar with YOLO, but it seems to be NOT able to calculate gradients in yolo_nms Lambda layer.
I will find out this problem a little more, so please wait for a while.

Thank you very much for making an effort and helping!!

Currently, as I noticed, that you are not using the YOLOv3 loss function, however, I just would like to let you know that I realized a small mistake that I did in passing the parameters to the YOLOv3 loss function in the gist I shared above. I passed the total number of classes as classes=9, but it should be classes=FLAGS.num_classes. Please consider correcting this for further work.

And I am very hopeful that we will be able to make tf-keras-vis work with this state-of-the-art object detector.

Thanks :)

[asquare92]

@keisen

Hi Keisen,

I just have a small update that I am successfully able to run the activation maximization with the mock loss functions you described in the gist you provided.

The model actually needed to initialize in the training mode instead of inference mode. Since yolo_boxes_0, yolo_boxes_1, yolo_boxes_2, and yolo_nms layers are just post-processing layers so they don't take part in the training and hence they don't have any gradients. Initializing the model in the training mode solves the gradient registry has no entry for: CombinedNonMaxSuppression problem.

However, I have got another issue when running the activation maximization with the original YOLOv3 loss function.
Here is the gist that is working with mock loss functions, but I still cannot interpret the output plots. And here is the gist with the YOLOv3 loss function that reproduces the error that I am now currently facing.

In my opinion, while activation maximization (e.g. gradient ascent), loss function needs a predicted value to maximize the total loss but I am not sure. Can you please take a look in your free time?

Thanks :)

[keisen]

Thank you for sharing your awesome work! This is the first time that I've seen the ActivaitionMaximization of YOLO.
I believe you've been almost getting the result you want. Lastly all we have to do is to define the loss (score) function to generate meaningful images.

Before I explain concretely what we should do, I have to let you know that a terminology in tf-keras-vis have mislead you. This problem will be improved in next updates.
I believe the loss function of tf-keras-vis should be called score function.
A loss function that compare model output with annotation data is used to train model. But a loss function required by tf-keras-vis just returns the score that want to maximize by ActivationMaximization.
It may be correct to be called loss from the viewpoint of gradient decent, but it's proper to be called score from the viewpoint of tf-keras-vis users.

So, the next step is NOT to use the YOLOv3 loss functions, but is to define the loss (score) function that returns the score corresponding to something you want to visualize.
(If there is no necessary score within a output tensor, the loss of the output return zero.)

For example, in the case of Imagenet model, when you want to visualize Ouzel (class No. 20), the score function is as follow:

def loss(output):
    # output shape is (samples, 1000)
    return output[:, 20]

And if the output does NOT include the necessary score:

def loss(output):
    return output * 0

Please feel free to ask if you have any questions or face any errors.
Thanks!

[asquare92]

Hi Keisen, I am sorry for taking too long to write you back. I was actually stuck into something else. And thank you for explaining the loss or score function as it makes sense now. However, can you please elaborate a little on the following.

So, the next step is NOT to use the YOLOv3 loss functions, but is to define the loss (score) function that returns the score corresponding to something you want to visualize.

Let's say, for example, I want to visualize the kind of patterns and textures that the filters in Yolov3 look for an image after training. What kind of score function should I choose in this case? Any lead or help would be appreciated.

[keisen]

I'm sorry for the late reply.

I want to visualize the kind of patterns and textures that the filters in Yolov3 look for an image after training. What kind of score function should I choose in this case?

Unfortunately, I'm not familiar with YOLO, so I don't know concretely that.
But ...

First, for visualizing the filter you want, you have to know the index of the filter in the layer outputs. (Otherwise, you have to visualize all filters one by one to find ones you want.)

Second, if the index was 10, you have to implement the loss function such as below:

def loss(layer_output):
    return layer_output[..., 10]

I hope it could help you.
Thanks!