multiple inputs CNN based regression problem about tf-keras-vis

[keisen]

Hi @linbingru

The first argument of Gradcam#__call__() MUST be the score function as follows.

function score(outputs):
    # TODO calculate score values
    return score_values

cam = gradcam(score, (X1, X2), penultimate_layer=-1)

Please see the API document and examples for details.

Thanks!

[keisen]

Hi, @linbingru .

Saliency is visualized by the gradients with respect to input values, Gradcam is visualized by the gradients with respect to the activation of the convolution layer you specified. (Because you specified dense_3 above, Gradcam refers to the convolutional layer just before it.)
These will highlight similar positions basically, however, if the difference is big, the model might NOT be trained enough or there are some other problems.

Could you submit the result of model.summary() and the code snippet for reproducing the issue?

Thanks!

[linbingru]

Hi,@keisen

Thank you very much for solving my question!
I slove the above problem, however, the new problem comes out.

"The result calculated by saliency map it can represent the weight position. However, the result calculated by gradcam was decreasing by value."

---

from tf_keras_vis.gradcam import Gradcam
from tf_keras_vis.utils.model_modifiers import ReplaceToLinear

# score function
def score_function(output):
    return output[:, 0]

# attention layer
layer_idx='dense_3'

# cam object
#replace2linear = ReplaceToLinear()
gradcam = Gradcam(model, clone=True)

# Generate heatmap with GradCAM
cam = gradcam(score_function, [X2[test], X1[test]], penultimate_layer='dense_3')

---

And when I try different score_function the result goes zero.

---

print(saliency_map)
[array([[0.03965255, 0.06007694, 0.08179716, 0.08468853, 0.15592117,
         0.16889493, 0.21754322, 0.19366294, 0.2498665 , 0.26016768,
         0.36875371, 0.434514  , 0.48417194, 0.43154544, 0.54129211,
         0.50707816, 0.61562472, 0.68896445, 0.76950206, 0.82405041,
         0.84028072, 0.80198821, 0.76708105, 0.99999993, 0.76179289,
         0.8125057 , 0.84014564, 0.84142979, 0.86374242, 0.70754946,
         0.78490675, 0.85101713, 0.95127031, 0.80641341, 0.69927845,.......]],dtype=float32)]

---

print(cam)
[array([[1.00000000e+00, 9.53662038e-01, 9.07324016e-01, 8.60986114e-01,
         8.14648151e-01, 7.68310249e-01, 7.21972227e-01, 6.75634325e-01,
         6.29296362e-01, 5.82958400e-01, 5.36620438e-01, 4.90282506e-01,
         4.43944544e-01, 3.97606581e-01, 3.51268649e-01, 3.10990214e-01,
         3.01009357e-01, 2.91028500e-01, 2.81047612e-01, 2.71066755e-01,
         2.61085898e-01, 2.51105040e-01, 2.41124198e-01, 2.31143340e-01,
         2.21162468e-01, 2.11181596e-01, 2.01200753e-01, 1.91219881e-01,
         1.81239024e-01, 1.71258166e-01, 1.62492141e-01, 1.56155795e-01,
         1.49819434e-01, 1.43483087e-01, 1.37146741e-01, 1.30810395e-01,
         1.24474056e-01, 1.18137702e-01, 1.11801349e-01, 1.05465010e-01,
         9.91286561e-02, 9.27923024e-02, 8.64559487e-02, 8.01196098e-02,
         7.37832561e-02, 6.95410892e-02, 6.73930794e-02, 6.52450845e-02,
         6.30970821e-02, 6.09490909e-02, 5.88010848e-02, 5.66530824e-02,.......]],dtype=float32)]

---

Best wish,
Thanks for helping me to solve my issue.