Qs about Paper about dvit_repo

[zhoudaquan]

Hello, I notice one figure (Fig.5) in your paper like this.

Would you please tell me the meaning of the thick blue vertical line? Or, how to get the conclusion: "In the deep blocks, the MHSA learns nearly uniform global attention maps with high similarity."
Respect.

Hi,

Thanks for joining the discussion regarding the paper. The thick line means that when aggregate the tokens in V, that portion has a larger weights. The "near uniform distribution" refers to the fact that, in the attention map, all the values along the "vertical direction" has similar values. This implicates that when aggregate those tokens, each vector in the attention map gives a similar spatial attention.

I hope this clarify your questions.

Best regards

[elk-april]

Thanks for your replying!!
Your answer is very helpful.