4-D equivalent? about set_transformer

[juho-lee]

The current module does not support 4D matrix multiplication, so you have to first reshape the matrix into a vector and apply the set transformer modules.

[zhiqihuang]

What if I have a set of matrices instead of a set of vectors? Is it possible to extend the Set Transformer framework to cover that scenario?

I played around with it a little (including making some small tweaks) but got bogged down with the .bmm call in the MAB module:

RuntimeError: Expected 3-dimensional tensor, but got 4-dimensional tensor for argument #1 'batch1' (while checking arguments for bmm)

@zabzug-pfpt hi, are you able to extend set-transformer to 4-D input?

[zabzug-pfpt]

I did not, I suspect that there is not an easy way of doing so. In my case, I decided to apply a convolutional layer + pooling over one dimension to reduce down from 4D to 3D, and then applied Set Transformer.