syorami / autoencoders-variants

Pytorch implementations of various types of autoencoders

• 👋 Hi, I’m @syorami
• 👀 I’m interested in Computer Vision, Semi-Supervised Learning, Self-Supervised Learning and Efficient Training etc.
• 💞️ I’m looking to collaborate on above topics
• 📫 How to reach me: [email protected]

autoencoders-variants's Issues

Why does sparse encoding using KL of two different labels have same 0 neurons?

Hi @syorami , All,

I hope to understand sparse encoding conceptually better hopefully with an answer to this question. Why do two different classes have the same neurons contributing to sparsity (the neurons with 0 you see below; i.e. in positions 1,8,21,25,26,27)? I would assume the goal of sparsity would be, by pushing the sparsity constraint, only neurons most sensitive to a specific class would show high activation, while the rest would be zeroed out; Thus for different classes different neurons should be pushed to 0, and not the same ones. If that is not the case, then what is the benefit of applying sparsity constraint on the neurons?

For class 5 of mnist:

    [ 0.0000, 11.8318, 16.0555,  3.6289, 15.2786,  7.6066,  7.1188,  0.0000,
      8.3991, 13.2328,  7.3709,  0.0000, 13.2765, 14.9766, 13.9301, 12.1690,
      8.8316, 10.5639, 15.9207, 11.9792,  0.0000,  7.9138,  6.3603,  8.8305,
      0.0000,  0.0000,  0.0000, 10.5949, 11.8915, 16.3092, 10.0973,  9.6339]]

For class 1 of mnist:

    [ 0.0000, 13.2143,  1.6391, 10.2586,  3.8103,  6.0026,  3.5844,  0.0000,
      3.9776,  9.9163,  2.1478,  0.0000,  8.2941,  3.7168,  1.9532,  3.7498,
      3.7483,  5.6161,  9.1870,  5.3832,  0.0000,  5.4140,  0.8535,  6.2867,
      0.0000,  0.0000,  0.0000,  5.9366,  5.6120,  6.0602,  9.4863,  8.5068]]

could you add reference papers for each method ?

sparse_autoencoder_l1, does the l1 constrain really make the representation sparse?

Did you find any paper or did you do any empirical experiment that proves that simply adding l1 loss to hidden representation encourages sparsity on the hidden representation

In sparseencoder-KL method, why is sparsity constraint applied on all fc layers across encoders and decoders?

Hi @syorami /Anyone,

As I understand it, the goal of sparse constraint is to enable only the "encoding" to be sparse. Why then is sparsity loss accumulated across every fc layer of both the encoder and decoder? Could you please let me know of this at the earliest if possible? If any reference to an article or text or blog that discusses this could suffice me too. Please help.

Thanks,

Is have error: output with shape [1, 28, 28] doesn't match the broadcast shape [3, 28, 28]

Make train any variant.