KLD calculation about vae

[botkevin]

Pretty sure he multiplies the KLD by .1 because that is his KLD weight hyperparameter

[botkevin]

Also, while working on a VAE that I wrote based on this, if I change the offending mean to a sum, my recon loss is much higher (2x), while my kl_loss starts as similar, but decreases faster with torch.mean; and all my reconstructed images are basically the same blurry image. I have no idea why this would change the reconstruction so much... I will have to do some investigation.

[botkevin]

Mean is equivalent to sum, it's just a scalar difference. Normally, using adam, if we don't have a composite loss, this scale doesn't matter, so if I change the sum to mean, the program should backpropagate the same. I changed the mean to a sum and decreased the kld weight, which fixed my problem. Basically when I changed the mean to a sum, I put too much weight on the kl weight and caused the latent distributions to be too strongly bound to the normal guassian.