Comments (3)
Hi @bkj,
The VAE losses are based on the KL divergence between the distribution of output and target. So you have to have a pdf assumption. BCE is the KL divergence for Bernoulli (binary) distributions. Categorical crossentropy is the one for multinomial distributions. If Gaussian is a good approximation for you case, there is a well define loss there as well…
That being said, try to just plugin your cost function and see if it works xD
it may not be statistically sound, but if it works, you're good. Just don't tell statisticians I suggested that for the sake of my reputation ahhahaha
from gumbel.
Alright thanks -- I'll give that a shot, but I won't tell anyone... I'm not sure whether Gaussian would be a good approximation in my case -- since there are image featurizations, I would expect them to be multimodal and would think Gaussian is a bad fit for that reason, but I don't have much intuition about these methods.
Another issue: In Jang's implementation, he is maximizing the ELBO:
loss=tf.reduce_mean(-elbo)
but it doesn't seem like you're doing that here. Wouldn't you want to change gumbel_loss
to:
def gumbel_loss(x, x_hat):
q_y = K.reshape(logits_y, (-1, N, M))
q_y = softmax(q_y)
log_q_y = K.log(q_y + 1e-20)
kl_tmp = q_y * (log_q_y - K.log(1.0/M))
KL = K.sum(kl_tmp, axis=(1, 2))
elbo = data_dim * bce(x, x_hat) + KL # ** changed `-` to `+`
return elbo
if you're minimizing that function? (This is also more in line w/ adding KL to BCE as in https://github.com/fchollet/keras/blob/master/examples/variational_autoencoder.py)
from gumbel.
Yeah, if the distribution is multimodal, then MSE and Gaussian assumptions are not really good. The idea of the using a proper cost functions is to make sure the amount of contributions of the reconstruction loss (in this case BCE) and the KL regularization to the ELBO are correctly balanced. You can always play with weighting the contributions of each term to the cost function. But then you may have an unnecessary hyperparameter. Just plug and play while you look for a maximum likelihood approach that reduces the loss to one close to what you want to use. This way you get that new loss and apply here. For example, for MSE with zero mean and unity variance assumption, the contributions would be data_dim *0.5*MSE + KL
. That 0.5 comes from the log of the Gaussian pdf once you ignore all constant terms.
good catch! yeah, please double check if the signs are correct. I'll review the PR.
from gumbel.
Related Issues (5)
- Minimizing KL (not maximizing) HOT 1
- How to sample the Gumbel noise? HOT 2
- ImportError: No module named 'agnez' HOT 1
- Great thanks for your code. I have a question. If I have a matrix, whose elements is discrete (i.e., binary state 0 or 1), needed to be trained under a given loss function, how should I train it?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gumbel.