Potential Problem of the loss function in ABML about few_shot_meta_learning

[ZichenMiao]

Hi Cuong,

Thanks for your reply.

The first thing I want to mention is that the logSigma's prior is Gamma Distribution not Gaussian. But I think it's a minor point.

Also, I totally agree that it's inappropriate to implement q(theta)=Dirac(theta-theta*) from the analytic perspective. Perhaps it's just an engineering approximation.

Anyway, I think it's better to add the weight-decay term if you really want to exactly reproduce the paper.

[cnguyen10]

Hi Zichen,
You are right. I did not implement the prior of theta into the calculation. Instead of setting the priors as shown in the paper, we can simply use the L2 regularization (weight_decay of the op_theta in Pytorch). In such case, the both the priors for the mean and logSigma would be Gaussian.
Besides, the main reason I did not do that is my disagreement with the assumption that sets q(theta) = Dirac_delta(theta - theta*). At the very first equation on page 2 of the paper, Jensen's in equality is used to derive the lower-bound. That derivation is associated with the assumption that q(theta) > 0. Therefore, setting q(theta) as a Dirac delta function violates that assumption.
Nevertheless, let me know if the explanation above resolves your concern.