PEPG questions about estool

[cammckenzie]

Thanks for the quick reply @hardmaru, much appreciated.

In regards to point 1, I appear to have misinterpreted the paper. I did notice that the algorithm doesn't appear to have any scaling for the standard deviation term either, but your implementation seems to scale by the batch size?

delta_sigma = (np.dot(rS, S)) / (2 * self.batch_size * stdev_reward)

Anyway, ultimately, I don't think that it's very important. Was there another paper that had the additional tricks that you implemented (annealing learning rates etc.) or you just pulled them from the standard machine learning approaches?

You're quite right on point 2, I'm not sure how I interpreted that as a uniform sample.
cheers

[hardmaru]

Hi @cammckenzie

Thanks for the message and interest.

For your first question regarding normalized by batch size, in the PEPG paper, in the right side of "Algorithm 1" on page 7 that I tried to base the implementation on, there is no scaling for population size. I guess the difference can be adjusted by adjusting the learning rate.

For the second point, I am not sampling from a uniform distribution, but sampling from a normal distribution using np.random.randn as you mentioned.

self.epsilon = np.random.randn(self.batch_size, self.num_params) * self.sigma.reshape(1, self.num_params)

Thanks

[hardmaru]

You're right, I incorporated scaling by batch (or population) size so that the same learn rate parameter can be used for various different population sizes (I think the paper should have done that too, just an obvious thing to do). The annealing and other tricks are standard tricks in deep learning, I recommend going through OpenAI's ES paper as well - they used Adam rather than vanilla SGD for instance.

[cammckenzie]

Thanks again @hardmaru, looking forward to your next blog whatever that may be.