Optimizer and traning frequency about deeprl

[ShangtongZhang]

Yes it's different. I remember you can set centered=True in PyTorch to make them the same. But I'm not sure.

Do you mean frame skip? It exists in atari wrapper.

[xfdywy]

Not only frame skip. I mean during the training process, you don't have to calculate gradient and then do bp every step.

By the way, you don't implement any clip for gradient?

[ShangtongZhang]

I mean during the training process, you don't have to calculate gradient and then do bp every step.

I don't know this trick, do you know where it comes from?

By the way, you don't implement any clip for gradient?

In my understanding gradient clip is only necessary in multi-process setting

[xfdywy]

update_freq=4 defined here
https://github.com/kuz/DeepMind-Atari-Deep-Q-Learner/blob/1d52e98152fccd968d63ca67e33f5cbf3e83a878/run_gpu#L14

and used here
https://github.com/kuz/DeepMind-Atari-Deep-Q-Learner/blob/1d52e98152fccd968d63ca67e33f5cbf3e83a878/dqn/NeuralQLearner.lua#L341

clip_delta=1
https://github.com/kuz/DeepMind-Atari-Deep-Q-Learner/blob/1d52e98152fccd968d63ca67e33f5cbf3e83a878/run_gpu#L31

[ShangtongZhang]

Thanks, I haven't read the original lua code before.
It's interesting to know that tricks.

[xfdywy]

Another issue................. In deepmind code, gradient clip is really "clip"

if self.clip_delta then
delta[delta:ge(self.clip_delta)] = self.clip_delta
delta[delta:le(-self.clip_delta)] = -self.clip_delta
end

But you use nn.utils.clip.grad_norm, which is more likely to do "normalization" .....
http://pytorch.org/docs/master/_modules/torch/nn/utils/clip_grad.html

[ShangtongZhang]

Yes .. I know that, just to conform with pytorch.

[xfdywy]

Ok, thanks! So many details................