Question about KL term about alpaca_farm

[lxuechen]

Hi, thanks for your question. I believe there's a misunderstanding.

In the current RLHF implementation, the local KL term in the original PPO algorithm isn't included in the objective.

Rather a global KL term (between the current policy model and the starting SFT model) is included to penalize the policy from deviating far away from the SFT policy. See Equation (2) in this paper. This is the (Monte Carlo estimate of the) KL being implemented in the code you quoted.

Note the implementation maximizes the reward, which in turn (partly) minimizes the KL through the penalty term. This means logprobs - ref_logprobs is encouraged to be low, which then means sampled sequences are encouraged to have high ref_logprobs. This makes sense, as the reward model is trained on samples generated from the initial SFT model, and thus it's most suitable to rate samples close to this distribution.