thomfoster / minrlhf Goto Github PK

Nice work, It helps me a lot!

I would like to ask about the reward augmentation in:

# `.get` computes augmented_reward_buffer = reward_buffer + beta * reward_augmentation_buffer
# zeros by default if no reward_augmenter function given during init
self.reward_augmentation_buffer = torch.zeros(size=(self.max_episodes, self.max_ep_length), dtype=torch.float32).to(self.device)
self.augmented_reward_buffer    = torch.empty(size=(self.max_episodes, self.max_ep_length), dtype=torch.float32).to(self.device)

and

def naive_logprob_augmenter(buf: Buffer)->None:
       buf.reward_augmentation_buffer[:, :] = -((buf.pi_t_logprobs_buffer - buf.pi_0_logprobs_buffer) ** 2)/2

Which paper first proposed this technique?

Hi,
Great repository, very minimal and clean indeed. I'm sure many other students will learn immensely from here.

I'm currently running the gpt2 as the active model, with bhadresh-savani/distilbert-base-uncased-emotion as reward model.

How I can save the final trained model to a local directory?
Could it be done similarly to transformers Trainer:

# Huggingface transformers trainer:
from transformers import Trainer
# ...
trainer.train()
trainer.save_model("my-model")

# minRLFH trainer
ppo_trainer.train()
ppo_trainer.save_model("my-model") ## <-- possible? 🤔

Hi,

Thank you very much for your contribution and this is an amazing project.

May I ask is it possible to release the code based on jax.

Thank you very much!

Best

I installed the minRLHF library using pipenv

pipenv run python -m pip install "minrlhf @ git+https://github.com/thomfoster/minRLHF.git"

And got this execution error:

Traceback (most recent call last):packages/minRLHF/ppo_trainer.py", line 7, in <module>
    from minRLHF.buffer import Buffer
  File "{...}/lib/python3.10/site-packages/minRLHF/buffer.py", line 6, in <module>
    from torch_discounted_cumsum import discounted_cumsum_right
ModuleNotFoundError: No module named 'torch_discounted_cumsum'

The solution was to simply install the missing torch-discounted-cumsum module

 pipenv install torch-discounted-cumsumÏ

I'm not sure if that's a pipenv installation issue, or somewhere in the library itself.

Hello! In your implementation in https://github.com/thomfoster/minRLHF/blob/main/minRLHF/buffer.py#L129, you perform sample-level normalization, why not batch-level normalization?