Unified reward function/model architecture for a wide range of tasks about palm-rlhf-pytorch

[James4Ever0]

RLHF requires creating multiple models like SFT, RM, PPO-tuned model. Is it possible to improve storage and memory efficiency, reduce computation if we freeze some huge layers of the pretrained model, only fine-tune certain layers to create SFT, RM, PPO using OpenDelta or other libraries/methods? I read that your repo is using LoRA but I'm not sure if it fulfills all goals described above. Common implementations like minRLHF requires four separate models, three are derived from the pretrained model as actor, critic and reference, in addition to an external sentiment rating model.

[James4Ever0]

To address this proposal even further, I think a good reward function can self-evolve and adapt to new environments (when the data source is no longer fixed static "archives" but streaming), making this model communicative, multipurpose, realtime and even into AGI. A good reward function can let the agent to learn from almost anything, including human feedback, computer system (sensor data, terminal/GUI input/output, internet, program threads and more) and self-invented signals. WebGPT is a clear example to make GPT3 into an active agent. There will be more to come.