Can we get a repo license doc btw? The setup.py says MIT but if there is no LICENSE doc I think it may be officially proprietary code until you add one.

Overview

This issue captures some of the key steps required to reproduce the Constitutional AI paper steps to fine tune a RLHF model with feedback generated by a RLAIF model.

Phase One

• Gather a dataset of harmful prompts
• Create a base script to compose prompts using a base constitution
• Generate a new dataset of prompts + responses using Carper's GPT-J RLHF to review / critique the output
• Fine-tune the original model on revised responses using supervised learning

Phase Two

• Sample the fine tuned model using the dataset of harmful prompts to create a new dataset with multiple outputs
• Train a "reward model' (i.e. https://github.com/Dahoas/reward-modeling) to select the best result (fine tuned preference model)
• Use RLAIF training to fine tune the RLHF model

Evaluation of instruction tuned models is difficult for many of the properties we actually care about.
Language modelling and multiple choice benchmarks may capture some aspects of knowledge and reasoning but don't capture many of the properties we care about in instruction tuned dialog agents, like long term coherence, multi task generalisation, ability to use tools, harmlessness, etc.
To address this, we can try to use LLMs to evaluate LLMs.

Ways to do this (in order of increasing complexity):

1. Generate a LM or forced choice QA dataset and evaluate the instruct model offline
2. Use reward functions on generations from some (possibly generated) prompt dataset (e.g learn RMs, zero-shot LLM reward functions, etc)
3. Online exploration and evaluation using another LLM

We should implement these into our repository.
Basic implementations would be:

1. A script that uses langchain and some seed prompts to generate a multiple choice dataset.
2. A script that prompts LLMs to rate outputs or generate critiques
3. A script that has an LLM attempt to use the LLM under test to complete some task, and a check for if that task was successfully completed.