Comments (4)
Thank you for your fast response!
But I meant annotation not evaluation. Do you think I would be able to adapt the annotators in my fork on my own, maybe using the Claude-related code from AlpacaEval?
from alpaca_farm.
Hi,
We currently do not have support for Claude in this repo for the AlpacaFarm evaluation standard.
However, we do have a separate evaluation library based on AlpacaFarm available here that has support for Claude: https://github.com/tatsu-lab/alpaca_eval
There are some differences in the evaluation procedure between the two, which is covered here: https://github.com/tatsu-lab/alpaca_eval#differences-with-alpacafarm
from alpaca_farm.
Or are annotators and evaluators more or less the same thing here?
from alpaca_farm.
Yup that could definitely work :)
Annotators and evaluators are almost the same there, if you're referring to the train/test split.
For collecting the preference data for training, we use the same set of pairwise evaluators, with the added step of flipping a random 25% of the labels (as detailed in the paper).
For doing the pairwise evaluation, we use the same evaluators/annotators but don't inject the extra noise.
from alpaca_farm.
Related Issues (20)
- [Bug] Error importing is_deepspeed_zero3_enabled HOT 1
- Question about KL term HOT 1
- Reproducibility of pretuned reward model
- Use with Llama-2-70b-hf? HOT 4
- KeyError: 'llama' in /recover_model_weights.py HOT 2
- Confusing detail preference mapping HOT 1
- recover_model_weights.py gives WARNING:root:Your base LLaMA checkpoint is converted with transformers==4.27.0.dev0
- Huge memory demand of recover_model_weights.py?
- RewardModel.from_pretrained() loads redundant weights (incurs extra ~30GB of RAM)
- Why use FSDP instead of Deepspeed? HOT 2
- lower cuda version? HOT 1
- integrity_check error with sft10k
- score with reward model HOT 2
- RecursionError: maximum recursion depth exceeded
- BaseAnnotator.__init__() got an unexpected keyword argument 'other_keys_to_keep' HOT 4
- PairwiseAutoAnnotator always "Annotating 0 examples with gpt4_3" HOT 1
- Repeated Deprecation Error HOT 3
- Error downloading pre-trained weights
- Using pretrained models HOT 3
- Inquiry Regarding Supervised Fine-Tuning with AlpacaFarm Framework for Pythia Models HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from alpaca_farm.