Comments (5)
it's a known issue to us as well.
- for now, for MT-Bench, we do the same thing (filtering out the -1 score) when computing the average. it's okay because, if it's a random API error, the average score is still unbiased (but more noise)
- for this reason, we did track the error rates so that we know how noisy the average score is
- perhaps a better way is to at least try a few times (if there is stochasticity in either the answer generation or teacher judgment) until we get something that is not
-1
- as for MT-Bench-Branch, as the users would like to know the actual per
qna.yaml
score, it's probably more important to get at least 1 score that is not-1
.
as it's on our radar anyway, i can actually take a stab to see why there is -1
and how to make sure we can get at least one meaningful score per Q&A
from eval.
i can actually take a stab to see why there is
-1
and how to make sure we can get at least one meaningful score per Q&A
The most common errors I am seeing are hitting the 4096 context length limit. That gives an easy way to recreate the issue.
from eval.
This is the error vllm gives if you set the context larger than the model supports:
ValueError: User-specified max_model_len (5000) is greater than the derived max_model_len (max_position_embeddings=4096 or model_max_length=None in model's config.json). This may lead to incorrect model outputs or CUDA errors. Make sure the value is correct and within the model context size.
Setting max_position_embeddings in the model config does get past the error but obviously results are no longer reliable. This is probably acceptable for testing but it's not a great experience for the user just trying things out.
from eval.
Related: instructlab/instructlab#1615
from eval.
Resolved with #49 and instructlab/instructlab#1597
from eval.
Related Issues (20)
- [Epic] MMLU Branch
- [Epic] MT Bench
- [Epic] MT Bench Branch
- Ensure that SDG path data branch matches evaluation branch HOT 7
- Error handling HOT 2
- question_id is not be stored consistently and is probably losing consistency
- LM Eval version needs to be bumped
- Enable MMLU tests to be run on an already served model
- Error running `tests/test_gen_answers.py` and `tests/test_judge_answers.py` with prometheus-8x7b-V2.0 HOT 5
- Eval needs debug logging
- Invalid sdg path with mmlu_branch prints stack trace and unobvious error
- mmlu breaks logging of caller
- mmlu isn't consuming multiple gpus HOT 1
- `test_branch_gen_answers` does not fail when no model is being served HOT 1
- Allow usage of GPT-4 as judge model
- MTBench Branch evaluator breaks when it encounters av3 taxonomy qna.yaml HOT 3
- Request: max_workers auto tuning
- Auto detect mixtral type judge models and auto set merge_system_user_message
- Convert E2E job to use AWS runners
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from eval.