Comments (3)
Thank you for your details. I also realized that the sampling temperature might cause this variation.
from open-instruct.
In addition, I also can't reproduce the reported F1 score for TdiQA, I obtain 44.6 rather than 45.7.
Scores:
{
"russian": {
"exact_match": 43.0,
"f1": 61.294599844599816
},
"indonesian": {
"exact_match": 44.0,
"f1": 66.59124435225426
},
"swahili": {
"exact_match": 22.0,
"f1": 36.53859841982154
},
"arabic": {
"exact_match": 24.0,
"f1": 40.90610851423544
},
"english": {
"exact_match": 63.0,
"f1": 79.64825824035252
},
"finnish": {
"exact_match": 42.0,
"f1": 60.03515278266136
},
"bengali": {
"exact_match": 4.0,
"f1": 9.64920634920635
},
"korean": {
"exact_match": 37.0,
"f1": 46.19523809523808
},
"telugu": {
"exact_match": 0.0,
"f1": 0.6190476190476191
},
"average": {
"f1": 44.60860602415744,
"exact_match": 31.0
}
}
from open-instruct.
Hi,
- EM = exact match = accuracy as reported in our paper. 44.4 is pretty close to the 44.5 we report for Tulu 7b, the difference may be due to the diff process or some other small difference in our setups.
- TydiQA has some variation based on model generations. I reran the script just now and got [43.4, 45.0, 44.3]. Considering your result is reasonably close to our reported result, I'm confident in saying that this is the result of this variation. The command to use to replicate these numbers is:
python -m eval.tydiqa.run_eval --data_dir <data_dir> --n_shot 1 --max_num_examples_per_lang 100 --model_name_or_path tulu-7b --tokenizer_name_or_path tulu-7b --max_context_length 512 --eval_batch_size 8 --use_chat_format --load_in_8bit
In general, I think as many of our evaluations are generation-based, there may be some variation between our reported results and the results you get from reproduction. If these differences are small (e.g. < ~1-2 points), then it should be okay.
from open-instruct.
Related Issues (20)
- Question about the decoding settings for alpaca_eval HOT 1
- Question about the evaluation of GSM8K HOT 1
- Potential bug for data format HOT 1
- Training script or params for TULU2-13B. Cannot reproduce TULU2 fine-tuned model HOT 6
- 微调报错 HOT 1
- flan-v2 data HOT 2
- QLoRA is very slow HOT 3
- wrong tokenization about eos token HOT 5
- mmlu run_eval.py: difference in evaluation results between run_eval.py(average_acc: 27.4) and normal results: (llama2-hf: 41.8)
- The alpaca eval is not working can't reproduce HOT 3
- Possible unreliability of MMLU estimates HOT 1
- about tulu v2 conversion HOT 2
- balanced_low_0 Related Issue During Inference HOT 1
- model weights after finetune can not load by vllm or huggingface HOT 1
- Tulu v2 Sanky Diagram HOT 4
- Support for tuning on chat data HOT 1
- File Not Found error while running finetune_lora_with_accelerate.sh HOT 1
- Embedding resizing logic may be flawed HOT 1
- [Issue with solution] Merging qlora error: open_instruct/merge_lora.py:25 TypeError: 'QuantState' object does not support item assignment HOT 1
- Seeking for Help: how to work deepspeed zero stage 3 with quantized reference model? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from open-instruct.