Comments (3)
Greetings!
Regarding the test results on PTB after pruning, the reason for the worse score lies in the unpruned llama2-7B model, which was approximately 47, significantly higher than llama-7B (~22) on PTB.
As for the issue of NaN during post-training, we encountered the same problem you reported. Currently, we are searching for the appropriate hyper-parameters to fine-tune the pruned model. If we obtain any new findings or find any bugs in our code, we will promptly update you.
from llm-pruner.
Thank you for the timely reply! Hope to get back with good news.
Cheers.
from llm-pruner.
@mmichaelzhang Have you resolved this issue? I also observed training loss explosion and encountered performance deterioration for llama2-7b using default llama settings:
Wikitext2 w/o tune | Ptb w/o tune | BoolQ Acc | PIQA Acc | HellaSwag Acc_norm | WinoGrande Acc | ARC-e acc | ARC-c Acc_norm | OBQA Acc_norm |
---|---|---|---|---|---|---|---|---|
19.24 | 72.61 | 37.83 | 52.34 | 26.64 | 49.41 | 25.08 | 27.82 | 28.40 |
As pruned model weights is quantized by int8 and frozen for post-training, I think the phenomenon is non-related with BF16/FP16 dtype, which is considered as the cause by authors:
from llm-pruner.
Related Issues (20)
- Question related to the model tuning HOT 2
- Pruning MQA?
- 在将部分层进行剪枝之后,不能直接通过tgi加载模型
- Adding a tutorial for adapting new models?
- 401 Client Error: Unauthorized for url: https://huggingface.co/decapoda-research/llama-7b-hf/resolve/main/tokenizer_config.json HOT 1
- cannot import name 'SiLUActivation' from 'transformers.activations' HOT 1
- Issue: Missing Generation of `pytorch_model.bin` File During Model Tuning HOT 3
- Cannot use huggface to load
- OSError: Can't load tokenizer for 'baffo32/decapoda-research-llama-7B-hf'. HOT 1
- ConnectionError: Couldn't reach https://raw.githubusercontent.com/wojzaremba/lstm/master/data/ptb.train.txt (ReadTimeout(ReadTimeoutError("HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Read timed out. (read timeout=100)"))) HOT 1
- The quantization of the compressed models
- 延迟评估 HOT 1
- 剪枝率值的问题
- Unable to reproduce the results for param_first and param_second in the paper after finetuning.
- RecursionError: maximum recursion depth exceeded HOT 1
- Is this method implementable on multi-GPUs?
- How to prune the embedding and lm_head?
- I tired Mistral 7b model, but I got this issue
- Pruning llama3
- Evaluation:UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from llm-pruner.