Comments (4)
Thanks for your attention to our work!
This error is caused by DeepSpeed-zero3 (They do not support backward in an evaluation mode). We need the eval mode because the MiniLLM training is similar to the RL training where dropout is usually not applied.
If you use zero3 because of insufficient GPU memory, you can try using zero-offload by setting the DeepSpeed config to the corresponding path .
If you insist on using zero3, you can set the dropout value to 0 in the model config files before running the script and add self.model.train()
before every step (for example, at this line). In this way, you can avoid using dropout and do backward with the training mode at the same time.
from lmops.
I was using your deepspeed config file for zero2 offload, which I changed to zero3 and added parameter offload. If I understand you correctly, optimizer offload with zero2 works, but adding parameter offload and using zero3 is not yet implemented?
Which model config file are you referring to exactly which contains the dropout value?
I managed to run llama7B_13B minillm with zero2 on a single H100 GPU by reducing the max sequence length to 256. In your opinion, will this degrade performance significantly?
Thank you for your help!
from lmops.
I was using your deepspeed config file for zero2 offload, which I changed to zero3 and added parameter offload. If I understand you correctly, optimizer offload with zero2 works, but adding parameter offload and using zero3 is not yet implemented?
Yes.
The "model config file" refers to the huggingface config files in the checkpoint directory you load from.
I think reducing the max sequence length to 256 will mainly affect the performance of long instructions/responses. On short instructions, the performance will not be affected much.
from lmops.
Alright, thank you for your help!
from lmops.
Related Issues (20)
- [MiniLLM] Got different results from validation during training and testing after training using the same data HOT 3
- No module named 'transformers.models.qwen_parallel.utils_qwen' HOT 4
- forward() got an unexpected keyword argument `token_type_ids` HOT 1
- [llm_retriever] Questions about the result
- bash install.sh的时候出错 HOT 6
- [MiniLLM] Processed RoBERTa Corpus dataset download HOT 1
- Retrieve Ckpt Download HOT 2
- DPR trian error
- llama version in Minillm HOT 11
- a lot of curse words in code HOT 1
- Is reward_fn equal to log_softmax HOT 2
- [AdaptLLM] How to evaluate the models' performance? HOT 3
- What is the python version of the uprise?
- What is the python version of uprise? HOT 1
- A problem occur when execute the first procedure. HOT 13
- 训练过程中CUDA out of memory HOT 10
- [UPRISE]CUDA out of memory. Tried to allocate 3.25 GiB. GPU HOT 4
- [UPRISE]When training the uprise,a problem happened. HOT 17
- [MiniLLM] llama3-instruct版本蒸馏 HOT 1
- [UPRISE}When use llm to score the prompt,the connection stopped. HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from lmops.