Comments (5)
Right now,
8*80GB A100
or16*40GB A100
[GPUs]. With the "accelerate" library you have offloading though so as long as you have enough RAM or even just disk for 300GB you're good to go (but slower).
Source: https://www.infoq.com/news/2022/07/bigscience-bloom-nlp-ai/
According to this post you can run it on consumer hardware at 3 minutes/token.
According to this post even on pretty good GPU hardware it can take 90 seconds/token though. Seems like you need really upper range systems to run it quickly.
from bigscience.
For inference only, what are the minimum requirements for RAM and GPU memories?
from bigscience.
About 350 GB of GPU RAM (~200 GB if you quantise to int8).
from bigscience.
About 350 GB of GPU RAM (~200 GB if you quantise to int8).
For inference only?
from bigscience.
Yep, need to get all those parameters into GPU RAM to run inference. Like I mentioned, you can use the accelerate framework to do "swapping" from CPU RAM to GPU RAM, which lets you do it with much less GPU RAM at a ridiculous speed penalty.
from bigscience.
Related Issues (19)
- Is the 13B - unmodified Megatron gpt2 - baseline available? ( tr1-13B-base) HOT 1
- Wrong tokenizer path in big model config HOT 1
- make a back up for final training data HOT 1
- can you share the slurm.conf you are using? HOT 3
- Sharing the 1.3B-Pile@300B model
- Zero_Stage=1 results in higher TFLOPS? HOT 1
- About training data for 1B3 models
- Fill in request for the second half of compute
- What is the number of epochs of the final training?
- mC4 sampling & pre-processing HOT 1
- Requirements to perform inference over the BigScience Bloom model
- eval opt-175B HOT 1
- How to get train-splits.txt and valid-splits.txt before training tr11-176B-ml HOT 1
- where can we get a bloomz-7b1 finetuned checkpoint
- Files for bias evaluation
- Where can I download the training script for bloom-7b1?
- why 384(12*2*16) will be the first time all pipeline stages be filled
- Why is deepspeed enabled in the Bloom training script?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bigscience.