ricardorei / lightning-text-classification Goto Github PK
View Code? Open in Web Editor NEWMinimalist implementation of a BERT Sentence Classifier with PyTorch Lightning, Transformers and PyTorch-NLP.
Minimalist implementation of a BERT Sentence Classifier with PyTorch Lightning, Transformers and PyTorch-NLP.
I'm having CUDA out of memory
even with --batch_size 1
. But it raises the error only after the first epoch. Any idea/advice on to solve such an issue?
Hi,
I really like how you combined PyTorch Lightning with the Transformers library. I tried to upgrade all dependencies to the latest version. There are some smaller issues, like renamed keyword arguments in PyTorch Lightning and so on. Eventually, I face the following error message.
Error:
AttributeError: Can't pickle local object 'LayerSummary._register_hook..hook'
Environment:
python = "^3.7"
torch = "^1.5.1"
torchvision = "^0.6.1"
transformers = "^2.11.0"
pytorch-lightning = "^0.8.1"
pytorch-nlp = "^0.5.0"
test-tube = "^0.7.5"
pandas = "^1.0.5"
sklearn = "^0.0"
Do you have any idea what the error might be? Is it a Lightning problem or a transformers problem? Any way to fix it?
Best
Dominique
Currently an error is thrown when using multi-GPU's and 16 bit precision.
To setup this, the following hparams are added to trainer:
parser.add_argument("--gpus", type=str, default='0,1,2', help="Which gpus")
parser.add_argument('--precision', default=16, type=int)
trainer=Trainer(
...
precision=hparams.precision,
Error trace:
File "/path_to/lightning-text-classification/classifier.py", line 183, in forward
word_embeddings = self.bert(tokens, mask)[0]
File "/home/me/.virtualenvs/sbert-env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/me/.virtualenvs/sbert-env/lib/python3.8/site-packages/transformers/modeling_bert.py", line 824, in forward
embedding_output = self.embeddings(
File "/home/me/.virtualenvs/sbert-env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/me/.virtualenvs/sbert-env/lib/python3.8/site-packages/transformers/modeling_bert.py", line 207, in forward
inputs_embeds = self.word_embeddings(input_ids)
File "/home/me/.virtualenvs/sbert-env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/me/.virtualenvs/sbert-env/lib/python3.8/site-packages/torch/nn/modules/sparse.py", line 124, in forward
return F.embedding(
File "/home/me/.virtualenvs/sbert-env/lib/python3.8/site-packages/torch/nn/functional.py", line 1814, in embedding
return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: arguments are located on different GPUs at /pytorch/aten/src/THC/generic/THCTensorIndex.cu:403
Checking this, at the call: word_embeddings = self.bert(tokens, mask)[0], both tokens and mask are on the same GPU, it appears an issue with the embeddings that I haven't been able to isolate exactly where yet.
Running with multi GPU and 32 bit precision works fine, as does 1 GPU and 16 bit precision. Error occurs with both dp and dpp distributed modes.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.