Comments (2)
python2 finetuning seems good, but Resource exhausted: OOM(P100)
19-09-27 14:29:48.886518: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba12398600 next 2020 of size 5120
2019-09-27 14:29:48.886523: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba12399a00 next 2021 of size 5120
2019-09-27 14:29:48.886527: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba1239ae00 next 2022 of size 6553600
2019-09-27 14:29:48.886538: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba129dae00 next 2023 of size 6553600
2019-09-27 14:29:48.886543: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba1301ae00 next 2024 of size 5120
2019-09-27 14:29:48.886548: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba1301c200 next 2025 of size 41943040
2019-09-27 14:29:48.886552: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba1581c200 next 2026 of size 6553600
2019-09-27 14:29:48.886557: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba15e5c200 next 2027 of size 5120
2019-09-27 14:29:48.886561: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba15e5d600 next 2028 of size 6553600
2019-09-27 14:29:48.886566: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba1649d600 next 2029 of size 5120
2019-09-27 14:29:48.886570: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba1649ea00 next 2030 of size 5120
2019-09-27 14:29:48.886575: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba1649fe00 next 2031 of size 5120
2019-09-27 14:29:48.886579: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba164a1200 next 2032 of size 5120
2019-09-27 14:29:48.886584: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba164a2600 next 2033 of size 5120
2019-09-27 14:29:48.886588: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba164a3a00 next 2034 of size 6553600
2019-09-27 14:29:48.886593: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba16ae3a00 next 2035 of size 6553600
2019-09-27 14:29:48.886597: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba17123a00 next 2036 of size 41943040
2019-09-27 14:29:48.886602: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba19923a00 next 2037 of size 5120
2019-09-27 14:29:48.886606: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba19924e00 next 2038 of size 6553600
2019-09-27 14:29:48.886611: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba19f64e00 next 2039 of size 41943040
2019-09-27 14:29:48.886615: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba1c764e00 next 2040 of size 41943040
2019-09-27 14:29:48.886620: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba1ef64e00 next 2041 of size 5120
2019-09-27 14:29:48.886624: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba1ef66200 next 2042 of size 5120
2019-09-27 14:29:48.886629: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x7fba1ef67600 next 2043 of size 5120
from ctrl.
It's a bit difficult to debug since I don't have the whole log file but from what you've posted, it seems like you didn't (re-?)patch keras.py
to have use_tpu=False
?
from ctrl.
Related Issues (20)
- Using ctrl for summarization HOT 2
- TPU configuration - fine tuning
- Is that a way to do "general" generation? HOT 1
- Source attribution - Cannot replicate results
- why set "seq_length = min(args.generate_num, 256)" HOT 1
- training curriculum used
- repeats the last word on AWS HOT 1
- License for pre-trained model
- Sampling method used for translation
- tips and scripts related to data collection HOT 1
- Are control codes required for finetuning?
- Altering the tone of the output
- Will BERT+transformer-decoder better than tensor2tensor for text-generation?
- control code not recognised HOT 2
- Issues with pytorch_generation.py when running the Colab exercise
- CTRL model can not work in huggingface transformers HOT 2
- 12 layer (huggingface gpt-2 equivalent) ctrl model?
- Cuda out of memory issue.
- A transformer decoer-based model or seq2seq model?
- control code
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ctrl.