Comments (10)
Likewise, trying to find a fix now.
My process keeps getting killed, even when running with 25GB RAM on Google Colab.
from generative_mlzsl.
@hugh920 could you let me know if you find a fix in the meantime 👍
from generative_mlzsl.
@dgbarclay When you train, does your memory increase with each epoch?How many epochs have you reached so far?
from generative_mlzsl.
@hugh920 Mine is being killed whilst parsing the data, it doesn't reach the beginning of training. It seems to fall within the block on line 203 of util.py. Are you able to begin training? Have you modified the code?
from generative_mlzsl.
@dgbarclay Mine can be trained without modifying the code. However, due to increased memory, it failed in the second round. I modified batch_size and made the model structure a little simpler so that he could continue to run. I noticed that the memory increased during the first two training rounds and stabilized after the third. I don't understand why.
from generative_mlzsl.
@hugh920 okay, I have not yet made it that far. I was running out of memory during forming the DataLoader so I'm having to refactor a little bit. Are you able to push your version so I can compare the two? It would help me out loads, cheers.
from generative_mlzsl.
@hugh920 are you able to run eval_nus_wide.sh without failure? I ultimately just need to be able to run this model to take image queries and give predictions, are you able to get the model in that state?
from generative_mlzsl.
@dgbarclay I took the ALF out and just used FLF, which didn't work well. It may not be what you need.
from generative_mlzsl.
@dgbarclay I also had a problem with processes being killed while loading data on other projects today. I have observed that the GPU is not utilized when loading data. It is a dataloader made by CPU, probably because the processing power of CPU is not up to it.It has nothing to do with memory size or GPU.
from generative_mlzsl.
Is the issue solved?
from generative_mlzsl.
Related Issues (16)
- Evaluation File HOT 11
- nus_zsl_test_vgg19.h5 vs s,nus_gzsl_test_vgg19.h5_features
- DataLoading CPU Hangs
- NUS_WIDE_pretrained_w2v_glove-wiki-gigaword-300 HOT 2
- Production of qualitative results HOT 4
- Urgent Query Regarding datasets
- About nus-wide dataset
- Links for downloading datasets, NUS_WIDE and OPENIMage HOT 2
- Memory problem HOT 4
- get_extract_data not defined HOT 3
- Google drive link not working HOT 1
- Training using NUS-WIDE dataset:gdrive not accessible HOT 2
- How much gpu memory does it costs? HOT 1
- dataset issue HOT 5
- Classification Results/Detection Results HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from generative_mlzsl.