Comments (4)
Hi! Sorry for the leate reply!
You can try to turn off the validation - then the pipeline would use less gpu memory and load will be a bit more balanced. However, as far as I know, this is a usual issue with DDP - it always occupies more memory on gpu #0 than on others. If you want to achieve equal memory consumption, try horovod - it should be pretty easy to set up with pytorch-lightning (however, I did not try it myself)
from lama.
This link might be also useful
from lama.
You can try to turn off the validation
how to turn off the validation? we should update .yaml?
from lama.
Just set very long period of validation, by adjusting .yaml
or by adding a command line argument to train.py, e.g. trainer.kwargs.check_val_every_n_epoch=10000
However, when turning validation off you have to adjust parameters for saving the checkpoints (see configs/trainer/*.yaml)
Instead of this
checkpoint_kwargs:
verbose: True
save_top_k: 5
save_last: True
period: 1
monitor: val_lpips_fid100_f1_total_mean
mode: max
Write this:
checkpoint_kwargs
verbose: True
period: 1
As a result, you'll have a model dumped after each epoch - so you'll have to score them afterwards and choose the best one.
from lama.
Related Issues (20)
- Can I separate the Feature Refinement to Improve the High-Resolution Image Inpainting technique
- A simple ckpt to pt model convertor
- Repeated Refinement?
- Error finetuning the big-lama-with-discr model HOT 7
- Data set training problem HOT 1
- After executing the training command, it has been stuck at this point without any progress in the training. HOT 1
- Inpaint a NEW thing? HOT 3
- Refinement with Multiple Images
- How to draw a loss function curve
- Dataset is empty if configuring img_suffix: .jpg in default.yaml
- ONNX Model done HOT 10
- Output Error: No inpainted in the output_dir HOT 1
- Can't install at image in Docker
- The completion effect is not good? HOT 1
- Does Llama support inpainting a given image onto the original image as opposed to just removing from the original image
- Can't find dataloader , no outputs HOT 1
- Is there a way to find out the result is quality?
- nccl time out HOT 1
- Supplementary materials link does not work HOT 1
- Post-processing for LaMa Model Output to Increase Quality
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from lama.