Comments (6)
Are you willing to elaborate on that, or did you make a descriptionless issue to just say sftgan's got bugs?
from trainner.
It wouldn't surprise me if sftgan is broken since nobody uses it and it's just been carried over from the original basicsr without any updates
from trainner.
Hello! Yes, the SFTGAN code may not be fully functional. It was abandoned by the original author and I have barely been keeping updated with the rest of the changes in the repository, it is waiting for a major rewrite, but as Joey mentioned, nobody used it and the results weren't much better than the models trained without the SFT layers.
The SPADE normalization layers in the project by the same name is based on the same theory as SFT and if I finish integrating Pix2pixHD, I planned to follow with SPADE and then update SFTGAN.
If you're interested, you could troubleshoot the issues and get it to work, but this original instance of SFTGAN depends on some external code that has been deprecated.
from trainner.
Hi@victorca25 , thank you for the comments, yes, the original sftgan was not fully functional, the author just released ver 0.0, some part of code are missing.
I am now debugging mainly on your code, and now it could be run without validation part(still need to be modified on the dataloader).
BTW, what is SPADE normalization layers? anyway, I think the SFT idea is great, because low resolution images are opt to be similar with each other, even if they are resized(downsize) from different HR images, so I think image prior is necessary
from trainner.
Luckily the validation part is not necessary for training and you can evaluate the model outputs manually offline, so at least it's possible to test it.
Regarding SPADE, this is the paper: https://arxiv.org/pdf/1903.07291.pdf and this is the original repo: https://github.com/NVlabs/SPADE. This version of the code is based on pix2pixHD, which is based on the "pix2pix and CycleGAN" code, which this repository also was originally based on, so it should be familiar to browse around.
The SPADE normalization layers also make use of segmentation maps for spatial conditioning like SFT layers as both are spatial feature transforms that are applied through the network in place of regular normalization layers (like batch or instance normalization). It can be a good reference for updating the SFTGAN codebase if needed, so much so that certain things could be combined (which was my original intention in leaving updating SFTGAN for later).
The deprecated code I referred to is the original code used to train the model to generate the segmentation maps for SFTGAN: https://github.com/lxx1991/caffe_mpi , since there are now better and more accurate models to produce them. Alternatively, it would be possible to use datasets with manually created segmentation maps like in the pix2pixHD and SPADE cases (taking into consideration the lower resolution on the input domain), and there is a chance this can help get even better results from SFTGAN than using the outputs from the segmentation model.
from trainner.
Hi@victorca25 , Thank so much for your feedback and kind information, I trained SFTGAN and test offline, the result is a little bit different with xintao's default model, my results are a little bit blur, which means the details are not reproduced well, not sure if it is because of the training hyperparameters, I used the same setting with the default one, the only difference is that in the default LRHRSeg_BG_Dataset they used both background image(with seg=1) and OST dataset(seg with pretrained model), I am not sure why xintao want to use DIV2K as background image, so I just used the OST.
SPADE normalization layer is a good idea, I will read/try it, and then come back here.
from trainner.
Related Issues (20)
- "CUDA out of memory. Tried to allocate 1.48 GiB" when trying to validate HOT 2
- [Feature Request] Curriculum Training for Augmentations
- Pixel Unshuffle is broken HOT 2
- Video dataloader crashes at 1x scale
- Video learning rate too high
- Add lr_crop_size in config HOT 1
- Update requirements.txt with proper versioning for torch
- Feature request/bug fix: Perform scaling and other operations in linear light HOT 3
- `nearest_aligned` is not aligned HOT 2
- How do i even use this HOT 2
- Is there any way to train video super resolution models using this? HOT 4
- lmdb has no valid image file HOT 3
- how to train for real ESRGAN HOT 1
- GPU usage at 0% during training HOT 7
- FutureWarning and UserWarning HOT 5
- I Have No Idea What I Am Doing to Cause This: HOT 5
- "cv2.error: OpenCV(4.7.0) D:\a\opencv-python\opencv-python\opencv\modules\imgproc\src\resize.cpp:4065: error: (-215:Assertion failed) inv_scale_x > 0 in function 'cv::resize'" HOT 1
- ETA is grossly over-estimated. HOT 2
- Pix2Pix 3->1 channel HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from trainner.