Comments (5)
You can lower the learning rate of the threshold encoder in config.py
, such as 1e-7.
if __C.OPT == 'Adam':
__C.LR_BASE_NET = 1e-5 # learning rate
__C.LR_BM_NET = 1e-7 #1e-6 # learning rate'
Thanks for your attention!
from iim.
adjust the learning rate does works! Thanks for your reply.
from iim.
Hi taohan@taohan10200 , after lowing the learning rate, NAN still appeared after 87 iterations. I saved the model and weights every 20 iterations, and felt amazed that based on 80th model, the model can be trained normally without NAN. Do you have any good suggestions?
By the way, there is no read_pred_and_gt module in misc.utils.py, causes vis4val.py cannot work properly, would you please commit this part codes?Thanks。
from iim.
In our training, NaN would appear even if we lowered the threshold some times. At this time, we usually lower the threshold again to avoid this problem. We recommend using the experimental configuration we provide under folder saved_exp_results
. In general, it may be the inverse gradients that make the module's training is instability. We have tried to solve this problem by optimizing the threshold learner, but it is still in testing, and we will update the new solution in the future.
We have updated the read_pred_and_gt module in misc.utils.py
.
Thanks~
from iim.
where is the path of the save model?
from iim.
Related Issues (20)
- 我在training時遇到以下錯誤 HOT 1
- 训练时有足够内存,依然得到out of memory问题 HOT 1
- 训练时有足够得内存,依然得到out of memory问题
- 模型保存为pt时报错RuntimeError: Could not export Python function call 'BinarizedF' HOT 1
- Great work but Threshold Encoder module is useless.Details look at the picture HOT 3
- How to visualize other datasets test set pictures? HOT 7
- 怎样转化为onnx模型 HOT 3
- threshold appear to be nan during the training process HOT 2
- RuntimeError: CUDA error: out of memory CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. HOT 1
- How to config to train on NVIDIA GeForce RTX 3090 Ti/PCIe/SSE2 HOT 6
- Training on general dataset HOT 2
- Could you please give me demo code, I mean real-time testing code?
- Memory allocation error HOT 1
- Confidence level? HOT 8
- 主页给的HRNet权重链接hrnetv2_w48_imagenet_pretrained.pth和Crowd_locator中模型结构不太一样 HOT 1
- 请问mask_50_60具体有什么作用?运行prepare_NWPU会出现jsons文件缺失错误(缺失的是测试图片的id,所以肯定是没有的) HOT 1
- 您好 我想问下每个数据集给出的mean_std是如何计算的,和我自己计算的结果不太相同。
- segfault with pytorch 1.17 and 2.0.0 HOT 1
- FileNotFoundError: [Errno 2] No such file or directory: '../ProcessedData/JHU\\train.txt'
- input video HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from iim.