Comments (14)
@sacmehta I have the same issues with you, not only validation loss, sometimes the training loss occurs inf of Average Loss, Average Regression Loss, but the classification loss continues to decline, how do you solve it? Thank you
from pytorch-ssd.
Hi @sacmehta , are you using your own dataset? The log says the regression loss is Inf. The regression loss here is Smooth L1 Loss. It might be helpful if you check out some input data and intermediate values. Plz also reference the implementation is PyTorch.
https://pytorch.org/docs/stable/nn.html#torch.nn.SmoothL1Loss .
from pytorch-ssd.
I am able to fix the above issue, but now I am getting another issue. Loss is not decreasing. Total loss is fixed at around 12. Any idea what might go wrong?
P.S.: I am using PyTorch weights and not Caffe weights.
from pytorch-ssd.
Hi @sacmehta , have you tried a smaller learning rate? Also, try a small subset of the training data to verify the process is right. If the process is all right, you should get a overfitted model with 0 loss.
from pytorch-ssd.
@sacmehta Hi, are you able to share your pretrained PyTorch ImageNet weights? I can try to reproduce it, since I am working on a similar project.
from pytorch-ssd.
@AlanStark If you use PyTorch’s vision api, you should be able to download them by setting the pretrained argument to True.
from pytorch-ssd.
Oh, I see. I was thinking something different.
from pytorch-ssd.
@qfgaohao Do you have any training logs for MobileNet?
from pytorch-ssd.
I am able to fix the above issue, but now I am getting another issue. Loss is not decreasing. Total loss is fixed at around 12. Any idea what might go wrong?
P.S.: I am using PyTorch weights and not Caffe weights.
Hi, @sacmehta, how do you fix the inf problem, I am lost in it.
from pytorch-ssd.
Make sure the feature map size used for prior generation is the same as feature maps from CNN used for SSD
from pytorch-ssd.
@sacmehta ,thanks a lot.
I have solved the problem, because my training data has very small boxes, so the smoothed l1 loss(log(0)=-inf) become -Inf.
from pytorch-ssd.
Well, I rewrote most of the SSD code from here:
https://github.com/amdegroot/ssd.pytorch
You can find my implementation here and see if it helps:
https://github.com/sacmehta/EdgeNets
from pytorch-ssd.
Thanks for your reply, could you tell me what caused the inf loss? I have modified network structure , So I want to be able to locate errors in my code.
from pytorch-ssd.
I don’t remember it. It’s been a while. You can try to plug-in your model in my codebase and see if that helps.
from pytorch-ssd.
Related Issues (20)
- Training doesn't start - I'm getting an error with the data loader HOT 4
- convert_to_caffe2_models.py: RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! HOT 2
- SSDSpec for image size smaller than 300 HOT 1
- Re-training SSD-Mobilenet - loss going up and down
- Error using run_ssd_live_demo.py HOT 1
- [Question] what these parameters mean? HOT 2
- base net for mobilenetv3 ssd HOT 1
- ssd-mobilenet to tensorrt
- pytorch模型量化时报错
- Mobilnet-ssdv2 HOT 1
- NAN values in Boxes HOT 1
- How can I train the model without pre-trained weight specified? HOT 1
- runtime error mobillenet-ssd-v2 HOT 5
- run_ssd_example.py box variable type does not mach opencv rectangle and putText functions
- FileNotFoundError: [Errno 2] No such file or directory: 'models/mobilenet_v1_with_relu_69_5.pth'
- cv2.error: OpenCV(4.5.5) : -1 : error: (-5:Bad argument) in function 'rectangle' HOT 1
- Could u add Nesterov momentum in SGD
- 4xgx
- RuntimeError: The size of tensor a (12828) must match the size of tensor b (3000) at non-singleton dimension 1 HOT 3
- ValueError encountered during retraining on Open Images Dataset HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pytorch-ssd.