Comments (17)
@zdx3578
1 if you're using the GPU instance on AWS, then please change the architecture setting into:
# CUDA architecture setting: going with all of them.
# For CUDA < 6.0, comment the *_50 lines for compatibility.
CUDA_ARCH := -gencode arch=compute_30,code=sm_30 \
-gencode arch=compute_50,code=sm_50 \
-gencode arch=compute_50,code=compute_50
Because the GPU in AWS does not support compute_35
2 Change sm_35 into sm_30 in lib/setup.py file
3 cd lib, remove these files: utils/bbox.c nms/cpu_nms.c nms/gpu_nms.cpp, if they exist.
And then make && cd ../caffe/ && make clean && make -j8 && make pycaffe -j8
from py-faster-rcnn.
@zdx3578 It seems you don't have enough GPU memories. You can try the smaller ZF model.
from py-faster-rcnn.
@twtygqyy: Implementing your suggestion + using --net=zf
allowed me to run the demo. ./tools/demo.py
still segfaults for the default run due to not enough memory. Is it really the case that the g2.8xlarge instance isn't powerful enough to run this? I had the same error output as @zdx3578
from py-faster-rcnn.
@aaronpolhamus the memory of AWS instance should be enough to run or even train the model with CUDNN (the most recent version is CUDNN v5) installed
from py-faster-rcnn.
@aaronpolhamus I forget to mention that the caffe version of this faster-rcnn repo can support CUDNN version <= v4
from py-faster-rcnn.
@twtygqyy: the default run ./tools/demo.py
still fails. Here's the error output:
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:505] Reading dangerously large protocol message. If the message turns out to be larger than 2147483647 bytes, parsing will be halted for security reasons. To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:78] The total number of bytes read was 548317115
I0529 20:02:37.205232 3760 net.cpp:816] Ignoring source layer data
I0529 20:02:37.314342 3760 net.cpp:816] Ignoring source layer drop6
I0529 20:02:37.329881 3760 net.cpp:816] Ignoring source layer drop7
I0529 20:02:37.329957 3760 net.cpp:816] Ignoring source layer fc7_drop7_0_split
I0529 20:02:37.330417 3760 net.cpp:816] Ignoring source layer loss_cls
I0529 20:02:37.330440 3760 net.cpp:816] Ignoring source layer loss_bbox
I0529 20:02:37.332777 3760 net.cpp:816] Ignoring source layer silence_rpn_cls_score
I0529 20:02:37.332852 3760 net.cpp:816] Ignoring source layer silence_rpn_bbox_pred
Loaded network /home/ubuntu/py-faster- rcnn/data/faster_rcnn_models/VGG16_faster_rcnn_final.caffemodel
F0529 20:02:37.756431 3760 syncedmem.cpp:56] Check failed: error == cudaSuccess (2 vs. 0) out of memory
*** Check failure stack trace: ***
./tools/demo.py --net=zf
succeeds, however. To your comments, which version of cuDNN should I be using for the py-faster-rcnn caffe install? I'm pretty sure that I'm using the most recent version now (cudnn-7.5-linux-x64-v5.0-ga.tgz
), but the installation barfed when I tried to compile with USE_CUDNN := 1
. I was able to successfully build caffe inside py-faster-rcnn and run the demo as ./tools/demo.py --net=zf
when I commented this flag out.
So two questions for you:
- Can I expect a successful install if I use a version of cuDNN that is <= 4?
- Once I configure with cuDNN, can I expect to be able to run the default demo example without running in to a memory error?
from py-faster-rcnn.
@aaronpolhamus According to the problem you described, using cuDNN v4 should solve the memory issue, because the caffe repo of faster-rcnn is the version of 2016 Feb, before the release of cuDNN v5. If you want to use v5, you have to update the caffe fork.
from py-faster-rcnn.
@twtygqyy: finally worked it out. not only do you need cuDNN v 3 or 4, but you also need to be running CUDA v7.0, rather than 7.5
from py-faster-rcnn.
@aaronpolhamus: I use CUDA v7.5 + cudnn v4 looks fine.
from py-faster-rcnn.
@ricepot100: that's really interesting. To get everything running I had to revert to the earlier versions I mention above. Are you on the g2.8xlarge instance?
from py-faster-rcnn.
@aaronpolhamus: No, I don't use EC2 but a local machine. 980GT 6G DDR
from py-faster-rcnn.
when i try install fast rcnn than i got like this error? how to slove it?
i use gtx 970 so i think enough memory? isnn't it?
Loaded network /home/rvlab/Music/fast-rcnn/data/fast_rcnn_models/vgg16_fast_rcnn_iter_40000.caffemodel
Demo for data/demo/000004.jpg
F0718 22:09:35.547049 13693 syncedmem.cpp:51] Check failed: error == cudaSuccess (2 vs. 0) out of memory
*** Check failure stack trace: ***
Aborted (core dumped)
from py-faster-rcnn.
@ricepot100 You wrote that "I use CUDA v7.5 + cudnn v4 looks fine".
However, on NVIDIA's web site, the download for cuDNN is called cuDNN-7.0-linux-x64-v4.0-prod.tgz
. In other words, version 4 of cuDNN is meant to work with version 7.0 of cuDNN.
from py-faster-rcnn.
@isarker as it mentioned "out of memory" , actually vgg16 takes more than 3G memory of CUDA for training, suggest you to use a GTX Titan X with 12G, :)
https://github.com/rbgirshick/py-faster-rcnn#requirements-hardware
from py-faster-rcnn.
@zhouphd thanks for your reply, i changed my gpu and it's works.
from py-faster-rcnn.
I had the same problem, but for me downgrading CUDNN to 4 was enough to solve it.
My configuration is Ubuntu 16.04, Cuda Toolkit 8.0, CUDNN 4.0. Apparently you don't need to downgrade the cuda toolkit and drivers to release 7.
The demo.py completed successfully on a 2Gb GeForce GTX 950.
from py-faster-rcnn.
Hi @Vandertic
I changed my CUDNN version to a 4, but I still get the same error, was there something else you did? I'm now using any AWS instance, so should that be a problem?
from py-faster-rcnn.
Related Issues (20)
- Error when run the demo
- 关于faster rcnn, 已经解决了win10下关于faster rcnn下训练自己数据集的问题
- error while making caffe HOT 1
- gpu_nms does not work. HOT 2
- Why do an operation that is repeated and may bring errors?
- win10 nms_gpu nvcc fatal : Host compiler targets unsupported OS.
- Anchors
- Need help in creating dataset class for multilabel object detection in a single image
- How to remove false detection (False Positives) in Faster RCNN
- How to add background as the training data? The background image is no goals. HOT 4
- so i have to compile cython with cuda or nvcc ?? HOT 2
- how to solve "raise EnvironmentError('The nvcc binary could not be ' OSError: The nvcc binary could not be located in your $PATH. Either add it to your path, or set $CUDAHOME make: *** [all] Error 1"
- difference between fast_rcnn, faster_rcnn_alt_opt and faster_rcnn_end2end
- image invalid, skipping
- continue training with .caffemodel HOT 1
- 2022
- two sibling layers in RPN are fully connected or 1x1 conv layers? HOT 2
- how comput stride size of vgg outputs?
- Object Detection Failure in FRCNN
- Do I need to install caffe if I just want to run demo.py? Or I can just start building the Cython modules?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from py-faster-rcnn.