Comments (6)
For some reason your image has negative values, which usually should not happen. Maybe it was pre-processed, for example, cubic interpolation can produce negative values. If they are just slightly below zero, you can clamp them to zero.
from fastsurfer.
The error happens right at the start.
- Are you using the latest release official docker image (version 2.2.0) converted to singularity?
That was the problem. I re-run singularity build fastsurfer-gpu.sif docker://deepmi/fastsurfer:latest
and now it works! 3' 75'' sec to get till the end with --seg_only
. Thank you very much!
- And also what is the resolution of your input image?
- Are you running anything else on the GPU? Maybe on the host system (window manager. ...)?
from fastsurfer.
Thanks!
It runs now, but I get the following error:
RuntimeError: CUDA out of memory. Tried to allocate 256.00 MiB (GPU 0; 9.78 GiB total capacity; 6.69 GiB already allocated; 265.00 MiB free; 7.00 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Is it possible to set max_split_size_mb
running singularity?
Better opening a new issue?
Thank you,
f4b
from fastsurfer.
Hi, this is a CUDA error and I don't think you can set these. It is strange that you have memory issues on a 10GB card. The question is what is already taking up the 7GiB. Are you running anything else on the GPU? When is this error happening (during the first segmentation network, or during view aggregation )? Oh, and what resolution is you input image?
from fastsurfer.
This is the error:
I am trying to set that variable, like
singularity exec --nv \
--no-home \
--env PYTORCH_CUDA_ALLOC_CONF='max_split_size_mb=1024'
......
Here is the error
Reading volume /data/pos_T1.nii.gz
The input image is not conformed.
A conformed image must satisfy the following criteria:
- Dimensions 256x256x256: False
- Voxel size 1x1x1: False
- Orientation LIA: False
- Dtype uint8: False
Conforming image to UCHAR, RAS orientation, and 1mm isotropic voxels
Input: min: 0.0 max: 4337.0
rescale: min: 0.0 max: 3148.662 scale: 0.08098678105176103
Output: min: 0.0 max: 255.0
Saving conformed original image to /output/sub-01/mri/orig.mgz
Cuda available: True, # Available GPUS: 1, Cuda user disabled (--no_cuda flag): False, --> Using device: cuda
Loading Axial
Successfully loaded Image from /data/pos_T1.nii.gz
Loading Axial Net from /fastsurfer/checkpoints/Axial_Weights_FastSurferCNN/ckpts/Epoch_30_training_state.pkl
Axial model loaded.
Traceback (most recent call last):
File "eval.py", line 485, in <module>
fastsurfercnn(options.iname, options.oname, options.conformed_name, use_cuda, small_gpu, logger, options)
File "eval.py", line 315, in fastsurfercnn
pred_prob = run_network(img_filename,
File "eval.py", line 213, in run_network
temp = model(images_batch)
File "/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/fastsurfer/FastSurferCNN/models/networks.py", line 79, in forward
decoder_output1 = self.decode1.forward(decoder_output2, skip_encoder_1, indices_1)
File "/fastsurfer/FastSurferCNN/models/sub_module.py", line 305, in forward
out_block = super(CompetitiveDecoderBlock, self).forward(concat_max)
File "/fastsurfer/FastSurferCNN/models/sub_module.py", line 102, in forward
x1 = self.prelu(x1_max)
File "/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/venv/lib/python3.8/site-packages/torch/nn/modules/activation.py", line 1076, in forward
return F.prelu(input, self.weight)
File "/venv/lib/python3.8/site-packages/torch/nn/functional.py", line 1500, in prelu
return torch.prelu(input, weight)
RuntimeError: CUDA out of memory. Tried to allocate 128.00 MiB (GPU 0; 9.78 GiB total capacity; 6.94 GiB already allocated; 25.19 MiB free; 7.25 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Thanks!
f4b
from fastsurfer.
The error happens right at the start.
- Are you using the latest release official docker image (version 2.2.0) converted to singularity?
- And also what is the resolution of your input image?
- Are you running anything else on the GPU? Maybe on the host system (window manager. ...)?
from fastsurfer.
Related Issues (20)
- CC values all 0 in aseg+DKT.stats HOT 5
- Issue creating symbolic links when running fastsurfer-gpu.sif HOT 6
- Error during smooth_aparc.py mode HOT 1
- Error during smooth_aparc.py mode_filter HOT 5
- Biasfield-corrected image input of the segstats.py HOT 4
- Use of '--rm' argument in build.py results in a TypeError HOT 4
- Is it possible to get segstats of cerebellum after run FastSurfer pipeline? HOT 3
- Some zeros in aseg.stats HOT 8
- Fooocus colab stopping with this error HOT 4
- FastSurfer surf pipeline did not finish: Missing .label files (not enough memory for mris_sample_parc) HOT 11
- conform.py bug HOT 2
- srun_fastsurfer.sh on HPC, surface pipeline fails for hundreds but works for tens of subjects HOT 8
- Question about content of wmparc.DKTatlas.mapped.mgz HOT 1
- Support for lesion masks? HOT 1
- Unmatched ROIs in predicted segmentation and provided FastSurfer_ColorLUT.tsv HOT 23
- FastSurfer Segmentation Modules: disable conformation of input image to isometric spaces HOT 5
- FastSurfer QuickSeg doesn't work with OASIS `.img` files HOT 3
- Docker build workflows HOT 5
- Model download issue HOT 22
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fastsurfer.