GPU memory / joint training about dilation

[CQFIO]

I only tried predict.py for a single image. I get "out of memory" error with Titan X.

[manuelschmidt]

I also started to play around with the code a bit and wanted to test it on the cityscapes testimage. Not training, just the predict.py.
And it shows me there out of memory when I try it on my gpu with --gpu 0. Did you guys find a solution for it? Is really 18GB of GPU memory required for it? If so - is there any possibility to reduce the memory footprint at testing time? I use cudnn v5.1 and CUDA 8.0.

By the way: It works on KITTI with GPU and on cityscapes with CPU only.

[CQFIO]

I have a similar issue. I can only run the program in CPU mode because it requires about 18 Gb in memory.

[fyu]

@Timo-hab @chenqifeng22 test_batch has to be 0 for cityscape joint training. Was it set correctly?

[fyu]

@chenqifeng22 You have to check your setup. A lot of people have confirmed to me that predict.py can work well.

[CQFIO]

@fyu I need to install cuDNN to make it work.

[Timo-hab]

@fyu thank you for your answer. test_batch wasnt set to 0. Now i can start the training, but dont see test results of course.
Is it possible to test the model on validation dataset while training, so i can see accuracy and loss, without the need for more GPU memory?

[CQFIO]

@manuelschmidt Depending on the model of GPU, the dilation network needs different amount of memory (which is quite strange to me).
There is a simple solution. Just make the input image size smaller in the prototype file. Also you need to change " if dataset.zoom > 1:" to "if dataset.zoom >= 1:" in the predict.py to prevent a bug.

[qualia0000]

@fyu I am training the ADE20K dataset on the Dilation network with the frontend mode in GPU with 8 GB memory and I get the error message 'out of memory'. Is 8 GB enough for the training?
My training batch is set as 10 and test batch 0.
Thank you

[fyu]

@qualia0000 Besides the batch size, the memory consumption also depends on the crop size.

[qualia0000]

@fyu Thank you for your reply. The crop size is set as 500. I'll try methods to make the consumption smaller.