OOM Issue on UNetXST -- Occupies 55GB of physical RAM about cam2bev

[bastilam]

Hello @ogkdmr,

I just did a clean install and download of the data. Cam2BEV did not work with my GPU out of the box because tensorflow 2.1 from the requirements.txt in this project does not work with the newer cuda version which I had already installed on my PC. The training only used my CPU and a lot of RAM.

I just installed tensorflow 2.4 using pip install tensorflow==2.4 inside my conda environment. Now, the training uses my GPU and it also uses a lot less RAM.

Maybe this is something you could try as well. Make sure the training actually uses the GPU.

Watch out for errors of this kind: Could not load dynamic library 'libcudnn.so.7'

In another terminal tab, you can watch whether your GPU memory is actually used with watch nvidia-smi

[ogkdmr]

Thank you @bastilam for taking the time of reproduce the issue. I can confirm that the model is training on the GPU. Perhaps we need to get more RAM for our machine, although we already have around 128GB.

We can close the issue for now. I will keep you updated if I resolve the issue.