OSError: [Errno 12] Cannot allocate memory about resnext.pytorch

[ZhuFengdaaa]

@penguinshin I fix this bug by adding 64G swap memory. When data loader forks workers to load data, the memory increases rapidly. You can try setting num_workers=1 first, then try allocating larger swap space.

[rajatagrawal193]

**Fixed this ** by allocating 4G swap memory. You can try allocating more memory if 4G does not suffice.
Follow this blog to allocate swap memory on your device.
https://www.digitalocean.com/community/tutorials/how-to-add-swap-space-on-ubuntu-16-04

[monacv]

Fixed the problem by allocating 64GB of swap memory from the external disk.

[prlz77]

OSError: [Errno 12] Cannot allocate memory sounds more like a RAM problem, not a GPU problem. Check you have enough RAM/SWAP, and the correct user permissions.

[usamec]

Fix is almost always:
echo 1 > /proc/sys/vm/overcommit_memory

https://stackoverflow.com/a/52311756/1391392

[nbansal90]

Yeah Exactly!
Meanwhile, I collected the output of lspci command:(for NVIDIA 1080 TI)

bansa01@vita:~/pytorch_resnext/tmp$ lspci -v -s 89:00.0
89:00.0 VGA compatible controller: NVIDIA Corporation Device 1b06 (rev a1) (prog-if 00 [VGA controller])
        Subsystem: ZOTAC International (MCO) Ltd. Device 1470
        Flags: bus master, fast devsel, latency 0, IRQ 105
        Memory at f4000000 (32-bit, non-prefetchable) [size=16M]
        Memory at 2ff80000000 (64-bit, prefetchable) [size=256M]
        Memory at 2ff90000000 (64-bit, prefetchable) [size=32M]
        I/O ports at b000 [size=128]
        [virtual] Expansion ROM at f5000000 [disabled] [size=512K]
        Capabilities: <access denied>
        Kernel driver in use: nvidia
        Kernel modules: nvidiafb, nouveau, nvidia_384_drm, nvidia_384

Does this give us any information as to where we might be going wrong. Can I change anything myself,(given that I have root permission)which could help me prevent this issue.

[ZhuFengdaaa]

Have you fixed that ? I am facing the same issue.

[penguinshin]

I am also running into the same problem, although I am running everything on a CPU. I have more than enough memory (the error occurs when I'm using only 10G out of 32G)

[prlz77]

Hi! As @ZhuFengdaaa confirms, it seems a peak memory problem although I am not able to reproduce it. Again as @ZhuFengdaaa suggests, this seems to be linked with the number of threads (also see https://discuss.pytorch.org/t/guidelines-for-assigning-num-workers-to-dataloader/813/6).

[prlz77]

Another related thread ruotianluo/self-critical.pytorch#11

[henryccl]

OSError: [Errno 12] Cannot allocate memory sounds more like a RAM problem, not a GPU problem. Check you have enough RAM/SWAP, and the correct user permissions.

Why does this have to do with permissions？and what should i do with permissions?

[BlueskyFR]

Why would some swap be needed? It slows down everything
I am having the same problem while trying to allocate 30 GB even though I have 1 TB free...

[jwr0218]

This problem is come from CPU memory allocation. check CPU Ram Memory