CUDA out of memory when using ELMo on a 12GB GPU about piqa

[seominjoon]

Will need to use --batch_size=32 for 12GB. For 24GB GPU, works well.
f1b4520

[jhyuklee]

FYI, using ELMo with 12GB TitanXP (default batch_size = 64) works well :)
It takes up 9963MiB, and reaches 0.632 dev_f1 @ 5001.

[seominjoon]

Weird.. which commit version did you use btw?

[jhyuklee]

I'm working on the forked repo (https://github.com/jhyuklee/piqa),
and the latest commit is 7dfb6fe.

[bdhingra]

Worth pointing out that when I tried running the code on a 12GB GPU I also ran into a memory error with the default batch size of 64. Reducing the batch size to 32 solved the problem, and also gave a better F1 score of 63.5% (before applying the ELMO fix from 72c7623).

[seominjoon]

@bdhingra thanks, 32 surely seems to work better (I got 63.7%).

[jhyuklee]

After updating to c6e8719, batch_size=64 makes OOM for me, too.