WLM test is failing about pytorch_memonger

[dmitrivainbrand]

Also it seems that weight update is missing. There is no training, just fprop and bprop in a loop. Probably not important for memory saving example but I'd suggest to add it for completeness.

[gwern]

I get the same error with Pytorch 1.0.0 (along with an OOM error on the VNet model, which seems odd given that saving memory is exactly what gradient checkpointing is supposed to do):

...
Optimized resnet (18): (1541782.471 usecs gpu) (1541732.073 usecs cpu)
Optimized resnet (19): (1544267.822 usecs gpu) (1544241.190 usecs cpu)
.test_memory_optimized.py:119: UserWarning: nn.init.kaiming_normal is now deprecated in favor of nn.init.kaiming_normal_.
  nn.init.kaiming_normal(m.weight)
/home/gwern/src/pytorch_memonger/models/optimized/vnet_new.py:133: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.
  out = self.softmax(out)
E/home/gwern/.local/lib/python3.6/site-packages/torch/nn/modules/rnn.py:46: UserWarning: dropout option adds dropout after all but last recurrent layer, so non-zero dropout expects num_layers greater than 1, but got dropout=0.2 and num_layers=1
  "num_layers={}".format(dropout, num_layers))
E
======================================================================
ERROR: test_vnet_optim (__main__.TestMemoryOptimized)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_memory_optimized.py", line 162, in test_vnet_optim
    loss.backward()
  File "/home/gwern/.local/lib/python3.6/site-packages/torch/tensor.py", line 102, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/gwern/.local/lib/python3.6/site-packages/torch/autograd/__init__.py", line 90, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: CUDA out of memory. Tried to allocate 1024.00 MiB (GPU 0; 10.91 GiB total capacity; 8.33 GiB already allocated; 1013.19 MiB free; 848.53 MiB cached)

======================================================================
ERROR: test_wlm_optim (__main__.TestMemoryOptimized)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_memory_optimized.py", line 214, in test_wlm_optim
    hidden = self.repackage_hidden(hidden)
  File "test_memory_optimized.py", line 178, in repackage_hidden
    return tuple(self.repackage_hidden(v) for v in h)
  File "test_memory_optimized.py", line 178, in <genexpr>
    return tuple(self.repackage_hidden(v) for v in h)
  File "test_memory_optimized.py", line 178, in repackage_hidden
    return tuple(self.repackage_hidden(v) for v in h)
  File "test_memory_optimized.py", line 178, in <genexpr>
    return tuple(self.repackage_hidden(v) for v in h)
  File "test_memory_optimized.py", line 178, in repackage_hidden
    return tuple(self.repackage_hidden(v) for v in h)
  File "test_memory_optimized.py", line 178, in <genexpr>
    return tuple(self.repackage_hidden(v) for v in h)
  File "test_memory_optimized.py", line 178, in repackage_hidden
    return tuple(self.repackage_hidden(v) for v in h)
  File "test_memory_optimized.py", line 178, in <genexpr>
    return tuple(self.repackage_hidden(v) for v in h)
  File "test_memory_optimized.py", line 178, in repackage_hidden
    return tuple(self.repackage_hidden(v) for v in h)
  File "/home/gwern/.local/lib/python3.6/site-packages/torch/tensor.py", line 422, in __iter__
    raise TypeError('iteration over a 0-d tensor')
TypeError: iteration over a 0-d tensor

----------------------------------------------------------------------
Ran 4 tests in 68.831s

FAILED (errors=2)

Also, there is an error in the README, where it says

# for checkpointed
python test_memory_optimized.py

# for baseline
python test_memory_optimized.py

presumably the second should actually be python test_memory_baseline.py? Which however errors out:

$ python test_memory_baseline.py
Traceback (most recent call last):
  File "test_memory_baseline.py", line 14, in <module>
    import models.baseline.vnet as vnet_baseline
  File "/home/gwern/src/pytorch_memonger/models/baseline/vnet.py", line 6, in <module>
    import torch.utils.checkpoint_new as checkpoint_new
ModuleNotFoundError: No module named 'torch.utils.checkpoint_new'