Training seems to stop after the first epoch about dcgan.torch

[soumith]

The fake images update too. It's just that the network has converged, and the changes are very minor.

[aferriss]

@soumith Well I would have thought so too, but why is the network converging so quickly? I ran the code on the same data set, but at epoch 100, the fake output looks exactly the same as it did at epoch 2 (mostly noise). This was not the case training on my other machine.

[aferriss]

Actually after running a few more tests I'm fairly sure this is some issue related to the gpu.

Here are two logs using the same random seed and data, the first is cpu and the second is gpu. The gpu converges extremely quickly, towards the end of epoch 1. The cpu never really converges and the error seems to occasionally spike.

I also installed the cudnn with the r4 bindings because I was getting the error mentioned in this post. Not sure if this could be related...
karpathy/neuraltalk2#44

GPU Log:

~/dc$ gpu=1 display_id=41 DATA_ROOT=/media/aferriss/SHARED/allImgs/Users/adamferriss/Desktop/dcgan.torch/myimages dataset=folder nThreads=12 th main.lua

Full log: https://gist.github.com/soumith/58cdb2ef9f8e0dbbf71a

[aferriss]

Ok, after a few frustrating hours I figured it out. I believe there is a problem with cudnn 4. I downgraded to cudnn 3 and now everything seems to work fine.

[soumith]

Hmm that's weird, I'll look into it too.

[shubhtuls]

I also encountered this issue on Ubuntu with GPU training and disabling the checkpoint save statements at the end of the epoch seemed to fix the issue.

[soumith]

Thanks to Kenneth Marino from CMU, I figured out the bug.
Pushing a patch shortly, sorry about this.
The issue is that the flattened parameter buffers gotten by the getParameters call get untied after util.save, because in util.save I typecast the network to CPU and then back to GPU.
I added util.save / util.load while making my code ready for release, so I did not notice this issue in my experiments.

[soumith]

The bug is fixed now via: 5b093d3

Sorry about this.

[aferriss]

@soumith Thanks so much! Glad to know I'm not crazy 👍