how to deal with overfitting? about dcgan_code

[Newmu]

etl_test.png is some debug code used it to check scaling/transpose bugs with image data formats - it shows you actual training examples - those are not samples.

The last number is cost of the discriminator, the second to last is cost of generator. A typical failure mode is the generator eventually being unable to fool the discriminator resulting in its cost climbing to 10-15-20 and the discriminator cost going to 0.

The first three are nearest neighbor euclidean distances of real examples to 1,000, 10,000, 100,000 generated samples. One way of measuring overfitting is whether that number is lower on training data than on validation data. It's a relative number as it's dataset dependent. Ideally it will continue to go down over the course of training.

[udibr]

Thanks, I've spotted my mistake with etl_test and the real generated images looks more resonable.
So it looks like both networks are matching each other, and the cost numbers it looks like no progress is being made. However, if I look at the sequence of the generated images after each iteration there is a very clear improvement from iteration to iteration. So what is going on? Anyway its amazing!

[udibr]

It now looks that what you described is happening the generator cost starts to rise and the discriminator falls dramatically. Any tips on how to rebalance this? (I am collecting more images, in the hope that this will help.)

[soumith]

@udibr some tips on re-balancing are given here: http://torch.ch/blog/2015/11/13/gan.html#balancing-the-gan-game

[Newmu]

We're still figuring out/understanding why this happens. Soumith's link has good advice. More data helps but does not prevent it - we've seen models go unstable before even finishing an epoch.

[udibr]

Thanks, the link is great. My problem is oscilliaitions between the two scores and not an explosition of one on the expense of the other. The image quality just get better after each iteration but this could be overfitting in action. I will re-run with more images and see if it helps.

[dribnet]

Hi @udibr - one thing to be on the lookout for in the faces code is the expected fuel dataset. The faces code expects data stored channel last like image[w][h][c] instead of the fuel standard channel first pattern of image[c][w][h].

The etl_test.png is a great test of whether this is is being unpacked correctly - as @Newmu says it's a logical passthrough of training samples and should not be garbled at all. If it is, there is a one line numpy.dstack you can insert to transform your data appropriately.

[nguyenductung]

Hi, I'm trying your network with my dataset (~50K samples) and I also think it is overfitting. Dropout may help prevent overfitting but it seems that you do not use dropout in your training script for the faces dataset. There is a dropout function in lib/ops.py, could you tell me how to use it?

[udibr]

add it after every relu line. For example the discriminator will look like:

        h = lrelu(dnn_conv(X, w, subsample=(2, 2), border_mode=(2, 2)))
        h = dropout(h, 0.2)
        h2 = lrelu(batchnorm(dnn_conv(h, w2, subsample=(2, 2), border_mode=(2, 2)), g=g2, b=b2))
        h2 = dropout(h2, 0.2)
        h3 = lrelu(batchnorm(dnn_conv(h2, w3, subsample=(2, 2), border_mode=(2, 2)), g=g3, b=b3))
        h3 = dropout(h3, 0.2)
        h4 = lrelu(batchnorm(dnn_conv(h3, w4, subsample=(2, 2), border_mode=(2, 2)), g=g4, b=b4))
        h4 = dropout(h4, 0.2)
        h4 = T.flatten(h4, 2)
        y = sigmoid(T.dot(h4, wy))

[nguyenductung]

@udibr Thanks. I'm trying it now.