tutorial: adversarial training seems slow. maybe i'm wrong about cleverhans

[goodfeli]

without looking at the code, I bet we're missing a stop_gradient somewhere

[goodfeli]

3.7 seconds per 100 batches for naive training

[goodfeli]

52 sec per 100 batches for adv training

[goodfeli]

in pylearn2, my result with adversarial training takes 3 sec per full epoch

[goodfeli]

in pylearn2, without adversarial training, my code runs in 1 sec per full epoch

[goodfeli]

naive training is forward-back, O(2).
adversarial training is forward-back, back(with different targets)-forward-back, so O(5) if no steps can be parallelized. So it should be roughly 2.5X slower than naive training in theory.
The pylearn2 implementation is 3X slower than naive training, so apparently in practice we can expect some extra overhead.
Doesn't explain why this is > 10X slower.

[goodfeli]

whoa, actually something is seriously weird.
1st 100 batches with adv training take 54 seconds
2nd 100 batches take 102 seconds
3rd 100 batches take 153 seconds

[npapernot]

You are right: the issue was due to my naive implementation which redefined the adversarial loss in the TF graph at each iteration (batch...). I fixed by introducing a new function that add the loss to the graph, and which returns the TF var to be evaluated at each iteration d7a95d3