A question about backbone networks about mtt-distillation

[GeorgeCazenavette]

Hello :)

So this tiny ConvNet has been precedent for dataset distillation works for some time now precisely because of how tiny it is.

Our method (and most others) involve a bi-level optimization that is very costly both in terms of time and memory.

We simply do not have the computing capabilities (nor the time) to use a large model like WideResNet50 as a backbone.

Hopefully someone will come up with a method that isn't so costly so we can then use realistic models to distill our datasets :)

Naively, I would guess that distilling on a larger model would distill better data for that larger model, which would potentially be more useful overall than data distilled for a small model.

It's hard to say whether or this data would transfer better to other models as well; cross-architecture generalization is still a big shortcoming of most distillation methods.

TLDR: Hard to say since we literally can't test it, but looking forward to when someone comes up with a more efficient method that allow us to try :)

[imesu2378]

Thanks for the detailed response! I too hope for more efficient methods in the future as I'm also working with 3090 gpus...

Just another question about Figure 8 Ablation study regarding the use of ZCA. You've mentioned in your paper that expert models trained without ZCA normalization take significantly longer to converge. Thus, need a larger value of T+.
So when someone tries to use your method on a expert model trained for 200 epochs it would still work? And if so the max start epoch would also have to scale to about 160? Then how about the sythetic steps is 20 still enough?

Thank you for your generous replies.

[GeorgeCazenavette]

The point of the T+ parameter is to ensure that we only look at large enough meaningful updates.

It's almost just a heuristic. We chose an initial value by observing when the performance of the expert models stop increasing in by large amounts.

As long as the updates (changes in the weights) are large enough and meaningful (i.e., not just jittering around the final resting point), then the method should still work.

As far as the number of synthetic steps, this is largely decoupled from T+. This just describes how many updates we should make on the synthetic data to equate to M epochs of training on the real data. Because of our learnable learning-rate, our method is somewhat robust to different values of N and M (Figure 5). Without this adaptive learning rate, matching N synthetic steps to M real epochs would be a near impossible task.

Hope this helps!