Comments (3)
Hello :)
So this tiny ConvNet has been precedent for dataset distillation works for some time now precisely because of how tiny it is.
Our method (and most others) involve a bi-level optimization that is very costly both in terms of time and memory.
We simply do not have the computing capabilities (nor the time) to use a large model like WideResNet50 as a backbone.
Hopefully someone will come up with a method that isn't so costly so we can then use realistic models to distill our datasets :)
Naively, I would guess that distilling on a larger model would distill better data for that larger model, which would potentially be more useful overall than data distilled for a small model.
It's hard to say whether or this data would transfer better to other models as well; cross-architecture generalization is still a big shortcoming of most distillation methods.
TLDR: Hard to say since we literally can't test it, but looking forward to when someone comes up with a more efficient method that allow us to try :)
from mtt-distillation.
Thanks for the detailed response! I too hope for more efficient methods in the future as I'm also working with 3090 gpus...
Just another question about Figure 8 Ablation study regarding the use of ZCA. You've mentioned in your paper that expert models trained without ZCA normalization take significantly longer to converge. Thus, need a larger value of T+.
So when someone tries to use your method on a expert model trained for 200 epochs it would still work? And if so the max start epoch would also have to scale to about 160? Then how about the sythetic steps is 20 still enough?
Thank you for your generous replies.
from mtt-distillation.
The point of the T+ parameter is to ensure that we only look at large enough meaningful updates.
It's almost just a heuristic. We chose an initial value by observing when the performance of the expert models stop increasing in by large amounts.
As long as the updates (changes in the weights) are large enough and meaningful (i.e., not just jittering around the final resting point), then the method should still work.
As far as the number of synthetic steps, this is largely decoupled from T+. This just describes how many updates we should make on the synthetic data to equate to M epochs of training on the real data. Because of our learnable learning-rate, our method is somewhat robust to different values of N and M (Figure 5). Without this adaptive learning rate, matching N synthetic steps to M real epochs would be a near impossible task.
Hope this helps!
from mtt-distillation.
Related Issues (20)
- GPU requirement HOT 2
- Question about imagenette HOT 1
- How did you get x̄ ± s in table 1 HOT 2
- question about learning rate HOT 2
- question on Tiny imagenet ipc=50 HOT 2
- What is ZCA HOT 2
- Grand loss curve HOT 1
- N << M or N >> M HOT 7
- Model training with the released synthetic CIFAR-10 (ipc=50) HOT 4
- ReparamModule Usage HOT 2
- CIFAR 10 ipc 10 Hyperparameters HOT 2
- About Hyper-paramters HOT 1
- how to do distillation for a model other than VGG? HOT 2
- Could you please share the variance of the performance in the KIP to NN column?
- details about the full real subsets of ImageNet HOT 2
- Running this project with PCAM instead of CFAR
- Normalization of dataset HOT 4
- Checkpoints of models HOT 1
- Quick Question regarding class index mapping for the custom imagenet subsets HOT 4
- Why grand_loss will gradually increase and become NaN
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mtt-distillation.