ablation study for the "mtl" and "feature fusion" strategies about class-incremental-learning

[yaoyao-liu]

Hi Michaelyq,

Thanks for your interest in our work.

I think you're using an old version of our released code with some bugs. The initialization of mnemonics will assign wrong labels to the exemplars. Please check if this bug will affect you.

We have already fixed the bugs that may affect you. You may download our new code using this link: Google Drive. This is also a preliminary version. You may use the code in train_v2.0 to run the ablation experiments for the feature fusion operation and the code in train_v1.0 to run the experiments without the feature fusion operation. we'll merge this version to GitHub when we finish cleaning up and checking.

For the feature fusion operation, we have the ablation results as follows,

Setting	Acc. for the last phase	Mean Acc.
Ours	58.21%	66.51%
2x free	54.18%	62.26%
Baseline	54.30%	63.17%

The improvements are not from the additional model capacity. The ablation results are shown in the table for the setting --nb_cl_fg=50 --nb_cl=10 --nb_protos 20 --imprint_weights. We can observe that using 2x free directly will not improve the results. The implementation for the ablation experiment 2x free is already included train_v2.0, you may use python main.py --nb_cl 10 --imprint_weights --fusion_mode free to run it. Please note that the feature fusion operation is a newly added technique. It achieves further improvements and it is not related to the original paper.

For the weight transfer operation, please please kindly refer to the Supplementary materials, Section B for details: arXiv link. Our mnemonics exemplars work with the weight transfer operation together to achieve the best accuracy.

Please don't hesitate to contact us or add further comments on this issue.

[Michaelyq]

Thanks for your constructive suggestions. The improper initialization of self.mnemonics in the old version may affect the performance of experiments I did. I will try your updated training code later.

According to your experimental results, it seems that the performance gain is achieved by the composition of weight transfer operation (WTO) and feature fusion together (not along with either side). I think it is sensible since self.network_mtl and self.network are two different networks and heterogeneous architectures should be better than the homogeneous architectures(tg_model = self.network() and free_model = self.network()).

However, I think the most important contribution (by going through your paper) should be the self-learnt exemplars (self.mnemonics in your code). So I did the 3rd ablation experiment (excluding WTO and feature fusion) first by replacing the herding exemplars in LUCIR with self.mnemonics, in order to see the contribution of self.mnemonics to the performance gain. I think it is very necessary to verify this point fist to see if dataset distillation-like techniques can be applied to the incremental learning field.

It will be much appreciated if you have the ablation experiment for my 3rd experimental setting. Hopefully I explain it clearly.

By the way, in your train_v2.0 version, is the variable X_train_cumuls accumulated from phase 0 to i using updated self.mnemonics for each phase? Or using original images from phase 0 to i-1 and updated self.mnemonicsin the current phase?

Thanks in advance!

Michael

[yaoyao-liu]

Hi Michael,

The feature fusion operation is only applied in train_v2.0. If you hope to run the experiments without the feature fusion operation, you may use train_v1.0 directly.

In Table 2 of the main paper, we provide ablation results for different exemplar methods based on the network with weight transfer operation. I think these ablation results can partially answer your question. As we have claimed, our mnemonics exemplars work with the weight transfer operation together to achieve the best accuracy. The ablation experiment you mentioned is reasonable. Thanks for the suggestion. We'll do that in our future work.

In train_v2.0, X_train_cumuls accumulated from phase 0 to i using updated self.mnemonics for each phase. After updating, the new exemplars will be assigned to the original dataset and we'll get these updated samples for the next phase.

Best,
Yaoyao

[Michaelyq]

Thanks for your kind reply.