neuralcollapseapplications / fscil Goto Github PK

I am not sure what produce_training_rect function is supposed to do and why is it used for few shot sessions only?

Dear authors,
Firstly, I would like to express my appreciation for your interesting and motivating work. Thank you for your contributions to the field.
I am writing to inquire about the codes you have provided. I have attempted to reproduce the results on CIFAR-100 using the codes, but unfortunately, I have encountered some issues. (I am unsure if these issues also occur in other datasets.)
Specifically, there seems some problem during the incremental sessions.
I observed a loss explosion after session 4.
I should mention that the experiments were conducted on the docker environment.
I am wondering if there are any problems with the current version of the codes that may have caused these issues.
I attached the log files.

20230410_150211.log
20230410_160540.log

Thank you in advance for your time and assistance. I look forward to hearing back from you soon.

I tried to reproduce your work using the provided docker. I tries to train and evaluate on cifar. Because I only have 1 gpu, I edited the config to have 512 sample_per_gpu instead of only 64. then I run this command

bash tools/dist_train.sh configs/cifar/resnet12_etf_bs512_200e_cifar.py 1 --work-dir /opt/logger/cifar_etf --seed 0 --deterministic && bash tools/run_fscil.sh configs/cifar/resnet12_etf_bs512_200e_cifar_eval.py /opt/logger/cifar_etf /opt/logger/cifar_etf/best.pth 1 --seed 0 --deterministic

the result is as follows

2023-04-14 21:32:58,114 - mmcls - INFO - loss1 5.088033676147461 ; loss2 5.087942123413086
2023-04-14 21:32:58,119 - mmcls - INFO - [198/200] Training session : 9 ; lr : 0.00025 ; loss : 5.035999774932861 ; acc@1 : 0.0
2023-04-14 21:32:58,119 - mmcls - INFO - loss1 5.036115646362305 ; loss2 5.035883903503418
2023-04-14 21:32:58,124 - mmcls - INFO - [199/200] Training session : 9 ; lr : 0.00025 ; loss : 5.110080718994141 ; acc@1 : 0.0
2023-04-14 21:32:58,127 - mmcls - INFO - [200/200] Training session : 9 ; lr : 0.00025 ; loss : 5.047082901000977 ; acc@1 : 12.5
2023-04-14 21:32:58,127 - mmcls - INFO - Evaluating session 9, from 0 to 100.
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 10000/10000, 17685.8 task/s, elapsed: 1s, ETA:     0s
2023-04-14 21:32:58,816 - mmcls - INFO - [09]Evaluation results : acc : 1.00 ; acc_base : 0.00 ; acc_inc : 2.50
2023-04-14 21:32:58,816 - mmcls - INFO - [09]Evaluation results : acc_incremental_old : 2.86 ; acc_incremental_new : 0.00
2023-04-14 21:32:58,888 - mmcls - INFO - 82.73 57.63 1.40 1.33 1.25 1.18 1.11 1.05 1.00 

the evaluation after session 0 is very low.

20231213_170107.log

Hi, we use the docker command provided to run the code. On the cifar100 dataset, starting from the fifth session, the accuracy dropped to very low. Below is my log file:
20231219_095703.log
Any help would be greatly appreciated

Thank you for your impressive work. When I was trying to reproduce the result with two or three GPUs, even if the cofigs had been changed, CIFAR100 still produced a strange result that was unreasonable while the other two is normal. So I'm wondering if there are subtle bugs in CIFAR100 training especially in the evaluation process. Otherwise, the acc won't decrease dramatically after session2.
Here are the logs.
cifar.log
cub.log
imagenet.log