The result of val dataset! about se-ssd

[MuMuJun97]

@Vegeta2020 @shxzhao I also got a lower AP results when using the model from this repository:

Evaluation official_AP_11: car AP(Average Precision)@0.70, 0.70, 0.70:
bbox AP:90.61, 89.47, 89.02
bev  AP:89.94, 88.11, 87.44
3d   AP:88.44, 79.07, 78.27
aos  AP:90.51, 89.10, 88.46
car AP(Average Precision)@0.70, 0.50, 0.50:
bbox AP:90.61, 89.47, 89.02
bev  AP:90.67, 89.73, 89.38
3d   AP:90.66, 89.67, 89.28
aos  AP:90.51, 89.10, 88.46

Evaluation official_AP_40: car AP(Average Precision)@0.70, 0.70, 0.70:
bbox AP:96.27, 93.00, 90.56
bev  AP:93.25, 89.64, 87.23
3d   AP:89.73, 83.08, 80.57
aos  AP:96.12, 92.55, 89.97
car AP(Average Precision)@0.70, 0.50, 0.50:
bbox AP:96.27, 93.00, 90.56
bev  AP:96.45, 95.39, 93.02
3d   AP:96.37, 95.22, 92.89
aos  AP:96.12, 92.55, 89.97

[shxzhao]

Did you use your own training model or the pre training model the repository provided?

[WWW2323]

Same as you, and I found that as the training progresses, the accuracy will get lower and lower from epoch 1 to epoch 60.

[shxzhao]

I forget to set the eval result per epoch, but my result of eval dataset is so low, can you provide your set of train.py or provide the train model to me. My email is [email protected]

[WWW2323]

I didn't change the code of train.py, and my final result is same as yours. The only thing I modified is #14. Maybe the bad result is caused by this modification, or the author of the paper missed something. Could you give us some suggestions? @Vegeta2020

[MuMuJun97]

@shxzhao I used the pre-trained student model the repository provided.

[pigtigger]

@WWW2323 I find the results from epoch1 to epoch60 get lower and lower too, sometimes it also will crash during epoch 50 - 60 because of "out of cuda memory ", I am wondering if the model tend to predict more proposals and get lower mAP.
Can you give me some advice?
@Vegeta2020

[FangGet]

This is my evaluation result using the provided model, it seems the same as in README.
If it's lower than that, I guess it's caused by data or dependencies preparation.

[WWW2323]

Maybe you need set batch size = 1 @pigtigger , i set the batch size = 1.
@FangGet Good job! did you change any code of the repository？such as #14, and what's your GPU type? Thanks!

[FangGet]

nop, I keep everything the same as the original, the model copy issue in #14 may be caused by python version, try using python 3.6.
Btw, my GPU type is RTX 2080TI

[pigtigger]

I got the same results as yours when evaluate the provided model, but when I use it the as initial weights for student and teacher model and start training, the performance drops a lot @FangGet

[gqding]

I also obtained the same results as stated in the README!

[pigtigger]

@WWW2323 Thanks .
do you get a good results after training ? could you tell me the model you use for initialize student and teacher? I also modified #14 , but not sure if it is the reason for bad performance

[WWW2323]

I have no idea, i can't even get good results before training @pigtigger

[FangGet]

@pigtigger I guess you should use the pre-trained model from CIA-SSD as the initial model, if SE-SSD pre-trained model is used, try to decrease the learning rate. But it's normal for performance drop as soft-target supervisor may disappear if the difference between teacher and student is ignorable, just my opinion.

[WWW2323]

I also get the same val set results as shown in README.md, but i run into another problem. When i use the model to inference test set and submit to kitti server, the results is as follows:

I don't know if I need to modify any configuration during testing, because the result of SE-SSD on test set is this:

If you get results similar to SE-SSD on test set, please @ me, thanks!! Or @Vegeta2020 can give some advice?

[shxzhao]

Do you use the pre-train model or the model you trained?@WWW2323

[WWW2323]

I use the pre-trained model provided in README to inference val set and test set. The result on the val set is more like the result in the SE-SSD paper, but the result on the test set is more like the result of the CIA-SSD. Now I don't know if the provided-model is the final model of SE-SSD or it is just used for SE-SSD initialization.

[shxzhao]

It should be the final model of SE-SSD.

[pigtigger]

I get the same results as yours @WWW2323
the final model's performance is similar to CIA-SSD, so I think it is not the final model and use it to initialize SE-SSD, but the training results is even lower.

[WWW2323]

Do you mean your result on the test set is the same as mine? @pigtigger

[pigtigger]

yes

…

------------------ Original ------------------ From: WWW2323 ***@***.***> Date: Wed,Aug 18,2021 0:24 PM To: Vegeta2020/SE-SSD ***@***.***> Cc: lihe ding ***@***.***>, Mention ***@***.***> Subject: Re: [Vegeta2020/SE-SSD] The result of val dataset! (#32) Do you mean your result on the test set is the same as mine? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[Vegeta2020]

Hi, guys. The provided model is the SE-SSD model trained on the train split and not on the trainval split, and the results given in README are based on the evaluation on the val split. (I'm afraid I cannot provide the trained model on the trainval and help you produce prediction files for submission to KITTI evaluation server, as it will violate the submission policy and lead to account banning.)
To get decent results, you must use a pre-trained model to initialize both student & teacher model. In our previous experiments, we often get decent results with a pre-trained model training from scratch, but there are only a few cases that we cannot, we guess it' related to the initialization.

[WWW2323]

Hi, everybody, do you reproduce the results? I trained a CIA-SSD model and use it for the initialization of SE-SSD. Although there is no drop as using provided model in README for initialization, there is no increase. So sad

[pigtigger]

@WWW2323 do you mean that you use your own model trained from CIA-SSD for initialization to train SE-SSD and get almost the same results as your initialization model ? I use CIA-SSD pretrained model in CIA-SSD README and get a huge drop. could you send your trained CIA-SSD to me ? I want to have a try .Thanks, my email is [email protected]

[AndyYuan96]

Hi, everybody, do you reproduce the results? I trained a CIA-SSD model and use it for the initialization of SE-SSD. Although there is no drop as using provided model in README for initialization, there is no increase. So sad

@WWW2323 , did you use the pretrained model provided by author in CIA-SSD rep, and train using SE-SSD.

[WWW2323]

No, you need train a CIA-SSD model yourself with CIA-SSD rep. The head of CIA-SSD in CIA-SSD rep is different from the head of CIA-SSD in SE-SSD.

[Eaphan]

I also obtained the same results as stated in the README!

Can you share the log of training with me? Thank you very much.

[yujmo]

Evaluation official: car AP(Average Precision)@0.70, 0.70, 0.70:
bbox AP:88.58, 77.79, 75.09
bev AP:84.71, 75.34, 67.57
3d AP:47.61, 41.61, 40.11
aos AP:87.11, 75.19, 72.10
car AP(Average Precision)@0.70, 0.50, 0.50:
bbox AP:88.58, 77.79, 75.09
bev AP:90.81, 89.37, 87.25
3d AP:90.62, 88.13, 79.81
aos AP:87.11, 75.19, 72.10

Evaluation coco: car coco [email protected]:0.05:0.95:
bbox AP:60.11, 56.47, 54.31
bev AP:53.67, 50.72, 47.66
3d AP:41.55, 37.92, 35.54
aos AP:59.18, 54.57, 52.16