About the model performance about fastflow3d

[shanjiayao]

Okay, thanks for your reply. I will continue to try to improve the performance of the model, and I will contact you as soon as there is progress. Thank you very much!

[Jabb0]

Hello @shanjiayao, thank you for the interest!

I have added our experimental setup and results to the README including comparison with the metric reported in the original paper.
Due to hardware limitations we were not able to train on a batch size of 64 as the original authors did. In my opinion this had an impact on the performance of our model, thus it is not performing as well as the authors have reported.

[shanjiayao]

Hello @shanjiayao, thank you for the interest!

I have added our experimental setup and results to the README including comparison with the metric reported in the original paper. Due to hardware limitations we were not able to train on a batch size of 64 as the original authors did. In my opinion this had an impact on the performance of our model, thus it is not performing as well as the authors have reported.

Emmmm, I had also reproduced this work, and I got similar results with you. Maybe you are right, the model is limited by the hardware. Hence I am going to try the gradient accumulation.

[Jabb0]

That is interesting to hear.

Gradient accumulation is implemented in this repository already as an option.
But keep in mind that it does not change the batch size used in the BatchNorm layers (at least to my knowledge).

And as BatchNorm is sensitive to the number of samples in each batch I have implemented a switch to use GroupNorm instead as well. Maybe you can get better results with that one and some hyperparameter search for the number of groups.

[shanjiayao]

That is interesting to hear.

Gradient accumulation is implemented in this repository already as an option. But keep in mind that it does not change the batch size used in the BatchNorm layers (at least to my knowledge).

And as BatchNorm is sensitive to the number of samples in each batch I have implemented a switch to use GroupNorm instead as well. Maybe you can get better results with that one and some hyperparameter search for the number of groups.

Oh, Thanks. I understand. So is groupnorm used by default in your experimental results? Maybe I can try it later! Besides that, I found it is very difficult to deal with the imbalance among these categories, is this also one of the reasons for the low accuracy?

[Jabb0]

The results in the readme are done with BatchNorm as in the original paper.
The default is BatchNorm.
But GroupNorm can be used instead via a command line option.

Yes the imbalance is likely a cause. That is why there is a downweight of the background class in the loss function.
In general points that are slower have less error (compare moving Vs. All in the table), indicating some bias towards background points (which have 0 flow).
If you have an idea for dealing with the imbalances try that.

Also the cyclist class has very few points. First bicycles are small and second (I guess) not common in the US.

However, a large portion of points belonging to vehicles are up to 1 m/s correct.