Can not reproduce the performance in your paper. about hamnet

[asrafulashiq]

About the number of classes, Please check the definition of self.classifier in system/system_th.py line 260, where output dimension is n_class + 1, i.e., 21.

The loss functions are named differently in the code. Specifically, loss_1_orig=L_BCL, loss_1_drop=L_HAL, loss_2_orig_supp=L_SAL, and loss_2_drop_supp=L_SSAL. Hope it clarifies.

[sanqingqu]

Thanks for your quick and detailed reply. In fact, I have tried to run your codes on THUMOS-14 Datasets, however, the overall performance is much lower than your report. Maybe I should spend some more time to figure out the problem.

[sanqingqu]

Sorry to bother you again!
I try to figure out why my reproduced performance on THUMOS-14 dataset is much lower than your report. I found that the definition of top-k value 'rat' in 'system/system_th.py' in line 40 is set to 10, while in your paper, you said that you set it to 50. I reset this param's value. However, the performance is still much lower than your report. In addition, I found that you did not introduce the validation step to save the best checkpoint but saved the latest checkpoint file after every training epoch done. The final performance is reported based on the last checkpoint file, which may not produce the best performance.
So, I try to implement thevalidation_step and validation_epoch_end function to your 'system/system_th.py' and validate the model's performance at every 10 epoch. However, during my running progress, the validation performance is still much lower than your report. Since previous works typically train the network more than 100 epochs, I try to train your HAM-Net about 500 epochs. However, my reproduced AVG performance is lower than 20.0 about 19.4, which is still much lower than your report of 39.8. Could you help me figure out the reasons? 😥
My implement platform is 'pytorch==1.7.1+cu110', with NVIDIA RTX 2080Ti.

Furthermore, I found a mistake in your paper, in Table 2 on page 7, your reported AVG is 39.8, however, the AVG should be 41.1 according to your reported performance under tIOU=[0.1, ..., 0.7].