comparisons about openlongtailrecognition-oltr

[matttrd]

@liuziwei7 thanks for your clarifications! We are actually working on a similar project and we obtained the same observations. I really like (and agree with) the approach of tackling open long-tailed recognition problems with a single framework. It is still an ongoing project but we were able to obtain good results with a theory-driven input sampling (without attention, hallucinator, etc). We will try our algorithm on your framework. Thanks again!

[zhmiao]

@matttrd Thanks for asking. The plain was only trained for 30 epochs and the rest of the methods were trained for 90 epochs. As discussed here: #4 (comment) , more epochs would not help the performance of plain model since it was already converged around 30 epochs.

[matttrd]

@zhmiao thanks for answering. Probably I'm missing something but I trained (using your code) the plain model (stage 1) with 90 epochs with the usual drop of the learning rate every 30 epochs getting 32.7% of overall top1 test accuracy and:

• few shot: 4%
• Median: 23.9%
• Many: 53.6%

I understand that the loss converges around 30 epochs but this is true only if you let the LR drop every 10 epochs as you did. I think this should be the fair comparison. Am I wrong?

[liuziwei7]

Thanks for reporting these results! Actually we have obtained some similar observations in our follow-up project.

A model trained on long-tailed dataset has some interesting behavior change with different initializations and epoches:

1. Large learning rate tends to make the model bias to many-shot classes;
2. Intermediate and late epoches are critical for the performance of few-shot classes.

We speculate the root of these phenomena come from the learning dynamics of long-tail-trained models. Therefore, we are considering update our manuscript to report a sequence of snapshot accuracies instead of a single final accuracy. It is yet an open question, and we believe it is definitely an interesting direction to further investigate.

Our aim of the open long-tailed recognition paper is to formally define and make clear this important real-world problem, instead of providing a silver-bullet solution. We welcome everyone to work on this topic and further improve the underlying approaches :)