Bias for Prediction of Unseen Classes about symnet

[silicx]

There are two different settings in out paper. The results in Tab. 1 follows AttrOperator [26], which only evaluates on the unseen classes so the bias does not affect (we use zero bias). The results in Tab. 2 follows TMN [33], which uses AUSUC to take the bias into account. AUSUC (Area Under Seen-Unseen accuracy Curve) is calculated by changing the bias to produce a series of seen/unseen accuracies and computing the area under the seen-unseen accuracy curve.
More details please refer to [26] and [33].

[Zoya-Hashmi]

Thank you for explaining. But I am still confused as to what bias is used for reporting the single value of seen and unseen accuracy, in table 2. Is it average of all the seen and unseen accuracies?

[silicx]

Sorry for omitting these two values. In Tab. 2, the Seen accuracy is calculated only on seen classes (i.e. bias=a small negative value, like -1000) and Unseen accuracy is calculated only on unseen classes (i.e. bias=a large positive value, like +1000), which is also adopted by [33].

[Zoya-Hashmi]

Thank you, I understand now. Just one last question. Can you explain the metrics of causal paper, which you have used for the third table of readme file from your repository? Thanks alot!

[Zoya-Hashmi]

Causal metrics evaluate accuracies with zero bias, I got it! Thank you!

[ASMIftekhar]

Hello,
I have an additional question, As you (silicx) mentioned,
"In Tab. 2, the Seen accuracy is calculated only on seen classes (i.e. bias=a small negative value, like -1000) and Unseen accuracy is calculated only on unseen classes (i.e. bias=a large positive value, like +1000), which is also adopted by [33]."
Does this accuracy the top 1 or top 3?

[silicx]

@ASMIftekhar it's top 1