yahshibu / nested-ner-tacl2020-flair Goto Github PK

Hi, I am running your model using both BERT and Flair, but it turns out that I get good results when training (i.e., on the validation set), but then when loading the best model to test on the testing partition, the results drop to almost 0 f1-score, what could be the problem? With the other embedding combinations, this does not happen.

Could you check if everything is working ok?

Regards

Hello.

In section 3.2 of your paper, you said GloVe is used for initializing but at this part, you used GoogleNews-vectors-negative300.bin. Could you please explain what is this?

Hi~

I have a question about the evaluation.
You have mentioned you ran experiments with different random seed 5 times, and you reported the averaged number on each table. But how do you calculate the averaged F1? I guess there are two methods,

1. average precision and recall scores of each experiment first, and then use these two numbers to calculate the F1 score.
2. calculate the F1 score of each experiment by using its precision and recall score first and then average these F1 scores.

But I think these two methods will result in different F1 score, which one is the method that you used in your experiments?

def f1(p, r):
    return 2 * p * r / (p + r)


def average(xs):
    return sum(xs) / len(xs)


precisions = [0.1, 0.2, 0.3]
recalls = [0.4, 0.5, 0.6]

print(f1(average(precisions), average(recalls)))  # method 1

f1s = [f1(p, r) for p, r in zip(precisions, recalls)]  # method 2
print(average(f1s))

# 0.28571428571428575
# 0.28190476190476194