How to compute the reported EM for MMLU? And TydiQA on tulu7b is not reproducible about open-instruct

[BaohaoLiao]

Thank you for your details. I also realized that the sampling temperature might cause this variation.

[BaohaoLiao]

In addition, I also can't reproduce the reported F1 score for TdiQA, I obtain 44.6 rather than 45.7.

Scores:
{
    "russian": {
        "exact_match": 43.0,
        "f1": 61.294599844599816
    },
    "indonesian": {
        "exact_match": 44.0,
        "f1": 66.59124435225426
    },
    "swahili": {
        "exact_match": 22.0,
        "f1": 36.53859841982154
    },
    "arabic": {
        "exact_match": 24.0,
        "f1": 40.90610851423544
    },
    "english": {
        "exact_match": 63.0,
        "f1": 79.64825824035252
    },
    "finnish": {
        "exact_match": 42.0,
        "f1": 60.03515278266136
    },
    "bengali": {
        "exact_match": 4.0,
        "f1": 9.64920634920635
    },
    "korean": {
        "exact_match": 37.0,
        "f1": 46.19523809523808
    },
    "telugu": {
        "exact_match": 0.0,
        "f1": 0.6190476190476191
    },
    "average": {
        "f1": 44.60860602415744,
        "exact_match": 31.0
    }
}

[hamishivi]

Hi,

1. EM = exact match = accuracy as reported in our paper. 44.4 is pretty close to the 44.5 we report for Tulu 7b, the difference may be due to the diff process or some other small difference in our setups.
2. TydiQA has some variation based on model generations. I reran the script just now and got [43.4, 45.0, 44.3]. Considering your result is reasonably close to our reported result, I'm confident in saying that this is the result of this variation. The command to use to replicate these numbers is: python -m eval.tydiqa.run_eval --data_dir <data_dir> --n_shot 1 --max_num_examples_per_lang 100 --model_name_or_path tulu-7b --tokenizer_name_or_path tulu-7b --max_context_length 512 --eval_batch_size 8 --use_chat_format --load_in_8bit

In general, I think as many of our evaluations are generation-based, there may be some variation between our reported results and the results you get from reproduction. If these differences are small (e.g. < ~1-2 points), then it should be okay.