Evaluation metric (acc vs. acc_norm) for lm-evaluation-harness tasks about llm-pruner

[horseee]

Hi😄 . Thank you very much for the detailed notes and the experimental results you contributed based on the new version of the lm-evaluation-harness!

[horseee]

Hi. I used these keys:
select_key = {
'boolq': 'acc',
'hellaswag': 'acc_norm',
'arc_easy': 'acc',
'piqa': 'acc',
'arc_challenge': 'acc_norm',
'winogrande': 'acc',
'openbookqa': 'acc_norm',
}

Besides, I saw the difference in the performance of the original LLaMA-7B. I double-checked the code and used the evaluation code in my repo to re-evaluate the performance of LLaMA-7B, and I got a very similar performance (on ARC-easy, 67.38 vs. 67.45). Does the performance of LLaMA-7B you listed above use the lm-evaluation-harness (a previous commit of lm-evaluation-harness) in my repo? Since the lm-evaluation-harness has changed a lot in these months, some results are not consistent.

[bokyeong1015]

Thank you for your response and clarification!

I apologize for the missing commit hash regarding the LLaMA-7B results. Upon your indication, I realized that I used a different hash for lm-evaluation-harness compared to your code.

---

Since the lm-evaluation-harness has changed a lot in these months, some results are not consistent.

Thanks for pointing out this point. To ensure clarity, I've updated the table above and will soon provide results using your evaluation code. Thanks again for your assistance.

[bokyeong1015]

Hi, I've added the results using your repo, which are fully reproducible. I also made an explicit note to avoid any confusion. Big thanks for your time and help!

Note

• underline score: reported metrics in the LLM-Pruner paper
• The scores reported in LLM-Pruner paper are fully reproducible using this repo, and the lm-evaluation-harness version affects the scores because of recent updates.
• [Table 1 of LLM-Pruner Paper] The evaluation is performed under different prompts, which is lower than the official results in the LLaMA paper.