Can NOT reproduce the alpaca_eval results of OpenChat V3.1 13B about openchat

[imoneoi]

Ok, I'll test it if I have time

[imoneoi]

We started the OpenAI API server and ran the evaluation using alpaca_eval with a context length of 4096. Most of our evaluations are around 89%, with a standard deviation of about 1%.

BTW, are you using alpaca_eval_gpt4 or alpaca_eval_gpt4_fn? The latter would produce 5% lower results similar to what you observed.

[REIGN12]

We started the OpenAI API server and ran the evaluation using alpaca_eval with a context length of 4096. Most of our evaluations are around 89%, with a standard deviation of about 1%.

BTW, are you using alpaca_eval_gpt4 or alpaca_eval_gpt4_fn? The latter would produce 5% lower results similar to what you observed.

Many thanks for your kind response! I doulbe check my evaluation setting; I confirm I am using alpaca_eval_gpt4. This is very strange, since I do not run any inference on my own and just use the model_output.json you released on alpaca_eval.

Is this related to gpt-4 secretly update their api? Have you tested the results recently?

[imoneoi]

Chances are, GPT4 is constantly being updated. The previous results were all around 89%. I haven't tested it recently.

BTW, have you checked the number of rated responses? Sometimes assessments are incomplete.

[REIGN12]

Chances are, GPT4 is constantly being updated. The previous results were all around 89%. I haven't tested it recently.

BTW, have you checked the number of rated responses? Sometimes assessments are incomplete.

I understand your points. I double check the rated responses, the total number of rated responses are 804, with 3 of them are rated as tie.

[imoneoi]

@REIGN12 I've tested today using the output file, got 0.01% higher results

openchat-v3.1-13b         89.50            1.08      805     1484.00

[imoneoi]

OPENAI_API_KEYS=sk-XXX PYTHONPATH="$PYTHONPATH:$(pwd)/src" python -m alpaca_eval.main evaluate --model_outputs results/openchat-v3.1-13b/model_outputs.json --annotators_config alpaca_eval_gpt4

[REIGN12]

@REIGN12 I've tested today using the output file, got 0.01% higher results

openchat-v3.1-13b         89.50            1.08      805     1484.00

Many thanks for your reply! So here the results are tested with gpt-4-0314? I am going to double check my gpt-4 api version.

[REIGN12]

@REIGN12 I've tested today using the output file, got 0.01% higher results

openchat-v3.1-13b         89.50            1.08      805     1484.00

Many thanks for your reply! So here the results are tested with gpt-4-0314? I am going to double check my gpt-4 api version.

I double check my api version, I am using gpt-4-0314.

[imoneoi]

This is gpt-4 (latest), e.g. the alpaca_eval_gpt4 evaluator. I evaluated with the latest alpaca_eval and following commands:

OPENAI_API_KEYS=sk-XXX PYTHONPATH="$PYTHONPATH:$(pwd)/src" python -m alpaca_eval.main evaluate --model_outputs results/openchat-v3.1-13b/model_outputs.json --annotators_config alpaca_eval_gpt4