kipok / nemo-skills Goto Github PK

Thanks a lot for releasing the models and the datasets!

I noticed that the link in the paper as well as the link in the README (We release a series of OpenMath models improved...) are broken.

Thank you for sharing the dataset.
I retrained the OpenMath dataset using Mistral-7B, but the accuracy for GSM8K is only 64% and for Math it is 24%. However, when using the nvidia/OpenMath-Mistral-7B-v0.1-hf from Hugging Face, the accuracy indeed reaches 80%.
I use the same praramter:
epoch=4
learning rate=1e-6
The training loss is as follow:

After 5000 steps, the performance remains almost unchanged, with GSM8K at 64 and Math at 24. ( I have tested checkpoint_5000, 10000, 15000)
Could you help me?

Exciting work! Could we use this to improve MMLU and other capability like coding now?

when run

python pipeline/run_labeling.py \
  --model_path <path to trtllm model> \
  --server_type tensorrt_llm \
  --output_dir ./synthetic-solutions/gsm8k-masked/ \
  --num_gpus 8 \
  --num_runs 128 \
  +prompt=code_base \
  ++prompt.few_shot_examples.examples_type=gsm8k_text_with_code \
  ++prompt.context_type=masked_solution \
  ++dataset=gsm8k-masked \
  ++split_name=train_full

The following error occurred:

Traceback (most recent call last):
  File "/code/nemo_skills/inference/generate_solutions.py", line 118, in generate_solutions
    prompts.append(get_prompt(cfg.prompt, input_dict=data_point))
  File "/code/nemo_skills/inference/prompt/utils.py", line 74, in get_prompt
    filled_examples.append(prompt_config.template.format(context=context.format(**example_dict), **example_dict))
KeyError: 'reference_masked_solution'

I use version v0.1, This error seems to be because the example in the text_with_code dictionary in the nemo_skills/inference/prompt/few_shot_examples/examples_gsm8k.py file does not have a reference_masked_solution field. Can you add it?

And The gsm8k-masked and math-masked datasets you provided are supposed to be reference_masked_solution fields, but you seem to have used the masked_reference_solution field.

Thanks for the awesome repo!

I was curious if you would consider open sourcing the generations from the models themselves used for evaluation? e.g., codellama 7b or codellama 70b etc

While the evaluation script you've provided is quite easy to run (and much appreciated!), from a cost side of things it takes a while to generate the solutions from multiple models, and it would be useful to have access to these generations for analysis & comparison.

Thanks for considering!