how to convert a new codegen model to ggml about turbopilot

[ravenscroftj]

hey there - great question. I haven't had time to run this myself but I think the answer might be in the tokenizer. If the tokenizer is producing the wrong token id inputs it could lead the model to produce garbage or just not do anything.

On line 18 the script loads the tokenizer for the model that is being converted -

print("Creating tokenizer")
tokenizer = CodeGenTokenizer.from_pretrained(args.code_model)

Now if you look in the config.json for the fine tuned model you see the tokenizer type is actually set to GPT2Tokenizer rather than CodeGenTokenizer. I don't know if it would be enough just to naively swap out the CodeGenTokenizer for a GPT2Tokenizer and run it again but maybe try that and post back?

[czkoko]

I tested other finetuned models with codegen_gptj_convert.py and succeeded, but the effect did not seem to be as good as the original codegen.

I noticed that ammarnasr/codegen-350M-multi-ft-on-code-x-glue-tc didn't seem to have finished training yet, and the state crashed.
https://wandb.ai/ammarnasr/codegen_ft_on_codexglue/runs/a8h4vc2c/overview?workspace=user-

[ravenscroftj]

Cool ok sounds like this is probably an incomplete/broken model so it isn't something that I'd be able to offer guidance on at present. Very happy to consider other models and architectures - I'm currently looking at santacoder as a possible alternative.