snipaid-nlg / gptj-model-tuning

Improving GPT-J for snippet generation.

GPT-J Model Tuning

We finetuned GPT-J for title and teaser snippet generation.

We evaluated the following approaches for model tuning:

• Classic model fine tuning
• Multitask fine tuning
• Prompt tuning

Results

Prompt tuning did not prove to be successfull.
Fine tuning and multitask fine tuning delivered promising results.

Finetuned models

We finetuned GPT-J-6B-8bit for title and teaser generation with multitask finetuning.
We finetuned on two datasets with different sizes.

Model	Capabilities	Dataset
gptj-title-teaser-1k	title and teaser generation	1.000 german online news from varying publishers
gptj-title-teaser-10k	title and teaser generation	10.000 german online news from varying publishers