lipiji / guyu

Chinese GPT2: pre-training and fine-tuning framework for text generation

Guyu (谷雨)

pre-training and fine-tuning framework for text generation

backbone code for "An Empirical Investigation of Pre-Trained Transformer Language Models for Open-Domain Dialogue Generation": https://arxiv.org/abs/2003.04195

@article{DBLP:journals/corr/abs-2003-04195,
  author    = {Piji Li},
  title     = {An Empirical Investigation of Pre-Trained Transformer Language Models
               for Open-Domain Dialogue Generation},
  journal   = {CoRR},
  volume    = {abs/2003.04195},
  year      = {2020},
  url       = {https://arxiv.org/abs/2003.04195},
  archivePrefix = {arXiv},
  eprint    = {2003.04195},
  timestamp = {Tue, 10 Mar 2020 13:33:48 +0100}}
}

torch>=1.0

Pre-training:

./prepare_data.sh
./train.sh
./inference.sh

Fine-tuning

Example: chat-bot

cd chat_bot
./prepare_data.sh
./fine_tune.sh
./inference.sh

Web Api:

./deploy.sh

Pre-trained models

• 12-layer, 768-hidden, 12-heads, Chinese (News + zhwiki, 200G) and English (Gigawords + Bookscorpus + enwiki, 60G)

• 24-layer, 1024-hidden, 16-heads, Chinese (News + zhwiki, 200G) and English (Gigawords + Bookscorpus + enwiki, 60G) Note: please use transformer_preln as the main model:https://github.com/lipiji/Guyu/blob/master/biglm.py#L8

• download them: https://github.com/lipiji/Guyu/tree/master/model

References:

• GPT2: https://openai.com/blog/better-language-models/
• https://github.com/jcyk/BERT

guyu's Issues

Is there any Problem here?

Guyu/transformer_preln.py

Line 56 in 18d79cd

= self.ff_layer_norm(x)

Traceback (most recent call last):
File "inference_dev.py", line 10, in
from inference import *
File "../inference.py", line 9, in
from biglm import BIGLM
File "../biglm.py", line 8, in
from transformer_preln import TransformerLayer, Embedding, LearnedPositionalEmbedding, SelfAttentionMask
File "../transformer_preln.py", line 56
= self.ff_layer_norm(x)
^
IndentationError: unexpected indent

可用huggingface的代码转为pytorch模型吗

目前预训练模型是200G中文数据吗

目前预训练模型是200G中文数据吗

有tf版本的预训练模型或者是现有模型转换成tf模型的方式吗

top_k_inc会报错

输入是三个不同的句子的时候：

s = ["有一天", "早上好啊", "好累啊"]

用

top_k_inc(s)

会报错：

请问一下 论文中的Rank score 是怎么算的啊

Then we can select the most appropriate one with some ranking strategies such as likelihood or mutual information. The likelihood
ranking function 中的 P(yt) 是指的是什么啊？

可以提供国内的下载地址吗

您好！大牛！谷歌这个经常下载不全，能否提供一个国内网盘的下载链接？

.

finetuning里面的training data 字与字之间一定要以空格分隔吗？

为什么10G语料预训练的模型效果比200G语料预训练的模型好？

在给出的train.txt上微调时，发现10G语料预训练的模型的loss、ppl均要显著低于200G语料预训练的模型（两者均采用fine_tune.sh中的命令行参数），请问这是为什么呀？

./train.sh 为什么无法加载pretrain 的ckpt？

能写个requirement文件吗，谢谢

使用的语料会开源吗？

如题，使用的中文语料会开源吗？