pre-training and fine-tuning framework for text generation
./prepare_data.sh
./train.sh
./inference.sh
Example: chat-bot
cd chat_bot
./fine_tune.sh
./inference.sh
./deploy.sh
-
12-layer, 768-hidden, 12-heads, Chinese (News + zhwiki, 200G) and English (Gigawords + Bookscorpus + enwiki, 60G)
-
24-layer, 768-hidden, 12-heads, Chinese (News + zhwiki, 200G) and English (Gigawords + Bookscorpus + enwiki, 60G)