chenmengdx / nlp_baai

NLP models and codes for BAAI-JD joint project.

Introduction

This is project for releasing some open-source natural language models from Joint Lab of BAAI and JDAI. Different from other open-source Chinese NLP models, we mainly focus on some basic models for dialogue systems, especially in E-commerce domain. Our corpus is very huge, currently we are using 42 GB Customer Service Dialogue Data (CSDD) for training, and it contain about 1.2 billion sentences.

We provide the pre-trained BERT and word embeddings. The charts below shows the data we use.

Task	Data Source	Sentences	Sentence Pairs
BERT Pre-Training	Customer Service Dialogue Data(CSDD)	1.2B	-
FAQ	LCQMC、CSDD	-	0.88M for fine-tuning, 80K for test

Task	Data Source	Tokens	Vocabulary Size
Word Embedding Pre-Training	CSDD	9B	1M

Download

The links to the models are here.

Model	Data Source	Link
BAAI-JDAI-BERT, chinese	CSDD	JD-BERT for Tensorflow
BAAI-JDAI-WordEmbedding	CSDD	JD-WORD-EMBEDDING with 300d

JD-BERT.tar.gz file contains items:
  |—— BAAI-JDAI-BERT
      |—— bert_model.cpkt.*                  # pre-trained weights
      |—— bert_config.json                   # hyperparamters of the model
      |—— vocab.txt                          # vocabulary for WordPiece
      |—— JDAI-BERT.md & INTRO.md            # summary and details
JD-WORD-EMBEDDING.tar.gz file contains items:
  |—— BAAI-JDAI-WORD-EMBEDDING
      |—— JDAI-Word-Embedding.txt            # word vectors, each line separated by whitespace
      |—— JDAI-WORD-EMBEDDING.md & INTRO.md  # summary and details

BERT

Pre-training Settings

Masking	Dataset	Sentences	Training Steps	Device	Init Checkpoint	Init Lr
WordPiece	CSDD	1.2B	1M	P40×4	BERTGoogle weight	1e-4

• When pre-training, we do the data preprocessing with our preprocessor including some generalization processing.
• The init checkpoint we use is <12-layer, 768-hidden, 12-heads, 110M parameters>, and bert_config.json and vocab.txt are identical to Google's original settings.

Fine-tuning

We use train data of LCQMC, QQ dataset and QA dataset for fine-tuning, and then just train 2 epoches with a proper init learning rate 2e-5 on each dataset respectively. QQ dataset and QA dataset are extracted from other CSDD.

Evaluation

We evaluate our pre-trained model on the FAQ task with test data of LCQMC, QQ dataset and QA dataset.

FAQ

Model	LCQMC	QQ Test	QA Test
BERT-wwm	88.7	80.9	86.6
Our BERT	88.6	81.9	87.5

LCQMC，QQ Test and QA Test are the test data containing 5k question pairs, 21k question pairs and 54k question&answer pairs respectively.

Word Embedding

Pre-training Settings

Window Size	Dynamic Window	Sub-sampling	Low-frequency Word	Iter	Negative Samplingfor SGNS	Dim
5	Yes	1e-5	10	10	5	300

• When pre-training, we use our tools for preprocessing and word segmentaion.
• We train the vectors based on Skip-Gram.

Evaluation

We show top3 similar words for some sample words below. We use cosine distance to compute the distance of two words.

Input Word	口红	西屋	花花公子	蓝月亮	联想	骆驼
Similar 1	唇釉	典泛	PLAYBOY	威露士	宏碁	CAMEL
Similar 2	唇膏	法格	富贵鸟	增艳型	15IKB	骆驼牌
Similar 3	纪梵希	HS1250	霸王车	奥妙	14IKB	健足乐