Perplexity about disan

[taoshen58]

I don’t think the perplexity is a evaluation metric for text classification task but language model. If you want the value, you can run the pretrained model and calculate the perplexity. Note that don’t forget to change the total loss (xentropy+l2) to the cross entropy loss only when evaluation.

[xuy2]

@taoshen58 For perplexity part, it seems that a word embedding matrix should also be multiplied. Which embedding matrix should be used? emb_mat_token or emb_mat_glove

[taoshen58]

Hi @xuy2

emb_mat_token includes the words from the training set where the embedding is initialized by glove if the corresponding word exists in the glove, otherwise is initialized by random.

emb_mat_glove is the words from glove where the overlapping words with emb_mat_token have been removed. This mat is certainly initialized by glove.

So, emb_mat_token need fine-tuned during training while the emb_mat_glove is not trainable but useful when testing.

[xuy2]

@taoshen58 It seems that I'd better use the matrix after generate_embedding_mat function, which includes 406079 words. I'll update the perplexity if my test is finished. Have you ever consider that using the disan model as transfer learning?

[taoshen58]

You can reduce the vocabulary size by using the word frequency in your corpus.

Is the transfer learning you mentioned similar to the OpenAI Transformer?

[xuy2]

https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf
Like this paper.