why the embinit.save's length is one bigger than the length of the wordlist? about nsc

[lan2720]

Hi, I feel strange about it too.

Another thing puzzles me is where the wordlist.txt comes from. Since I try to generate IMDB wordlist using its train.txt and dev.txt. Just get 90000+ words, rather than 100000+ words as authors described.
Is there anything I missed?

[wangjialin114]

@lan2720 About the wordlist, i think you should just use the author gives to you.
About where it comes from , i guess , maybe the original dataset(that have not been split).Then use tokenization to get it.

[lan2720]

@wangjialin114 Thanks but I don't think so. Because the authors say they use the datasets provided by Tang. In Tang's zip file, train/dev/test data have been split already. And he didn't provide other files like wordlist.txt. So wordlist.txt was created by the authors using train and dev sets. But now I just cannot get the same result according to this idea.

[huimchen]

• @wangjialin114
  Thanks for your question. As you said, there is one "UNK" word embedding in the embinit.save, which is the last embedding.

[huimchen]

• @lan2720
  Thanks for your question. I get the wordlist.txt from a more original dataset in Tang 2015"Document Modeling with Gated Recurrent Neural Network for Sentiment Classification". And the dataset in this paper is a subset from the original dataset. Maybe I state not so clearly.

[lan2720]

@huimchen Thanks for your clear answer!