askerlee / topicvec Goto Github PK

Hi Li, I'm trying to run PSDVec using Japanese Wikipedia text.

How did you deal with the stopwords in top1gram-wiki.txt? I ask you this because top2gram-wiki.txt is not uploaded on vectors&corpus Dropbox. And why there are no stopwords in top1gram-reuters.txt? I'd like to know when to delete the stopwords?

Sorry if it sound a stupid question.
Thank you!

Hi,

I've seen that reuters and rcv1 seems hardcoded into the code.

In order to use a corpus without any label (1 txt file per document), what is the simplest way to achieve it ?

Thank you for your help !

Hi!
I have a corpus of documents without categories, and I would generate topics without adding this kind of information. However, it seems that this code is only oriented to corpus with categories.
Is there any straight way to do this?

Hello!

I've just tried your last code, however, the *.bat scripts should be updated with the new flag adopted from 0.75 version forward within the file "topicExp.py".

For example, "-p" become "-i".

Bye

Hi askerlee, thanks for your great work!

Would this work on short text like tweets? If so, what parameters should I change?

Thanks.

Hi,
I have a large text file of about 2GB from which I want to form topic vector and visualize them through topic clouds. I wanted to ask which file should I use to generate topic vectors.
Another Question I want to ask is can I Generate topic vector using a CSV file having words and their counts across all documents?

Thanks

I am working on a dataset quite "noisy", so it's very difficult to exactly detect a sentence (for example, I have a lot of abbreviation with points, so these points are detected as the end of phrases).

Do you think that having many short sentences (often with just 3 words) could compromise the algorithm performances? Is it important to preserve the information about words belonging to a sentence?

PS: Furthermore, if the punctuation is filtered, the information about a "phrase" is completely lost, as documents became a bag of words, could it work also in this case?

hi all,

if i want to try it on other language, how can i train 25000-180000-500-BLK-8.0.vec.npy and get top1grams-wiki.txt?
for example chinese language, I have pre trained w2v model of chn wikipedia. Can I get these files from this pre trained model?
Thanks!

Hi askerlee!

I would ask you if the logLikelihood computed by "calcLoglikelihood" function is normalised by the number of words in the corpus?
If not, it could be easily done?

Thank you!

Hi, I noticed that in your paper 6.1, as the inefficiency of optimizing likelihood function including both Z and V, you choose to divide the process into two stages. First, get word embeddings and then take them as input in the second stage.

I wonder if it's ok when I input embeddings generated by other algorithm (e.g. word2vec ) instead of PSDvec.

I've tried it and got some wried results. My corpus includes 10000 docs that contains 3223788 validated words. The embedding as input is generated using w2v.

In iter1, loglike is 1.3e11, iter2 0.7e11, and as the process continues, the loglike keep decrease. Hence the best result always occurs after the first iterator instead of the last round. However, the output is quite reasonable based on "Most relevant words", but the strange behaviour of likelihood really bothers me.