katsuya94 / hmm-lda Goto Github PK

Hi,
I was trying to re-implement your python code to c++ so that it can handle larger data. While doing so I realized that the function run_counts() is the main performance bottleneck. Specifically the loops where you compute
self.num_same_words_assigned_to_topic = [self.count_num_same_words_assigned_to_topic(word) for word in xrange(self.vocab_size)]
count_num_same_words_assigned_to_topic(word) is a 2D loop of documents * topics and it is called vocab_size times (one time for each word in the vocab. Calling this function for each word is redundant because it can be done in documents * topics iterations like so

for(int i = 0; i < n_documents; i++) {
	doc_size = documents[i].size();
	for(int j = 0; j < doc_size; j++) {
		word = documents[i][j];
		if(class_assignments[i][j] == 0)
			num_same_words_assigned_to_topic[word][topic_assignments[i][j]] += 1;
	}
}

Kindly note that this is a c++ code snippet but this can be very easily ported to python. This should give a good speed improvement in the overall performance. Correct me if I'm doing something wrong.