KeyError: ')' about articlepairmatching

[BangLiu]

1. I tried different strategies for keywords extraction. You can just use keyword or main_keywords.
2. The time is document publish time. Category is the topic category, like "current events", "entertainment", "economy" and so on. It is determined by a topic classification model. As publish time is not always available in any datasets, therefore I don't use it.
3. You can just calculate the IDF in the dataset, or from a news articles corpus. I think maybe I deleted some low-frequency words. You can check the code.
4. That is what my previous project does:
   https://arxiv.org/abs/1803.00189
   And the code is also in my git repository.

[BangLiu]

This means word_to_ix doesn't contain the key ")". Can you check what it contains? ")" shall be contained in this dictionary.

[BangLiu]

in main.py
word_to_ix = {word: i for i, word in enumerate(W2V)}
I think maybe you have some decoding problem. I saw different platforms may have different encoding decoding issues for Chinese text. You can try to remove ".decode("utf-8")". Or you can check word_to_ix to see what is the problem.

[tingwt]

thanks, I have solved the probelm.

[BangLiu]

Is it due to encoding?

[tingwt]

Yes, and I also have some other questions.
1、The data contains keywords\main_keywords\ner_keywords, what's the differences? especially for the ner_keywords.
2、The data contains the category and time of two documents, but the code only uses the category1, why ? will the two features influence the model? and how are the categories divided?
dataset2featurefile(
"../../../../data/raw/event-story-cluster/same_event_doc_pair.txt",
"../../../../data/processed/event-story-cluster/same_event_doc_pair.cd.debug.json",
"label", "category1", "time1", "time2", "content1", "content2",
["keywords1", "ner_keywords1"], ["keywords2", "ner_keywords2"],
col_title1=None, col_title2=None, use_cd=True,
draw_fig=True, parallel=False, extract_range=range(2), print_fig=True)
3、Does the IDF.txt contain all words? and what kinds of methods do you use to get the model?
4、Whether it can be used to organize the development of a news along time?

[tingwt]

Thanks for your explainations.
Great job!