Comments (7)
- I tried different strategies for keywords extraction. You can just use keyword or main_keywords.
- The time is document publish time. Category is the topic category, like "current events", "entertainment", "economy" and so on. It is determined by a topic classification model. As publish time is not always available in any datasets, therefore I don't use it.
- You can just calculate the IDF in the dataset, or from a news articles corpus. I think maybe I deleted some low-frequency words. You can check the code.
- That is what my previous project does:
https://arxiv.org/abs/1803.00189
And the code is also in my git repository.
from articlepairmatching.
This means word_to_ix doesn't contain the key ")". Can you check what it contains? ")" shall be contained in this dictionary.
from articlepairmatching.
in main.py
word_to_ix = {word: i for i, word in enumerate(W2V)}
I think maybe you have some decoding problem. I saw different platforms may have different encoding decoding issues for Chinese text. You can try to remove ".decode("utf-8")". Or you can check word_to_ix to see what is the problem.
from articlepairmatching.
thanks, I have solved the probelm.
from articlepairmatching.
Is it due to encoding?
from articlepairmatching.
Yes, and I also have some other questions.
1、The data contains keywords\main_keywords\ner_keywords, what's the differences? especially for the ner_keywords.
2、The data contains the category and time of two documents, but the code only uses the category1, why ? will the two features influence the model? and how are the categories divided?
dataset2featurefile(
"../../../../data/raw/event-story-cluster/same_event_doc_pair.txt",
"../../../../data/processed/event-story-cluster/same_event_doc_pair.cd.debug.json",
"label", "category1", "time1", "time2", "content1", "content2",
["keywords1", "ner_keywords1"], ["keywords2", "ner_keywords2"],
col_title1=None, col_title2=None, use_cd=True,
draw_fig=True, parallel=False, extract_range=range(2), print_fig=True)
3、Does the IDF.txt contain all words? and what kinds of methods do you use to get the model?
4、Whether it can be used to organize the development of a news along time?
from articlepairmatching.
Thanks for your explainations.
Great job!
from articlepairmatching.
Related Issues (20)
- File "C:\Users\username\Downloads\ArticlePairMatching-master\src\models\CCIG\data\resource_loader.py", line 53, in load_W2V_VOCAB W2V_VOCAB = pickle.load(open(W2V_VOCAB_PKL_FILE, 'rb')) UnpicklingError: invalid load key, 'v'. HOT 2
- File "/home/username/ArticlePairMatching/src/models/CCIG/data/ccig.py", line 30, in draw_ccig pos = gt.sfdp_layout(g) AttributeError: module 'graph_tool' has no attribute 'sfdp_layout' HOT 3
- How to get ner word by textrank? HOT 2
- 使用git clone时出现问题 HOT 1
- Can't find "same_event_doc_pair.no_cd.json". I have used 'git lfs clone' to clone the project. HOT 1
- 求解为啥安装graph-tool时,老是提示boost安装错误 HOT 2
- Is there any way to increase the speed of preprocessing?
- 有关实际使用的问题 HOT 1
- 请问你们使用的是什么分词工具?
- English
- 请问数据处理feature_extractor.py要多久,怎么我运行了三天一直是这个界面? HOT 11
- 请问对于英文数据集,用什么工具分词,分词用空格还是用什么隔开;特征和属性用什么方法抽取
- 数据集下载问题
- 运行feature_extractor.py报错
- 关于graph-tool的问题 HOT 2
- AssertionError HOT 3
- RuntimeError: Legacy autograd function with non-static forward method is deprecated. Please use new-style autograd function with static forward method.
- 测试时间问题 HOT 2
- Question about Time spent on training test dataset HOT 1
- RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x5 and 0x16)。有朋友遇到这个问题吗,维度不一致 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from articlepairmatching.