danithaca / netizen Goto Github PK

从HTML->XML或者从XML->TXT有个很奇怪的现象：所有txt都会集中到
第一个node里
面，造成第一个node就是整个thread, 似乎有些不太对。
很难recreate. 一个例子是tianya: 143469

When there's a <img> tag in the HTML data, BeautifulSoup will ignore the
rest of the test after <img>. Need to fix somehow.

腐败 as verb or 腐败 as noun are treated as the same term in the current
TextNetwork.py script. This should be fine for our purpose, but I'm a
little concerned that it'll mess up the user dictionary.

People's Daily still has the title, author, 版面, 编号, etc mixed up with
the contents. Need to clean/extract the metadata from the text contents.

Some considerations:
1. Replies sometimes quote the earlier messages. The quoted messages should
be removed
2. BeautifulSoup sometimes don't work very well (eg, the <img> tag). Need
to work more.

In tianya-milk network, there are synonym terms CCTV, 央视, **电视台,
which should be collapsed together into CCTV. 

The current Chinese analyzer in Lucene uses the old ICTCLAS code. Perhaps
it can be set to use the new ICTCLAS library. This requires first to wrap
the ICTCLAS library in the StandardAnalyzer Lucene class.

However, this is not very important because Lucene now is only used for
search terms in text. The network building process is to first output the
terms, and then build the network.

The current logic is error-prone and perhaps not optimized for code-reuse.
Think about refactoring it.