This code is incorrect for BNS. abs(norm.ppf(tpr) - norm.ppf(fpr))
And you must filter away words that occur < 3 times (tp+fp counts).
If you only want positively associated words, drop the abs().

from sklearn.base import BaseEstimator and make implementation

This is a followup to the issue Running library leaves lots of temporary directories

Turns out tmp directories are created as soon as the library is imported and they are not deleted afterwards. Steps to reproduce:

1. start python3 in the terminal
2. type the command
   from DocumentFeatureSelection import interface
3. Observe 5 new empty directories created in tmp/
4. Press Ctrl-D to exit python session
5. Observe tmp directories are still there

Regards,

Plans to make faster

1. process to make feature dictionary
2. Process to make element of csr matrix
3. Use multi-thread instead of multi-processing

http://scikit-learn.org/stable/modules/feature_extraction.html

consider to use gensim

• ラベルeが保持している文数(list)
• ラベルとindexの関係(dict)
• ラベルeに単語wが出現した回数(term-frequency matrix. Scipy.csr)

http://stackoverflow.com/questions/18041848/efficient-way-to-hold-and-process-a-big-dict-in-memory-in-python

• This part is taking a lot of time because of O^2 computation cost
• Computing SOA
• Computing PMI

Copied and pasted the example code to and got an Assertion Error as per the end of this post,
I'm running this code on Windows 10 and Python 3.6 with Anaconda.

I printed out the scored_matrix and found that some of the values are of inf and therefore caused the assertion error seeing that the assertion error checks the value for integer of float.

But I don't know why it will generates infinity values...

Thank you for making this package!

(0, 0) inf
(0, 1) inf
(0, 2) 1.309320092201233
(0, 3) inf
(0, 4) 0.7615997791290283
(0, 5) 0.27188003063201904
(0, 6) inf
(1, 0) inf
(1, 1) inf
(1, 2) 1.309320092201233
(1, 3) inf
(1, 4) 0.7615997791290283
(1, 5) 0.27188003063201904
(1, 6) inf

File "C:\Users\Ryan\Anaconda3\lib\site-packages\DocumentFeatureSelection\models.py", line 276, in convert_score_matrix2score_record
frequency_matrix=self.frequency_matrix

File "C:\Users\Ryan\Anaconda3\lib\site-packages\DocumentFeatureSelection\models.py", line 367, in get_feature_dictionary
weight_value_index_items = self.make_non_zero_information(weighted_matrix)

File "C:\Users\Ryan\Anaconda3\lib\site-packages\DocumentFeatureSelection\models.py", line 317, in make_non_zero_information
self.__get_value_index(row_indexes[i], column_indexes[i], weight_csr_matrix))

File "C:\Users\Ryan\Anaconda3\lib\site-packages\DocumentFeatureSelection\models.py", line 292, in __get_value_index
assert isinstance(row_index, (int, int32, int64))

AssertionError

This is because this line should be calculated by each doc

https://github.com/Kensuke-Mitsuzawa/DocumentFeatureSelection/blob/master/DocumentFeatureSelection/common/labeledMultiDocs2labeledDocsSet.py#L51

Problem

The interface is expecting dict object, however dict object tend to be super huge, and therefore there is no space left on memory.

Solution-1

Merit

Use corpus object of textacy.
This object can generate document-term matrix easily.

Negative point

It's not cleared this repository.

Solution-2

use shelve.

good point

python standard package

I have been looking through the examples in the repo of how BNS is computed and came across the following. As can be seen from the naming of the methods, in the 1st case it is Term Frequency meanwhile in the 2nd case it is Document Frequency. Could you please clarify which is correct?

1. in examples/basic_example.py that uses DocumentFeatureSelection/interface.py, line 128:

   matrix_data_object = data_converter.DataConverter().convert_multi_docs2term_frequency_matrix(...)

2. in tests/test_bns_python3.py line 105:

   data_csr_matrix = data_converter.DataConverter().labeledMultiDocs2DocFreqMatrix(...)

Thank you in advance,

this line causes error.

This is because from distutils.extension import Extension is not imported at this line.

scikit-learnのようにfit_transform()という名前で関数を作成する。

各indexごとに値を求めて、matrixを返すだけ

Problem

In computing PMI on size(input_matrix)=20 * 10972064 , a code consumes around 20G.

Way to decrease memory size

• for matrix objects
  • numpy.memmap
  • pytables
• for saving other objects
  • use persistent dict class for keeping it

Purpose

It makes this tool available also from any languages.

{'category-name': [ unigram-object, unigram-object, bigram-object ]}

This is an incorrect situation in the computation of TFIDF.
That should be different computation.

{'category-name': [ unigram-object, unigram-object ]}

{'category-name': [ bigram-object ]}

マルチプロセス化を検討のこと

https://github.com/Kensuke-Mitsuzawa/document_feature_selection/blob/master/document_feature_selection/pmi/pmi_csr_matrix.py#L181

I have been learning how to use this library (have certain interest in Binormal Separation) and when running tests with the command:

pyhton3 setup.py test

I noticed that there are lots of tmp directories created that are not deleted afterwards. Would it be possible to implement automatic cleanup in the library itself?

I am using Python 3.4.2