industria / erl_classifier Goto Github PK

In the module ec_classifier math:pow is used in calculating p(t|c) because we get a frequency distribution and the probability needs to be multiplied. But following the change to sum-log implementation p(t|c) should not be multiplied but summed like log(p(t|c)) so this needs to be changed:

Solutions could be either:

1. calculate log(P(t|c)) and multiply by frequency
2. Don't do a frequency distribution and just send a raw list of term id and do a simple sum. (This would do more lookups in the class term frequency table for large documents)

There is a lot of code copied from the Danish stemmer to the new Swedish stemmers although they are very similar so we should refactor to a scandinavian module containing the common functions, which will also make it quite easy to expand with z norweigan stemmer later.

Make the language of the classifier a configuration passed through like the classes from the .app file.

This is not going to enable classifiers with multiple languages, it will just make i possible to make an any-of classifier in swedish as well as in danish.