hrbrmstr / misinfo Goto Github PK

The current # words/document normalization is done based on overall word count BUT the phrases can have multiple words so we need a better normalization standard or support choices for normalization approaches.

there are plenty more bias phrases that can be added to existing lists and other biases that should be built-in. We should enhance these lists ASAP

The phrase matching is super naive. For example, may (from one of the lists) matches "May 10" vs "may have". We likely need to do more advanced NLP so we pick up part of speech to enable better tagging OR write some sort of preprocessor to pick up more common "mis-hits" before doing the comparison.