dabinat / cvtools Goto Github PK

Here you are joining separate words. Examples in Catalan:

• esmentar-les becomes esmentarles
• col·laborar becomes collaborar

Disallowed symbols should become white spaces before splitting the line.

The hyphen could be a valid character or not. It depends on the language and the kind of word tokenization we are doing. In any way, it cannot be removed. It must be replaced by a white space.

In Catalan l·l cannot be split. This is very language-specific.

In a sentence like L"'Admiralspalast", nel Mitte, ha un programma di varietà. is not detecting Admiralspalast. Probably the tool consider L"'Admiralspalast", as word.

Right now roman numbers are a problem because they cannot be interpreted, basically they are like symbols but composited.

Ideally is better to remove stuff like X, XIII, MCXX and so on.
Example list at https://gist.github.com/riverrun/ac91218bb1678b857c12#file-roman_numerals-py-L14

So we can append to the blacklist generated this stuff.