Import / Export in CSV about sci-corpus

[dpizetta]

Tiago, do you know how to solve this?

sqlite3.ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings.

Because if I convert into UNICODE, they show this:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 30: ordinal not in range(128)

Its just a local problem? Like some character that could not be decoded?

[tiagocampo]

No, its not a local problem!

Now we are using unicode to encode the chars. Maybe we can change this. I also have this problem and when I get this I go through the sentence and replace all the strange chars like [-, , /, etc]. Usually this solves the problem.

[dpizetta]

I fix it, the second problem was a local problem...with the file...I do cut and paste of some phrases and many strange characters still in the text. Now its OK and all of that are unicode. But I will treat what you comment as a exception maybe in another commit. TKS