mkonicek / nlp Goto Github PK

Hi Martin,

Amazing work, thank you for making this more accessible to beginners. I've recently been fascinated by the idea of using word embeddings to 'intersect' or 'combine' word meanings and was curious if you had any ideas on how to apply this using your code.

I got this idea from this paper: https://www.aclweb.org/anthology/W16-0203.pdf

Where you'd give it 2 words like:

flame and caring and it would output: cook

Or:

life and road and it would output: journey

Due to my unfamiliarity with the math they're using in the article, and how to implement this, I'm curious to know if you have any ideas on how to apply this using your existing code here. Like some new function that takes 2 words, or n words and combines their meanings or intersects them?

I am using fasttext pre-trained model based on english wikipedia. It works as expected...

https://github.com/shantanuo/pandas_examples/blob/master/nlp/fasttext_english.ipynb

But when I try the same code with some other language, I get an error as shown on this page...

https://github.com/shantanuo/pandas_examples/blob/master/nlp/fasttext_marathi.ipynb

The error is related to unicode:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 15: invalid start byte

I tried to open the file using Raw Binary option. I changed the function load_words_raw in load.py file:

with open(file_path, 'rb') as f:

And now I get a different error:

ValueError: could not convert string to float: b'\x00l\x02'

I have no idea how to handle this.