nkthiebaut / zeugma

📝Natural language processing (NLP) utils: word embeddings (Word2Vec, GloVe, FastText, ...) and preprocessing transformers, compatible with scikit-learn Pipelines. 🛠

Hi 👋, I'm Nico

a data scientist with a strong interest in natural language processing, machine learning interpretability and fairness.

• 🔭 I’m currently working on ML @ Hired

• 🌱 I’m currently learning Graph neural networks

• 📝 I regularly write articles on https://data4thought.com/

• 📫 How to reach me [email protected]

Connect with me:

Languages and Tools:

 

zeugma's Issues

Error when install zeugma

Hello, am facing the following error when am installing zeugma
pip install zeugma

Collecting zeugma
Using cached zeugma-0.46.tar.gz (7.2 kB)
Note: you may need to restart the kernel to use updated packages.
ERROR: Command errored out with exit status 1:
command: 'C:\Users\space\anaconda3\python.exe' -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\Users\space\AppData\Local\Temp\pip-install-cnkvoo1g\zeugma\setup.py'"'"'; file='"'"'C:\Users\space\AppData\Local\Temp\pip-install-cnkvoo1g\zeugma\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' egg_info --egg-base 'C:\Users\space\AppData\Local\Temp\pip-pip-egg-info-4m8xro_z'
cwd: C:\Users\space\AppData\Local\Temp\pip-install-cnkvoo1g\zeugma
Complete output (7 lines):
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\space\AppData\Local\Temp\pip-install-cnkvoo1g\zeugma\setup.py", line 14, in
long_description=open("README.rst", "r").read(),
File "C:\Users\space\anaconda3\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 1013: character maps to
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

How to change the default download directory

Where does the model get downloaded when a call such as EmbeddingTransformer('glove') is made? And how can the default path be changed?

Error when loading embeddings model from local file...

If the file path contains upper case characters then it is not recognized because the EmbeddingTransformer __init__ method does a conversion to lower case before testing the os.path.exists(model)

Best regards

Olivier Terrier

Defining own tokenizer

It would be good if we were able to define our own tokenizer or be able to pass a list of tokens to transform. I'm using a workaround at the moment but in sklearn you're able to pass your own tokenizer and preprocessor.

Is there documentation?

I'm playing around with different word-embedding loaders. Is there documentation available for this one?

Issues with Gensim 4.0.0

I tried to run the basic example of zeugma, but it seems like zeugma has an issue with the newest version of gensim 4.0.0--at least for 'fastText'?

I try to run embeddings = fastText.transform(['what is zeugma', 'a figure of speech']) and get the following error:

AttributeError: The vocab attribute was removed from KeyedVector in Gensim 4.0.0.
Use KeyedVector's .key_to_index dict, .index_to_key list, and methods .get_vecattr(key, attr) and .set_vecattr(key, attr, new_val) instead.
See https://github.com/RaRe-Technologies/gensim/wiki/Migrating-from-Gensim-3.x-to-4

and

~/.local/lib/python3.7/site-packages/zeugma/embeddings.py in (x)
74 )
75 raw_tokens = raw_text.split()
---> 76 return list(filter(lambda x: x in self.model.vocab, raw_tokens))
77
78 tokens = preprocess_text(text)

Unfortunately, I need the newest gensim version elsewhere. Is there any workaround to get zeugma going?