ikegami-yukino / madoka-python

Memory-efficient Count-Min Sketch Counter (based on Madoka C++ library)

Hi there 👋

madoka-python's Issues

Is it possible to filter out low counts?

Hi, thanks for the package.

I'm running word counts on large corpora (e.g. Wikipedia text) and found this package after Python dictionaries was giving me memory errors.

I'm new to sketching algorithms so forgive my ignorance but is it possible to filter out low counts after building the sketch object? For example, I'd like to get the top 1 million words sorted by counts in descending order. This is to save some memory.

Tangential question: once the counts are in a sketch object, it can't be converted to a dictionary because there is no key information?

Cheers

Installation issue

I'm having trouble pip installing this package. I'm using Python 3.7.1 and macOS High Sierra.

pip install madoka

Results in the following error

Collecting madoka
  Using cached https://files.pythonhosted.org/packages/da/eb/95288b1c4aa541eb296a6271e3f8c7ece03b78923ac47dbe95d2287d9f5e/madoka-0.7.1.tar.gz
Building wheels for collected packages: madoka
  Running setup.py bdist_wheel for madoka ... error
  Complete output from command /anaconda3/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/tf/8lz5km5n0n509s9y_f1wpj3h0000gn/T/pip-install-dg494dpd/madoka/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /private/var/folders/tf/8lz5km5n0n509s9y_f1wpj3h0000gn/T/pip-wheel-xxs7ruaf --python-tag cp37:
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.macosx-10.7-x86_64-3.7
  creating build/lib.macosx-10.7-x86_64-3.7/madoka
  copying madoka/madoka.py -> build/lib.macosx-10.7-x86_64-3.7/madoka
  copying madoka/__init__.py -> build/lib.macosx-10.7-x86_64-3.7/madoka
  running build_ext
  building '_madoka' extension
  creating build/temp.macosx-10.7-x86_64-3.7
  creating build/temp.macosx-10.7-x86_64-3.7/src
  gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/anaconda3/include -arch x86_64 -I/anaconda3/include -arch x86_64 -I/anaconda3/include/python3.7m -c madoka_wrap.cxx -o build/temp.macosx-10.7-x86_64-3.7/madoka_wrap.o -std=c++11
  warning: include path for stdlibc++ headers not found; pass '-std=libc++' on the command line to use the libc++ standard library instead [-Wstdlibcxx-not-found]
  madoka_wrap.cxx:3059:10: fatal error: 'stdexcept' file not found
  #include <stdexcept>
           ^~~~~~~~~~~
  1 warning and 1 error generated.
  error: command 'gcc' failed with exit status 1

  ----------------------------------------
  Failed building wheel for madoka
  Running setup.py clean for madoka
Failed to build madoka
Installing collected packages: madoka
  Running setup.py install for madoka ... error
    Complete output from command /anaconda3/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/tf/8lz5km5n0n509s9y_f1wpj3h0000gn/T/pip-install-dg494dpd/madoka/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/var/folders/tf/8lz5km5n0n509s9y_f1wpj3h0000gn/T/pip-record-eg369wuf/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_py
    creating build
    creating build/lib.macosx-10.7-x86_64-3.7
    creating build/lib.macosx-10.7-x86_64-3.7/madoka
    copying madoka/madoka.py -> build/lib.macosx-10.7-x86_64-3.7/madoka
    copying madoka/__init__.py -> build/lib.macosx-10.7-x86_64-3.7/madoka
    running build_ext
    building '_madoka' extension
    creating build/temp.macosx-10.7-x86_64-3.7
    creating build/temp.macosx-10.7-x86_64-3.7/src
    gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/anaconda3/include -arch x86_64 -I/anaconda3/include -arch x86_64 -I/anaconda3/include/python3.7m -c madoka_wrap.cxx -o build/temp.macosx-10.7-x86_64-3.7/madoka_wrap.o -std=c++11
    warning: include path for stdlibc++ headers not found; pass '-std=libc++' on the command line to use the libc++ standard library instead [-Wstdlibcxx-not-found]
    madoka_wrap.cxx:3059:10: fatal error: 'stdexcept' file not found
    #include <stdexcept>
             ^~~~~~~~~~~
    1 warning and 1 error generated.
    error: command 'gcc' failed with exit status 1

    ----------------------------------------
Command "/anaconda3/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/tf/8lz5km5n0n509s9y_f1wpj3h0000gn/T/pip-install-dg494dpd/madoka/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/var/folders/tf/8lz5km5n0n509s9y_f1wpj3h0000gn/T/pip-record-eg369wuf/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/var/folders/tf/8lz5km5n0n509s9y_f1wpj3h0000gn/T/pip-install-dg494dpd/madoka/