kyamagu / faiss-wheels Goto Github PK

View Code? Open in Web Editor NEW

260.0 7.0 31.0 182 KB

Unofficial faiss wheel builder

License: MIT License

Python 60.41% Shell 39.59%

python nearest-neighbor-search

faiss-wheels's People

Stargazers

Watchers

faiss-wheels's Issues

Can't install faiss-cpu-noavx2 on colab

Describe the bug
pip install faiss-cpu-noavx2 fails in a colab env:

!pip install faiss-cpu-noavx2
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Collecting faiss-cpu-noavx2
  Using cached faiss-cpu-noavx2-1.6.3.tar.gz (16 kB)
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: faiss-cpu-noavx2
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  Building wheel for faiss-cpu-noavx2 (setup.py) ... error
  ERROR: Failed building wheel for faiss-cpu-noavx2
  Running setup.py clean for faiss-cpu-noavx2
Failed to build faiss-cpu-noavx2

To Reproduce
Describe the steps to reproduce the behavior:
just create a colab and execute the command pip install faiss-cpu-noavx2

Desktop (please complete the following information):

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.5 LTS"
NAME="Ubuntu"
VERSION="20.04.5 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.5 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Python version: 3.9.16

Additional context
I was trying to install https://pypi.org/project/image-embeddings/ when this error occured
Here is a colab if you want to reproduce: https://colab.research.google.com/drive/1yU5lyoRZ3xJbwf_UHldQtfWc-Tlaca3n?usp=sharing

GPU package size limit

#33 seems to bring significant file size increase, causing pypi upload failure.
Needs to inspect the cause and find a workaround; perhaps by dropping some CUDA_ARCHITECTURE.

module 'faiss._swigfaiss' has no attribute 'delete_FloatVector'

On Google Colab:

following installation works fine:
python3 -m pip install --upgrade faiss-gpu
But during import import faiss gives me the following error:

3.I downgraded version and install
python3 -m pip install --upgrade faiss-gpu==1.7.1
works fine for me

CPython 3.12 wheels

(cc @kyamagu for visibility)

There are currently no CPython 3.12 wheels on PyPI (https://pypi.org/project/faiss-cpu/#files). This causes an error on platforms using CPython 3.12 which are unable to build faiss-cpu from source.

Can the contributors publish a new minor/patch release that includes CPython 3.12 for all platforms?

Example output, on an M1 Mac running CPython 3.12:

...
Building wheels for collected packages: faiss-cpu
  Building wheel for faiss-cpu (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for faiss-cpu (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [12 lines of output]
      running bdist_wheel
      running build
      running build_py
      running build_ext
      building 'faiss._swigfaiss' extension
      swigging faiss/faiss/python/swigfaiss.i to faiss/faiss/python/swigfaiss_wrap.cpp
      swig -python -c++ -Doverride= -I/usr/local/include -Ifaiss -doxygen -module swigfaiss -o faiss/faiss/python/swigfaiss_wrap.cpp faiss/faiss/python/swigfaiss.i
      Traceback (most recent call last):
        File "/opt/miniconda3/envs/jupyter-ai-jl4-pip/bin/swig", line 5, in <module>
          from swig import swig
      ModuleNotFoundError: No module named 'swig'
      error: command '/opt/miniconda3/envs/jupyter-ai-jl4-pip/bin/swig' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for faiss-cpu
Failed to build faiss-cpu
ERROR: Could not build wheels for faiss-cpu, which is required to install pyproject.toml-based projects

Support M1 mac

It seems the following fix enables faiss build on ARM64 mac. Good to investigate further packaging issues.
facebookresearch/faiss#1655

Missing python_callbacks.h in sdist build

sdist package does not bundle python_callbacks.h required for building a wheel package.

There is an issue that the source tree has hard-coded #include <faiss/python/python_callbacks.h>, that should be located inside the C++ header location. One option is to apply a patch to rewrite this to #include "python_callbacks.h" and bundle the source in the source package.

Move to CMake build

See facebookresearch/faiss#1313

why did faiss.ProductQuantizer() get an incorrect result？

code：
import numpy as np
import faiss

ver = np.random.randn(1000,512)
ver = ver .astype(np.float32)
print(ver .shape)
pq = faiss.ProductQuantizer(512, 4, 3) #dim:512 seg:4 bit:3
pq.train(ver)
codes = pq.compute_codes(ver)
print(codes.shape)
print(codes)

result： #got seg:2
(1000, 512)
(1000, 2)
[[ 58 5]
[ 62 7]
[ 63 11]
...
[171 7]
[ 57 7]
[ 92 1]]

Check faiss configure output

In faiss/Makefile, the following is always true in CI environment regardless of CUDA installation. Check why.

ifneq ($(strip $(NVCC)),)
	OBJ         += $(GPU_OBJ)
	HEADERS     += $(GPU_HEADERS)
endif

faiss-gpu: index_cpu_to_gpu() hangs (doesn't occur with the conda package)

When I install faiss-gpu via pip, index_cpu_to_gpu() seems to hang forever. For example, this code sample hangs for me:

import faiss

index_flat = faiss.IndexFlatL2(16)
gpu_index_flat = faiss.index_cpu_to_gpu(faiss.StandardGpuResources(), 0, index_flat)

The index_cpu_to_gpu() will hang, spinning a CPU at 100% seemingly forever, mostly in libnvidia-ptxjitcompiler.so.510.54.

I confirmed this on two different machines with different GPUs (A100 and A10G).

I found two workarounds:

Downgrade to faiss==1.5.3. I hit the problem with 1.6.0 or later.
Use the conda package. I don't observe the issue with conda, even with the latest version of faiss-gpu (1.7.2)

Maybe faiss-gpu wheel isn't built with CUDA 11 support, and that's why it doesn't work with A100 / A10G?

Running setup.py install for faiss-gpu did not run successfully

Describe the bug
Running setup.py install for faiss-gpu did not run successfully
cuda_11.7
But CPU Versions works!

(env) PS C:\Server\htdocs\ml2> pip install faiss-gpu

  Downloading faiss-gpu-1.6.4.post2.tar.gz (25 kB)
  Preparing metadata (setup.py) ... done
  WARNING: Generating metadata for package faiss-gpu produced metadata for project name faiss-cpu. Fix your #egg=faiss-gpu fragments.
Discarding https://files.pythonhosted.org/packages/7d/00/b3aaad408a44e4f5d87ebfcd75d0b14eeaed9fe9bc7a9f5e185ff1d503d6/faiss-gpu-1.6.4.post2.tar.gz (from https://pypi.org/simple/faiss-gpu/): Requested faiss-cpu from https://files.pythonhosted.org/packages/7d/00/b3aaad408a44e4f5d87ebfcd75d0b14eeaed9fe9bc7a9f5e185ff1d503d6/faiss-gpu-1.6.4.post2.tar.gz has inconsistent name: expected 'faiss-gpu', but metadata has 'faiss-cpu'
  Downloading faiss-gpu-1.6.4.tar.gz (3.4 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [7 lines of output]
      running egg_info
      creating C:\Users\szcze\AppData\Local\Temp\pip-pip-egg-info-k360kexh\faiss_cpu.egg-info
      writing C:\Users\szcze\AppData\Local\Temp\pip-pip-egg-info-k360kexh\faiss_cpu.egg-info\PKG-INFO
      writing dependency_links to C:\Users\szcze\AppData\Local\Temp\pip-pip-egg-info-k360kexh\faiss_cpu.egg-info\dependency_links.txt
      writing top-level names to C:\Users\szcze\AppData\Local\Temp\pip-pip-egg-info-k360kexh\faiss_cpu.egg-info\top_level.txt
      writing manifest file 'C:\Users\szcze\AppData\Local\Temp\pip-pip-egg-info-k360kexh\faiss_cpu.egg-info\SOURCES.txt'
      error: package directory 'C:\Users\szcze\AppData\Local\Temp\pip-install-8jg13zcy\faiss-gpu_62ba4c34a62d4ef6981c53e51f8b7158\faiss\faiss\python' does not exist
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

To Reproduce
I am using Windows 10 and Python 3.8.10 (venv)
cuda_11.7.r11.7/compiler.31442593_0

(env) PS C:\Server\htdocs\ml2> pip install faiss-gpu

Expected behavior
A clear and concise description of what you expected to happen.

Desktop (please complete the following information):

OS: Win 10
Architecture: [e.g. x86_64, arm64]
Python: 3.8
Version:3.8.2

Additional context
Add any other context about the problem here.

Cannot install faiss-gpu on Arch Linux

Describe the bug
I am trying to install faiss-gpu.

To Reproduce

pip install faiss-gpu

outputs

Defaulting to user installation because normal site-packages is not writeable
ERROR: Could not find a version that satisfies the requirement faiss-gpu (from versions: none)
ERROR: No matching distribution found for faiss-gpu

I even tried to download the .whl file from https://pypi.org/project/faiss-gpu/#files and input

pip install --use-wheel --no-index --find-links=[FILEPATH] faiss-gpu

Expected behavior
Normal installation like any other pip package

Desktop (please complete the following information):

OS: Artix Linux
Architecture: x64
Python: 3.11 but also tried with pyenv -> pyenv global 3.9.8. Same results
Version: 1.7.2

Additional context
I am trying to download/install this, in order to make install this https://github.com/Mangio621/Mangio-RVC-Fork
In the above's makefile there is this line basically pip install faiss-gpu fairseq gradio ffmpeg ffmpeg-python praat-parselmouth pyworld numpy==1.23.5 numba==0.56.4 librosa==0.9.1 and it fails at the very first, others have no problem if i run them by themselves.

Installation fails in clean python env (numpy requirement before installation)

Hey,

First of all: Thanks for maintaining the unofficial PyPI packages!

I noticed that with the latest 1.6.4. the installation via pip fails if you don't run the install from an env already having numpy installed. This is particularly problematic in fresh deployments / dockerfiles.
I see that you are using numpy here in setup.py. From my perspective, there should ideally be no external packages needed before the installation. Could you maybe move back to standard python modules there?

faiss-wheels/setup.py

Line 31 in c11b8de

INCLUDE_DIRS = [np.get_include(), FAISS_INCLUDE]

Error message:

Collecting faiss-cpu (from -r requirements.txt (line 19))
  Downloading https://files.pythonhosted.org/packages/5d/3d/8ab01d4421c3889e8d3379661f5453ebbbec209f91ee4d067af13ca18a13/faiss-cpu-1.6.4.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-iiioulxg/faiss-cpu/setup.py", line 6, in <module>
        import numpy as np
    ModuleNotFoundError: No module named 'numpy'
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-iiioulxg/faiss-cpu/
The command '/bin/sh -c pip3 install -r requirements.txt' returned a non-zero code: 1

Error when installing faiss-gpu 1.7.2 on Windows

File "C:\Users<user>\AppData\Local\Temp\pip-install-nppl0xcx\faiss-gpu_3375d4d88f02454c85440c4a6dcee4e7\setup.py", line 58, in _remove_flag
compiler = self.compiler.compiler
AttributeError: 'MSVCCompiler' object has no attribute 'compiler'

ERROR: Failed building wheel for faiss-gpu

Compatibility issue with faiss-cpu wheels on arm64 vs x86

Describe the bug
FAISS has different wheels for arm64 chips & Intel/AMD chips. If you vectorise a document using arm64, you should use FAISS to interact with this document on a arm64 machine.

So right now, my mac is arm64 and I vectorised docs on my mac using the arm64 wheel. However in the Azure functions, it attempts to use the Intel wheel which is slightly different. Intel/AMD chips use something called AVX2 support in FAISS (no idea what it actually is but it's different to the way arm64 machines use FAISS).

How can this be solved?
Normalise the wheels across all architectures to solve this compatibility issue problem - happy to help?

faiss-cpu 1.7.4 not working with mkl 2023.1.0

Describe the bug
For environment with faiss-cpu 1.7.4 and mkl 2023.1.0, python import faiss attempt to load libmkl_intel_lp64.so.1 while only libmkl_intel_lp64.so.2 is installed.

To Reproduce
Describe the steps to reproduce the behavior:

Installation is from anaconda and using conda channels: defaults, pytorch, conda-forge

conda config --add channels defaults
conda config --add channels pytorch
conda config --add channels conda-forge
conda update conda

conda install faiss-cpu==1.7.4 mkl=2023.1.0

import faiss

Expected behavior
Successful import

Actual behavior

>>> import faiss
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/miniconda/lib/python3.8/site-packages/faiss/__init__.py", line 16, in <module>
    from .loader import *
  File "/opt/miniconda/lib/python3.8/site-packages/faiss/loader.py", line 65, in <module>
    from .swigfaiss import *
  File "/opt/miniconda/lib/python3.8/site-packages/faiss/swigfaiss.py", line 13, in <module>
    from . import _swigfaiss
ImportError: libmkl_intel_lp64.so.1: cannot open shared object file: No such file or directory

Desktop (please complete the following information):

OS: [e.g. Linux]
Architecture: [e.g. x86_64]
Python: [e.g. 3.8]
Version: [e.g. v1.7.4]

Additional context
Discovered issue when use faiss-cpu and mkl under Azure Machine Learning environment.

AttributeError: 'MSVCCompiler' object has no attribute 'compiler' when trying to install faiss-cpu on Windows 11 and Python 3.8.10

Describe the bug
When running Windows 11 and Python 3.8.10 attempting to install the latest version of faiss-cpu I get AttributeError: 'MSVCCompiler' object has no attribute 'compiler'

To Reproduce

pip install faiss-cpu

Expected behavior
faiss-cpu should be installed just fine.

Desktop (please complete the following information):

OS: Windows 11
Architecture: x86_64
Python: 3.18.10
Version: 1.7.3

Additional context
I can resolve the problem by installing a previous version

pip install faiss-cpu==1.7.2

so it is not a blocker. I think I noticed on my other machine with Python 3.7 that faiss-cpu installed just fine without changing the version. I dont have it on hand to add any other details.

Problems building custom wheels

Hi, I've been trying to follow the instructions to build faiss wheels with AVX2 and MKL.

I clone the repository, build faiss without python and GPU and run:

> export FAISS_OPT_LEVEL='avx2'
> export FAISS_LDFLAGS='-l:libfaiss_avx2.a -lgfortran'
> pip install --no-binary :all: faiss-cpu
Collecting faiss-cpu
  Using cached faiss-cpu-1.6.5.tar.gz (28 kB)
Skipping wheel build for faiss-cpu, due to binaries being disabled for it.
Installing collected packages: faiss-cpu
    Running setup.py install for faiss-cpu ... error
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-ejk03m5r/faiss-cpu/setup.py'"'"'; __file__='"'"'/tmp/pip-install-ejk03m5r/faiss-cpu/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-f04b57bf/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.8/faiss-cpu
         cwd: /tmp/pip-install-ejk03m5r/faiss-cpu/
    Complete output (60 lines):
    running install
    running build
    running build_py
    running build_ext
    building 'faiss._swigfaiss' extension
    swigging faiss/faiss/python/swigfaiss.i to faiss/faiss/python/swigfaiss_wrap.cpp
    swig -python -c++ -Doverride= -I/usr/local/include -Ifaiss -DSWIGWORDSIZE64 -o faiss/faiss/python/swigfaiss_wrap.cpp faiss/faiss/python/swigfaiss.i
    faiss/faiss/python/swigfaiss.i:229: Error: Unable to find 'faiss/impl/platform_macros.h'
    faiss/faiss/python/swigfaiss.i:233: Error: Unable to find 'faiss/utils/ordered_key_value.h'
    faiss/faiss/python/swigfaiss.i:234: Error: Unable to find 'faiss/utils/Heap.h'
    faiss/faiss/python/swigfaiss.i:235: Error: Unable to find 'faiss/utils/partitioning.h'
    faiss/faiss/python/swigfaiss.i:236: Error: Unable to find 'faiss/utils/hamming.h'
    faiss/faiss/python/swigfaiss.i:339: Error: Unable to find 'faiss/utils/utils.h'
    faiss/faiss/python/swigfaiss.i:340: Error: Unable to find 'faiss/utils/distances.h'
    faiss/faiss/python/swigfaiss.i:341: Error: Unable to find 'faiss/utils/random.h'
    faiss/faiss/python/swigfaiss.i:343: Error: Unable to find 'faiss/MetricType.h'
    faiss/faiss/python/swigfaiss.i:344: Error: Unable to find 'faiss/Index.h'
    faiss/faiss/python/swigfaiss.i:345: Error: Unable to find 'faiss/Clustering.h'
    faiss/faiss/python/swigfaiss.i:347: Error: Unable to find 'faiss/utils/extra_distances.h'
    faiss/faiss/python/swigfaiss.i:351: Error: Unable to find 'faiss/impl/ProductQuantizer.h'
    faiss/faiss/python/swigfaiss.i:353: Error: Unable to find 'faiss/VectorTransform.h'
    faiss/faiss/python/swigfaiss.i:354: Error: Unable to find 'faiss/IndexPreTransform.h'
    faiss/faiss/python/swigfaiss.i:355: Error: Unable to find 'faiss/IndexFlat.h'
    faiss/faiss/python/swigfaiss.i:356: Error: Unable to find 'faiss/IndexLSH.h'
    faiss/faiss/python/swigfaiss.i:357: Error: Unable to find 'faiss/impl/PolysemousTraining.h'
    faiss/faiss/python/swigfaiss.i:358: Error: Unable to find 'faiss/IndexPQ.h'
    faiss/faiss/python/swigfaiss.i:359: Error: Unable to find 'faiss/InvertedLists.h'
    faiss/faiss/python/swigfaiss.i:360: Error: Unable to find 'faiss/DirectMap.h'
    faiss/faiss/python/swigfaiss.i:363: Error: Unable to find 'faiss/IndexIVF.h'
    faiss/faiss/python/swigfaiss.i:368: Error: Unable to find 'faiss/IVFlib.h'
    faiss/faiss/python/swigfaiss.i:369: Error: Unable to find 'faiss/impl/ScalarQuantizer.h'
    faiss/faiss/python/swigfaiss.i:370: Error: Unable to find 'faiss/IndexScalarQuantizer.h'
    faiss/faiss/python/swigfaiss.i:371: Error: Unable to find 'faiss/IndexIVFSpectralHash.h'
    faiss/faiss/python/swigfaiss.i:372: Error: Unable to find 'faiss/impl/HNSW.h'
    faiss/faiss/python/swigfaiss.i:373: Error: Unable to find 'faiss/IndexHNSW.h'
    faiss/faiss/python/swigfaiss.i:374: Error: Unable to find 'faiss/IndexIVFFlat.h'
    faiss/faiss/python/swigfaiss.i:379: Error: Unable to find 'faiss/OnDiskInvertedLists.h'
    faiss/faiss/python/swigfaiss.i:382: Error: Unable to find 'faiss/impl/lattice_Zn.h'
    faiss/faiss/python/swigfaiss.i:383: Error: Unable to find 'faiss/IndexLattice.h'
    faiss/faiss/python/swigfaiss.i:386: Error: Unable to find 'faiss/IndexIVFPQ.h'
    faiss/faiss/python/swigfaiss.i:387: Error: Unable to find 'faiss/IndexIVFPQR.h'
    faiss/faiss/python/swigfaiss.i:388: Error: Unable to find 'faiss/Index2Layer.h'
    faiss/faiss/python/swigfaiss.i:390: Error: Unable to find 'faiss/IndexBinary.h'
    faiss/faiss/python/swigfaiss.i:391: Error: Unable to find 'faiss/IndexBinaryFlat.h'
    faiss/faiss/python/swigfaiss.i:392: Error: Unable to find 'faiss/IndexBinaryIVF.h'
    faiss/faiss/python/swigfaiss.i:393: Error: Unable to find 'faiss/IndexBinaryFromFloat.h'
    faiss/faiss/python/swigfaiss.i:394: Error: Unable to find 'faiss/IndexBinaryHNSW.h'
    faiss/faiss/python/swigfaiss.i:395: Error: Unable to find 'faiss/IndexBinaryHash.h'
    faiss/faiss/python/swigfaiss.i:397: Error: Unable to find 'faiss/impl/ThreadedIndex.h'
    faiss/faiss/python/swigfaiss.i:401: Error: Unable to find 'faiss/IndexShards.h'
    faiss/faiss/python/swigfaiss.i:405: Error: Unable to find 'faiss/IndexReplicas.h'
    faiss/faiss/python/swigfaiss.i:409: Error: Unable to find 'faiss/MetaIndexes.h'
    faiss/faiss/python/swigfaiss.i:594: Error: Unable to find 'faiss/impl/io.h'
    faiss/faiss/python/swigfaiss.i:595: Error: Unable to find 'faiss/index_io.h'
    faiss/faiss/python/swigfaiss.i:596: Error: Unable to find 'faiss/clone_index.h'
    faiss/faiss/python/swigfaiss.i:600: Error: Unable to find 'faiss/AutoTune.h'
    faiss/faiss/python/swigfaiss.i:601: Error: Unable to find 'faiss/index_factory.h'
    faiss/faiss/python/swigfaiss.i:602: Error: Unable to find 'faiss/MatrixStats.h'
    faiss/faiss/python/swigfaiss.i:848: Error: Unable to find 'faiss/impl/AuxIndexStructures.h'
    error: command 'swig' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-ejk03m5r/faiss-cpu/setup.py'"'"'; __file__='"'"'/tmp/pip-install-ejk03m5r/faiss-cpu/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-f04b57bf/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.8/faiss-cpu Check the logs for full command output.
WARNING: You are using pip version 20.1.1; however, version 20.3.3 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.

First of all, I notice that there is no swigfaiss.i file — only swigfaiss.swig. So I rename it and start building the local package instead of the pypi one:

> export FAISS_OPT_LEVEL='avx2'
> export FAISS_LDFLAGS='-l:libfaiss_avx2.a -lgfortran'
> python3 setup.py install

However, this fails because ldd cannot find libfaiss_avx2.a. Then I specify it like so:

> export FAISS_LDFLAGS='-Lfaiss/build/faiss/libfaiss_avx2.a -lgfortran'

This results in a successful compilation. However, when I try to import faiss:

> root@fb3557b7b9e5:/opt/instance/faiss/faiss_debug/faiss-wheels-compile/faiss-wheels2# ipython3
Python 3.8.5 (default, Jul 28 2020, 12:59:40)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import faiss
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
/usr/local/lib/python3.8/dist-packages/faiss_cpu-1.6.5-py3.8-linux-x86_64.egg/faiss/loader.py in <module>
     30         logger.info("Loading faiss with AVX2 support.")
---> 31         from .swigfaiss_avx2 import *
     32     else:

ModuleNotFoundError: No module named 'faiss.swigfaiss_avx2'

During handling of the above exception, another exception occurred:

ImportError                               Traceback (most recent call last)
<ipython-input-1-9e6a9f0b5bbf> in <module>
----> 1 import faiss

/usr/local/lib/python3.8/dist-packages/faiss_cpu-1.6.5-py3.8-linux-x86_64.egg/faiss/__init__.py in <module>
     15
     16 # We import * so that the symbol foo can be accessed as faiss.foo.
---> 17 from .loader import *
     18
     19

/usr/local/lib/python3.8/dist-packages/faiss_cpu-1.6.5-py3.8-linux-x86_64.egg/faiss/loader.py in <module>
     37     # we import * so that the symbol X can be accessed as faiss.X
     38     logger.info("Loading faiss.")
---> 39     from .swigfaiss import *

/usr/local/lib/python3.8/dist-packages/faiss_cpu-1.6.5-py3.8-linux-x86_64.egg/faiss/swigfaiss.py in <module>
     11 # Import the low-level C/C++ module
     12 if __package__ or "." in __name__:
---> 13     from . import _swigfaiss
     14 else:
     15     import _swigfaiss

ImportError: /usr/local/lib/python3.8/dist-packages/faiss_cpu-1.6.5-py3.8-linux-x86_64.egg/faiss/_swigfaiss.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZTIN5faiss9IndexFlatE

Okay, one more try. Beside building the faiss, I also install it using cmake --install build. Then, again, I run:

> pip install --no-binary :all: faiss-spu

This also compiles successfully, but:

In [1]: import faiss
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
/usr/local/lib/python3.8/dist-packages/faiss/loader.py in <module>
     30         logger.info("Loading faiss with AVX2 support.")
---> 31         from .swigfaiss_avx2 import *
     32     else:

ModuleNotFoundError: No module named 'faiss.swigfaiss_avx2'

During handling of the above exception, another exception occurred:

ImportError                               Traceback (most recent call last)
<ipython-input-1-9e6a9f0b5bbf> in <module>
----> 1 import faiss

/usr/local/lib/python3.8/dist-packages/faiss/__init__.py in <module>
     15
     16 # We import * so that the symbol foo can be accessed as faiss.foo.
---> 17 from .loader import *
     18
     19

/usr/local/lib/python3.8/dist-packages/faiss/loader.py in <module>
     37     # we import * so that the symbol X can be accessed as faiss.X
     38     logger.info("Loading faiss.")
---> 39     from .swigfaiss import *

/usr/local/lib/python3.8/dist-packages/faiss/swigfaiss.py in <module>
     11 # Import the low-level C/C++ module
     12 if __package__ or "." in __name__:
---> 13     from . import _swigfaiss
     14 else:
     15     import _swigfaiss

ImportError: /usr/local/lib/python3.8/dist-packages/faiss/_swigfaiss.cpython-38-x86_64-linux-gnu.so: undefined symbol: dgemm_

Looks like the README instructions are not complete and the swigfaiss.swig in the repo should be renamed. However, I still cannot build the wheel. Please advise

Windows 10 import error

Hey, It seems like the same issue as #38 prevails even in 1.7.1:

    import faiss
C:\Python\lib\site-packages\faiss\__init__.py:18: in <module>
    from .loader import *
C:\Python\lib\site-packages\faiss\loader.py:65: in <module>
    from .swigfaiss import *
C:\Python\lib\site-packages\faiss\swigfaiss.py:13: in <module>
    from . import _swigfaiss
E   ImportError: DLL load failed while importing _swigfaiss: The specified module could not be found.

Windows 10, Python 3.9

how to use？

hi,dear
any demo to test the faiss?
thx

AttributeError: 'MSVCCompiler' object has no attribute 'compiler'

i install faiss-gpu by both pip install faiss-gpu and conda install faiss-gpu.
however, results is below:
Using cached faiss-gpu-1.6.3.tar.gz (17 kB)
Requirement already satisfied: numpy in d:\anaconda3\envs\deep_sort_pytorch\lib\site-packages (from faiss-gpu) (1.18.2)
Building wheels for collected packages: faiss-gpu
Building wheel for faiss-gpu (setup.py) ... error
ERROR: Command errored out with exit status 1:
command: 'D:\anaconda3\envs\deep_sort_pytorch\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\Users\ly\AppData\
Local\Temp\pip-install-ti6ccsjw\faiss-gpu_4d5a94b24e0c468b855399b4ebb64aeb\setup.py'"'"'; file='"'"'C:\Users\ly\AppData\Local\Temp\
pip-install-ti6ccsjw\faiss-gpu_4d5a94b24e0c468b855399b4ebb64aeb\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().r
eplace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\ly\AppData\Local\Temp\pip-
wheel-3x2ioo54'
cwd: C:\Users\ly\AppData\Local\Temp\pip-install-ti6ccsjw\faiss-gpu_4d5a94b24e0c468b855399b4ebb64aeb
Complete output (36 lines):
running bdist_wheel
running build
running build_ext
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\ly\AppData\Local\Temp\pip-install-ti6ccsjw\faiss-gpu_4d5a94b24e0c468b855399b4ebb64aeb\setup.py", line 111, in
ext_modules=[swigfaiss]
File "D:\anaconda3\envs\deep_sort_pytorch\lib\site-packages\setuptools_init.py", line 153, in setup
return distutils.core.setup(**attrs)
File "D:\anaconda3\envs\deep_sort_pytorch\lib\distutils\core.py", line 148, in setup
dist.run_commands()
File "D:\anaconda3\envs\deep_sort_pytorch\lib\distutils\dist.py", line 955, in run_commands
self.run_command(cmd)
File "D:\anaconda3\envs\deep_sort_pytorch\lib\distutils\dist.py", line 974, in run_command
cmd_obj.run()
File "D:\anaconda3\envs\deep_sort_pytorch\lib\site-packages\wheel\bdist_wheel.py", line 299, in run
self.run_command('build')
File "D:\anaconda3\envs\deep_sort_pytorch\lib\distutils\cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "D:\anaconda3\envs\deep_sort_pytorch\lib\distutils\dist.py", line 974, in run_command
cmd_obj.run()
File "D:\anaconda3\envs\deep_sort_pytorch\lib\distutils\command\build.py", line 135, in run
self.run_command(cmd_name)
File "D:\anaconda3\envs\deep_sort_pytorch\lib\distutils\cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "D:\anaconda3\envs\deep_sort_pytorch\lib\distutils\dist.py", line 974, in run_command
cmd_obj.run()
File "C:\Users\ly\AppData\Local\Temp\pip-install-ti6ccsjw\faiss-gpu_4d5a94b24e0c468b855399b4ebb64aeb\setup.py", line 45, in run
build_ext.run(self)
File "D:\anaconda3\envs\deep_sort_pytorch\lib\distutils\command\build_ext.py", line 339, in run
self.build_extensions()
File "C:\Users\ly\AppData\Local\Temp\pip-install-ti6ccsjw\faiss-gpu_4d5a94b24e0c468b855399b4ebb64aeb\setup.py", line 50, in build_extensions
self._remove_flag('-Wstrict-prototypes')
File "C:\Users\ly\AppData\Local\Temp\pip-install-ti6ccsjw\faiss-gpu_4d5a94b24e0c468b855399b4ebb64aeb\setup.py", line 58, in _remove_flag
compiler = self.compiler.compiler
AttributeError: 'MSVCCompiler' object has no attribute 'compiler'

Faiss-cpu requires numpy, but numpy not installed as a dependency of faiss-cpu

Describe the bug
Doing a simple pip install of faiss-cpu and then importing the library into python will fail because it depends on numpy.

To Reproduce
Describe the steps to reproduce the behavior:

python3 -m pip install faiss-cpu
python3 "import faiss"

Sees the following error

Collecting faiss-cpu
  Downloading faiss_cpu-1.7.4-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (3.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.2/3.2 MB 24.8 MB/s eta 0:00:00
Installing collected packages: faiss-cpu
Successfully installed faiss-cpu-1.7.4
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Traceback (most recent call last):
  File "/io/test-script.py", line 1, in <module>
    import faiss
  File "/usr/local/lib/python3.10/dist-packages/faiss/__init__.py", line 11, in <module>
    import numpy as np
ModuleNotFoundError: No module named 'numpy'

Expected behavior
Installing faiss-cpu should pull in all its dependencies automatically.

Desktop (please complete the following information):

OS: Linux
Architecture: x86_64 or aarch64
Python: 3.9
Version: latest

Additional context
Add any other context about the problem here.

Segfault when used with tensorflow

Thanks for working on these wheels. They are very useful.

On Debian Stretch with Python 3.6, I get a segfault if I try to import faiss before tensorflow, but not vice-versa.

This should reproduce the issue:

docker run --rm -it python:3.6-stretch bash
pip install faiss-cpu tensorflow
python -c "import faiss; import tensorflow" # segfaults
python -c "import tensorflow; import faiss" # works

The segfault only occurs with faiss-cpu, it does not occur with faiss-gpu.

windows 10 python39 error

import faiss
Traceback (most recent call last):
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python39\lib\site-packages\faiss\loader.py", line 34, in
from .swigfaiss import *
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python39\lib\site-packages\faiss\swigfaiss.py", line 13, in
from . import _swigfaiss
ImportError: DLL load failed while importing _swigfaiss: 找不到指定的模块。

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "", line 1, in
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python39\lib\site-packages\faiss_init_.py", line 17, in
from .loader import *
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python39\lib\site-packages\faiss\loader.py", line 39, in
from .swigfaiss import *
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python39\lib\site-packages\faiss\swigfaiss.py", line 13, in
from . import _swigfaiss
ImportError: DLL load failed while importing _swigfaiss: 找不到指定的模块。

Add support to release linux aarch64 wheels

Problem

On aarch64, ‘pip install faiss-gpu’ builds the wheels from source code and gives the below error-

ERROR: Command errored out with exit status 1: 
   command: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-mjddlp5q/faiss-gpu/setup.py'"'"'; __file__='"'"'/tmp/pip-install-mjddlp5q/faiss-gpu/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-r241ub16 
       cwd: /tmp/pip-install-mjddlp5q/faiss-gpu/ 
  Complete output (75 lines): 
  running bdist_wheel 
  running build 
  running build_py 
  running build_ext 
  building 'faiss._swigfaiss' extension 
  swigging faiss/faiss/python/swigfaiss.i to faiss/faiss/python/swigfaiss_wrap.cpp 
  swig -python -c++ -Doverride= -I/usr/local/include -Ifaiss -DSWIGWORDSIZE64 -o faiss/faiss/python/swigfaiss_wrap.cpp faiss/faiss/python/swigfaiss.i 
  faiss/faiss/python/swigfaiss.i:250: Error: Unable to find 'faiss/impl/platform_macros.h' 
  faiss/faiss/python/swigfaiss.i:254: Error: Unable to find 'faiss/utils/ordered_key_value.h' 
  faiss/faiss/python/swigfaiss.i:255: Error: Unable to find 'faiss/utils/Heap.h' 
  faiss/faiss/python/swigfaiss.i:260: Error: Unable to find 'faiss/utils/AlignedTable.h' 
  faiss/faiss/python/swigfaiss.i:261: Error: Unable to find 'faiss/utils/partitioning.h' 
  faiss/faiss/python/swigfaiss.i:262: Error: Unable to find 'faiss/utils/hamming.h' 
  faiss/faiss/python/swigfaiss.i:365: Error: Unable to find 'faiss/utils/utils.h' 
  faiss/faiss/python/swigfaiss.i:366: Error: Unable to find 'faiss/utils/distances.h' 
  faiss/faiss/python/swigfaiss.i:367: Error: Unable to find 'faiss/utils/random.h' 
  faiss/faiss/python/swigfaiss.i:369: Error: Unable to find 'faiss/MetricType.h' 
  faiss/faiss/python/swigfaiss.i:372: Error: Unable to find 'faiss/Index.h' 
  faiss/faiss/python/swigfaiss.i:373: Error: Unable to find 'faiss/Clustering.h' 
  faiss/faiss/python/swigfaiss.i:375: Error: Unable to find 'faiss/utils/extra_distances.h' 
  faiss/faiss/python/swigfaiss.i:379: Error: Unable to find 'faiss/impl/ProductQuantizer.h' 
  faiss/faiss/python/swigfaiss.i:380: Error: Unable to find 'faiss/impl/AdditiveQuantizer.h' 
  faiss/faiss/python/swigfaiss.i:381: Error: Unable to find 'faiss/impl/ResidualQuantizer.h'

Resolution

On aarch64, ‘pip install faiss-gpu’ should download the wheels from PyPI

@kyamagu Please let me know your interest in releasing aarch64 wheels.

Move to 1.6.3?

It would be really nice if this pip package could install faiss 1.6.3 official version.
I would have liked to make a PR for this but I am not sure how to proceed.

Get a CUDA error when searching in a Faiss index having too small vectors

Describe the bug
Get a CUDA error when searching in a Faiss index having too small vectors

Faiss assertion 'err__ == cudaSuccess' failed in void faiss::gpu::ivfInterleavedScanImpl_32_(faiss::gpu::Tensor<float, 2, true>&, faiss::gpu::Tensor<long int, 2, true>&, faiss::gpu::DeviceVector<void*>&, faiss::gpu::DeviceVector<void*>&, faiss::gpu::IndicesOptions, faiss::gpu::DeviceVector<int>&, int, faiss::MetricType, bool, faiss::gpu::Tensor<float, 3, true>&, faiss::gpu::GpuScalarQuantizer*, faiss::gpu::Tensor<float, 2, true>&, faiss::gpu::Tensor<long int, 2, true>&, faiss::gpu::GpuResources*) at /project/faiss/faiss/gpu/impl/scan/IVFInterleaved32.cu:13; details: CUDA error 9 invalid configuration argument
        Aborted (core dumped)

Faiss assertion 'err__ == cudaSuccess' failed in int faiss::gpu::getNumDevices() at /project/faiss/faiss/gpu/utils/DeviceUtils.cu:36; details: CUDA error 401 the operation cannot be performed in the present state

To Reproduce

import faiss
import numpy as np
nb_vectors = 80_000
dim = 128
k = 5
nlist = 10
nprobe = 2
vectors = np.random.rand(nb_vectors, dim)
index = faiss.index_factory(dim, "IVF16384,Flat")
index.nlist = nlist
index.nprobe = nprobe
options = faiss.GpuMultipleClonerOptions()
options.shard = True
options.common_ivf_quantizer = True
index = faiss.index_cpu_to_all_gpus(index, options)
index.train(vectors)
index.add(vectors)
results = index.search(vectors, k)

This code works with dim = 1024.

I also tried with different indexes and different parameters (nlist, etc.) but it always fails for a certain vector size (and not when increasing the size).

When I try to install different versions of Faiss (nightly and old version) I face incompatibility issues such as:

AttributeError: module 'faiss._swigfaiss' has no attribute 'delete_ParameterRangeVector'

TypeError: in method 'GpuIndexIVFFlat_train', argument 3 of type 'float const *'

Desktop:

OS: Linux Ubuntu 20.04.6 LTS
GPU: Tesla V100 / Tesla A100
Architecture: x86_64
Python: 3.9.17
Version: v1.7.3 (faiss-gpu @ https://github.com/kyamagu/faiss-wheels/releases/download/v1.7.3/faiss_gpu-1.7.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl#sha256=1a9755132bb81dc3daecd16e0b5471ddf0246e555a889ea311813f867fdcca88)

Illegal instruction (core dumped) on py3.7

hi, kyamagu,

Thanks for this repo!

Because I need py3.7, so I have to install faiss on py3.8

But without luck, here is the minial reproduce process:

docker run --rm  --name Test-service -ti python:3.7-buster sh

// then in the container

# pip install faiss-cpu
WARNING: The directory '/root/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting faiss-cpu
  Downloading faiss_cpu-1.6.1-cp37-cp37m-manylinux2010_x86_64.whl (7.1 MB)
     |████████████████████████████████| 7.1 MB 330 kB/s 
Collecting numpy
  Downloading numpy-1.18.1-cp37-cp37m-manylinux1_x86_64.whl (20.1 MB)
     |████████████████████████████████| 20.1 MB 278 kB/s 
Installing collected packages: numpy, faiss-cpu
Successfully installed faiss-cpu-1.6.1 numpy-1.18.1
# python -c "import faiss"
Illegal instruction (core dumped)

But python:3.8 seems fine. Have any suggestion? Thanks in advance!

I also search the Illegal instruction (core dumped) issues in facebook's faiss repo, but most of them are 1.5.3 problem.

ImportError: DLL load failed: The specified module could not be found.

ImportError: DLL load failed: The specified module could not be found.

Can anyone help me with this? I've installed with pip install faiss-cpu and python version is 3.7.9

It's running good in colab, but not in local pc

I've installed library using pip install faiss-cpu, but while importing import faiss, it's returning error

ModuleNotFoundError: No module named 'faiss.swigfaiss_avx2'

Hi,

With faiss-cpu on linux I get an error:

Traceback:
env/lib/python3.6/site-packages/faiss/loader.py:31: in <module>
    from .swigfaiss_avx2 import *
E   ModuleNotFoundError: No module named 'faiss.swigfaiss_avx2'

During handling of the above exception, another exception occurred:
env/lib/python3.6/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
xxx/tests/handlers/xxx.py:6: in <module>
    from xxx.handlers import xxx as xxx_handler
xxx/handlers/xxx.py:2: in <module>
    import faiss
env/lib/python3.6/site-packages/faiss/__init__.py:17: in <module>
    from .loader import *
env/lib/python3.6/site-packages/faiss/loader.py:39: in <module>
    from .swigfaiss import *
env/lib/python3.6/site-packages/faiss/swigfaiss.py:13: in <module>
    from . import _swigfaiss
E   ImportError: dlopen: cannot load any more object with static TLS

Seems related to other issues, but I'm still not sure of the solution:
facebookresearch/faiss#866
https://github.com/facebookresearch/faiss/issues?q=is%3Aissue+swigfaiss_avx2

segfault when doing index.search using faiss-cpu

Running the faiss example

import numpy as np
d = 64                           # dimension
nb = 100000                      # database size
nq = 10000                       # nb of queries
np.random.seed(1234)             # make reproducible
xb = np.random.random((nb, d)).astype('float32')
xb[:, 0] += np.arange(nb) / 1000.
xq = np.random.random((nq, d)).astype('float32')
xq[:, 0] += np.arange(nq) / 1000.

import faiss                   # make faiss available
index = faiss.IndexFlatL2(d)   # build the index
print(index.is_trained)
index.add(xb)                  # add vectors to the index
print(index.ntotal)

k = 4                          # we want to see 4 nearest neighbors
D, I = index.search(xb[:5], k) # sanity check
print(I)
print(D)
D, I = index.search(xq, k)     # actual search
print(I[:5])                   # neighbors of the 5 first queries
print(I[-5:])                  # neighbors of the 5 last queries

I'm getting a segfault when doing the search.

OS: ubuntu 20.04
version : 1.6.3 of faiss-cpu

any idea what could be wrong ?

Can't find headers when trying to use existing binaries

I'm trying to follow https://github.com/kyamagu/faiss-wheels#linux so I can build my own binaries without AVX2

rom1504@rom1504-W35-37ET:~/faiss$ pip install --no-binary :all: faiss-cpu
Defaulting to user installation because normal site-packages is not writeable
Collecting faiss-cpu
  Using cached faiss-cpu-1.6.3.tar.gz (17 kB)
Requirement already satisfied: numpy in /usr/lib/python3/dist-packages (from faiss-cpu) (1.16.2)
Skipping wheel build for faiss-cpu, due to binaries being disabled for it.
Installing collected packages: faiss-cpu
    Running setup.py install for faiss-cpu ... error
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-ti_pc5up/faiss-cpu/setup.py'"'"'; __file__='"'"'/tmp/pip-install-ti_pc5up/faiss-cpu/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-sq3a3ca8/install-record.txt --single-version-externally-managed --user --prefix= --compile --install-headers /home/rom1504/.local/include/python3.7m/faiss-cpu                                                                                                                                  
         cwd: /tmp/pip-install-ti_pc5up/faiss-cpu/                                                                                                                          
    Complete output (56 lines):                                                                                                                                             
    running install                                                                                                                                                         
    running build                                                                                                                                                           
    running build_ext                                                                                                                                                       
    building 'faiss._swigfaiss' extension                                                                                                                                   
    swigging python/swigfaiss.i to python/swigfaiss_wrap.cpp                                                                                                                
    swig -python -c++ -Doverride= -I/usr/local/include/faiss -DSWIGWORDSIZE64 -o python/swigfaiss_wrap.cpp python/swigfaiss.i                                               
    python/swigfaiss.i:230: Error: Unable to find 'faiss/utils/Heap.h'                                                                                                      
    python/swigfaiss.i:231: Error: Unable to find 'faiss/utils/hamming.h'                                                                                                   
    python/swigfaiss.i:312: Error: Unable to find 'faiss/utils/utils.h'

These headers are actually present :

rom1504@rom1504-W35-37ET:~/faiss$ ls /usr/local/include/faiss
AutoTune.h     Index2Layer.h           IndexBinaryHNSW.h  IndexHNSW.h     IndexIVFPQR.h           IndexPreTransform.h     IVFlib.h               utils
clone_index.h  IndexBinaryFlat.h       IndexBinaryIVF.h   index_io.h      IndexIVFSpectralHash.h  IndexReplicas.h         MatrixStats.h          VectorTransform.h
Clustering.h   IndexBinaryFromFloat.h  index_factory.h    IndexIVFFlat.h  IndexLattice.h          IndexScalarQuantizer.h  MetaIndexes.h
DirectMap.h    IndexBinary.h           IndexFlat.h        IndexIVF.h      IndexLSH.h              IndexShards.h           MetricType.h
impl           IndexBinaryHash.h       Index.h            IndexIVFPQ.h    IndexPQ.h               InvertedLists.h         OnDiskInvertedLists.h
rom1504@rom1504-W35-37ET:~/faiss$ ls /usr/local/include/faiss/utils/
distances.h  extra_distances.h  hamming.h  hamming-inl.h  Heap.h  random.h  utils.h  WorkerThread.h

What could be wrong ?

Move to Github Actions from Travis CI

Github Actions support docker containers to easily build manylinux wheels.
Check how to install and build dependencies for faiss wheels.

install faiss-cpu error in win10

hi,dear
when I install the faiss-cpu with
pip install faiss-cpu
then I got the error

Requirement already satisfied: numpy in d:\python\lib\site-packages (from faiss-cpu) (1.16.4)
Building wheels for collected packages: faiss-cpu
  Running setup.py bdist_wheel for faiss-cpu: started
  Running setup.py bdist_wheel for faiss-cpu: finished with status 'error'
  Complete output from command d:\python\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\\xulm1\\AppData\\Local\\Temp\\pip-install-o6e2hn4b\\faiss-cpu\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d C:\Users\xulm1\AppData\Local\Temp\pip-wheel-8av9rpam --python-tag cp36:
  running bdist_wheel
  running build
  running build_ext
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "C:\Users\xulm1\AppData\Local\Temp\pip-install-o6e2hn4b\faiss-cpu\setup.py", line 107, in <module>
      ext_modules=[_swigfaiss]
    File "d:\python\lib\site-packages\setuptools\__init__.py", line 145, in setup
      return distutils.core.setup(**attrs)
    File "d:\python\lib\distutils\core.py", line 148, in setup
      dist.run_commands()
    File "d:\python\lib\distutils\dist.py", line 955, in run_commands
      self.run_command(cmd)
    File "d:\python\lib\distutils\dist.py", line 974, in run_command
      cmd_obj.run()
    File "d:\python\lib\site-packages\wheel\bdist_wheel.py", line 192, in run
      self.run_command('build')
    File "d:\python\lib\distutils\cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "d:\python\lib\distutils\dist.py", line 974, in run_command
      cmd_obj.run()
    File "d:\python\lib\distutils\command\build.py", line 135, in run
      self.run_command(cmd_name)
    File "d:\python\lib\distutils\cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "d:\python\lib\distutils\dist.py", line 974, in run_command
      cmd_obj.run()
    File "C:\Users\xulm1\AppData\Local\Temp\pip-install-o6e2hn4b\faiss-cpu\setup.py", line 45, in run
      build_ext.run(self)
    File "d:\python\lib\distutils\command\build_ext.py", line 339, in run
      self.build_extensions()
    File "C:\Users\xulm1\AppData\Local\Temp\pip-install-o6e2hn4b\faiss-cpu\setup.py", line 50, in build_extensions
      self._remove_flag('-Wstrict-prototypes')
    File "C:\Users\xulm1\AppData\Local\Temp\pip-install-o6e2hn4b\faiss-cpu\setup.py", line 58, in _remove_flag
      compiler = self.compiler.compiler
  AttributeError: 'MSVCCompiler' object has no attribute 'compiler'

So could you pls help me?
thx

inconsistent name: expected 'faiss-gpu', but metadata has 'faiss-cpu'

Describe the bug
Requested faiss-cpu from https://files.pythonhosted.org/packages/7d/00/b3aaad408a44e4f5d87ebfcd75d0b14eeaed9fe9bc7a9f5e185ff1d503d6/faiss-gpu-1.6.4.post2.tar.gz has inconsistent name: expected 'faiss-gpu', but metadata has 'faiss-cpu'

To Reproduce
Describe the steps to reproduce the behavior:
pip install faiss-gpu

Desktop (please complete the following information):

OS: macOS 12.6.6
Architecture: x86_64
Python: 3.11.3

Windows build

Faiss now supports windows conda build.

For wheels, there needs to be stable BLAS/LAPACK distribution that CMake can find. Perhaps OpenBLAS is the first option, but it is unknown how to resolve internal fortran-related dependency in msbuild environment.

Release v1.7.4 on PyPI

The latest release of faiss-wheels is currently v1.7.4, but the latest release on PyPI is v1.7.2. This means that faiss-gpu cannot be used with python 3.11 at the moment, even though support was already added in v1.7.3. Any chance we can update PyPI? Happy to help if there is work involved.

ImportError: DLL load failed while importing _swigfaiss: The specified module could not be found.

Hello, I am getting this error on windows 10 with python 3.9, faiss 1.7.2 after importing faiss. Same happens with conda-forge version.

Release date for faiss-cpu that includes numpy dependency fix?

Support was added to download the correct dependencies back in June, when will a new minor release be cut to incorporate this change?

AttributeError: 'MSVCCompiler' object has no attribute 'compiler'

use pip install faiss-cpugot error, here is the log

Building wheels for collected packages: faiss-cpu
  Building wheel for faiss-cpu (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: 'C:\ProgramData\Anaconda3\envs\blink37\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\hx\\AppData\\Local\\Temp\\pip-install-drqmfmaj\\faiss-cpu\\setup.py'"'"'; __file__='"'"'C:\\Users\\hx\\AppData\\Local\\Temp\\pip-install-drqmfmaj\\faiss-cpu\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\hx\AppData\Local\Temp\pip-wheel-3rrts8id'
       cwd: C:\Users\hx\AppData\Local\Temp\pip-install-drqmfmaj\faiss-cpu\
  Complete output (36 lines):
  running bdist_wheel
  running build
  running build_ext
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "C:\Users\hx\AppData\Local\Temp\pip-install-drqmfmaj\faiss-cpu\setup.py", line 107, in <module>
      ext_modules=[_swigfaiss]
    File "C:\ProgramData\Anaconda3\envs\blink37\lib\site-packages\setuptools\__init__.py", line 165, in setup
      return distutils.core.setup(**attrs)
    File "C:\ProgramData\Anaconda3\envs\blink37\lib\distutils\core.py", line 148, in setup
      dist.run_commands()
    File "C:\ProgramData\Anaconda3\envs\blink37\lib\distutils\dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "C:\ProgramData\Anaconda3\envs\blink37\lib\distutils\dist.py", line 985, in run_command
      cmd_obj.run()
    File "C:\ProgramData\Anaconda3\envs\blink37\lib\site-packages\wheel\bdist_wheel.py", line 223, in run
      self.run_command('build')
    File "C:\ProgramData\Anaconda3\envs\blink37\lib\distutils\cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "C:\ProgramData\Anaconda3\envs\blink37\lib\distutils\dist.py", line 985, in run_command
      cmd_obj.run()
    File "C:\ProgramData\Anaconda3\envs\blink37\lib\distutils\command\build.py", line 135, in run
      self.run_command(cmd_name)
    File "C:\ProgramData\Anaconda3\envs\blink37\lib\distutils\cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "C:\ProgramData\Anaconda3\envs\blink37\lib\distutils\dist.py", line 985, in run_command
      cmd_obj.run()
    File "C:\Users\hx\AppData\Local\Temp\pip-install-drqmfmaj\faiss-cpu\setup.py", line 45, in run
      build_ext.run(self)
    File "C:\ProgramData\Anaconda3\envs\blink37\lib\distutils\command\build_ext.py", line 340, in run
      self.build_extensions()
    File "C:\Users\hx\AppData\Local\Temp\pip-install-drqmfmaj\faiss-cpu\setup.py", line 50, in build_extensions
      self._remove_flag('-Wstrict-prototypes')
    File "C:\Users\hx\AppData\Local\Temp\pip-install-drqmfmaj\faiss-cpu\setup.py", line 58, in _remove_flag
      compiler = self.compiler.compiler
  AttributeError: 'MSVCCompiler' object has no attribute 'compiler'
  ----------------------------------------
  ERROR: Failed building wheel for faiss-cpu
  Running setup.py clean for faiss-cpu
Failed to build faiss-cpu
Installing collected packages: faiss-cpu
    Running setup.py install for faiss-cpu ... error
    ERROR: Command errored out with exit status 1:
     command: 'C:\ProgramData\Anaconda3\envs\blink37\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\hx\\AppData\\Local\\Temp\\pip-install-drqmfmaj\\faiss-cpu\\setup.py'"'"'; __file__='"'"'C:\\Users\\黄 祥\\AppData\\Local\\Temp\\pip-install-drqmfmaj\\faiss-cpu\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\hx\AppData\Local\Temp\pip-record-qsj3c0ie\install-record.txt' --single-version-externally-managed --compile --install-headers 'C:\ProgramData\Anaconda3\envs\blink37\Include\faiss-cpu'
         cwd: C:\Users\hx\AppData\Local\Temp\pip-install-drqmfmaj\faiss-cpu\
    Complete output (38 lines):
    running install
    running build
    running build_ext
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\hx\AppData\Local\Temp\pip-install-drqmfmaj\faiss-cpu\setup.py", line 107, in <module>
        ext_modules=[_swigfaiss]
      File "C:\ProgramData\Anaconda3\envs\blink37\lib\site-packages\setuptools\__init__.py", line 165, in setup
        return distutils.core.setup(**attrs)
      File "C:\ProgramData\Anaconda3\envs\blink37\lib\distutils\core.py", line 148, in setup
        dist.run_commands()
      File "C:\ProgramData\Anaconda3\envs\blink37\lib\distutils\dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "C:\ProgramData\Anaconda3\envs\blink37\lib\distutils\dist.py", line 985, in run_command
        cmd_obj.run()
      File "C:\ProgramData\Anaconda3\envs\blink37\lib\site-packages\setuptools\command\install.py", line 61, in run
        return orig.install.run(self)
      File "C:\ProgramData\Anaconda3\envs\blink37\lib\distutils\command\install.py", line 545, in run
        self.run_command('build')
      File "C:\ProgramData\Anaconda3\envs\blink37\lib\distutils\cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "C:\ProgramData\Anaconda3\envs\blink37\lib\distutils\dist.py", line 985, in run_command
        cmd_obj.run()
      File "C:\ProgramData\Anaconda3\envs\blink37\lib\distutils\command\build.py", line 135, in run
        self.run_command(cmd_name)
      File "C:\ProgramData\Anaconda3\envs\blink37\lib\distutils\cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "C:\ProgramData\Anaconda3\envs\blink37\lib\distutils\dist.py", line 985, in run_command
        cmd_obj.run()
      File "C:\Users\hx\AppData\Local\Temp\pip-install-drqmfmaj\faiss-cpu\setup.py", line 45, in run
        build_ext.run(self)
      File "C:\ProgramData\Anaconda3\envs\blink37\lib\distutils\command\build_ext.py", line 340, in run
        self.build_extensions()
      File "C:\Users\hx\AppData\Local\Temp\pip-install-drqmfmaj\faiss-cpu\setup.py", line 50, in build_extensions
        self._remove_flag('-Wstrict-prototypes')
      File "C:\Users\hx\AppData\Local\Temp\pip-install-drqmfmaj\faiss-cpu\setup.py", line 58, in _remove_flag
        compiler = self.compiler.compiler
    AttributeError: 'MSVCCompiler' object has no attribute 'compiler'
    ----------------------------------------
ERROR: Command errored out with exit status 1: 'C:\ProgramData\Anaconda3\envs\blink37\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\hx\\AppData\\Local\\Temp\\pip-install-drqmfmaj\\faiss-cpu\\setup.py'"'"'; __file__='"'"'C:\\Users\\hx\\AppData\\Local\\Temp\\pip-install-drqmfmaj\\faiss-cpu\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\hx\AppData\Local\Temp\pip-record-qsj3c0ie\install-record.txt' --single-version-externally-managed --compile --install-headers 'C:\ProgramData\Anaconda3\envs\blink37\Include\faiss-cpu' Check the logs for full command output.

Segfault in macOS in the presence of other OpenMP dependent packages

Describe the bug
faiss may crash when there is another imported package that depends on libomp.

To Reproduce
Import a package that depends on OpenMP on macOS, then import faiss-cpu package. Use faiss as usual.

Expected behavior
Should not segfault.

Desktop (please complete the following information):

OS: macOS
Architecture: arm64
Python: any
Version: v1.7.3

Additional context
facebookresearch/faiss#2463
benfred/implicit#613 (comment)
dmlc/xgboost#7039 (comment)
https://bugs.llvm.org/show_bug.cgi?id=50579

ERROR: Could not build wheels for faiss-cpu, which is required to install pyproject.toml-based projects in FreeBSD

Describe the bug
In FreeBSD pip install of faiss-cpu failed building wheel for faiss-cpu

To Reproduce

pip install faiss-cpu

Collecting faiss-cpu
Using cached faiss-cpu-1.7.4.tar.gz (57 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: faiss-cpu
Building wheel for faiss-cpu (pyproject.toml) ... error
error: subprocess-exited-with-error

× Building wheel for faiss-cpu (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [8 lines of output]
running bdist_wheel
running build
running build_py
running build_ext
building 'faiss._swigfaiss' extension
swigging faiss/faiss/python/swigfaiss.i to faiss/faiss/python/swigfaiss_wrap.cpp
swig -python -c++ -Doverride= -I/usr/local/include -Ifaiss -doxygen -module swigfaiss -o faiss/faiss/python/swigfaiss_wrap.cpp faiss/faiss/python/swigfaiss.i
error: command 'swig' failed: No such file or directory
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for faiss-cpu
Failed to build faiss-cpu
ERROR: Could not build wheels for faiss-cpu, which is required to install pyproject.toml-based projects

Desktop:

OS: FreeBSD 13.1-RELEASE-p3 GENERIC
Architecture: amd64
Python: 3.9.16
Version: 1.7.4

Reduce wheel package size for faiss-gpu CUDA 11.0 build

The CUDA 11.0 build in #56 bloats the wheel package size from 85.5 MB to 216.5 MB. Needs to investigate file size reduction.

Wheels for Python 3.8

Would you publish wheels for python3.8 ?

ERROR: Could not build wheels for faiss-cpu, which is required to install pyproject.toml-based projects in linux Alpine

Describe the bug
Can't build wheels.

To Reproduce

pip install faiss-cpu

Expected behavior

#11 827.7       swigging faiss/faiss/python/swigfaiss.i to faiss/faiss/python/swigfaiss_wrap.cpp
#11 827.7       swig -python -c++ -Doverride= -I/usr/local/include -Ifaiss -doxygen -DSWIGWORDSIZE64 -module swigfaiss -o faiss/faiss/python/swigfaiss_wrap.cpp faiss/faiss/python/swigfaiss.i
#11 827.7       faiss/faiss/python/swigfaiss.i:268: Error: Unable to find 'faiss/impl/platform_macros.h'
#11 827.7       faiss/faiss/python/swigfaiss.i:272: Error: Unable to find 'faiss/utils/ordered_key_value.h'
#11 827.7       faiss/faiss/python/swigfaiss.i:273: Error: Unable to find 'faiss/utils/Heap.h'
#11 827.7       faiss/faiss/python/swigfaiss.i:278: Error: Unable to find 'faiss/utils/AlignedTable.h'

...

#11 827.7       [end of output]
#11 827.7   
#11 827.7   note: This error originates from a subprocess, and is likely not a problem with pip.
#11 827.7   ERROR: Failed building wheel for faiss-cpu
#11 827.7 Successfully built numexpr numpy PyYAML
#11 827.7 Failed to build faiss-cpu
#11 827.7 ERROR: Could not build wheels for faiss-cpu, which is required to install pyproject.toml-based projects

Desktop (please complete the following information):

OS: Alpine
Architecture: x86_64
Python: 3.10.12
Version: v1.7.4
swig: 4.1.1
pip: 23.2.1(latest)

Failure to serialize in joblib

Describe the bug
Consistently getting the following error when installing the faiss-cpu==1.7.3 module on Mac OS (Silicon M2 Chip):
ModuleNotFoundError: No module named 'faiss.swigfaiss_avx2'

To Reproduce
pip install faiss-cpu==1.7.3

Expected behavior
When installing the package, it is expected that the package scripts should include a script called:

swigfaiss_avx2.py

But this is missing when trying to install the module on my mac and I think this is causing the issue

Desktop (please complete the following information):

OS: Mac
Architecture: arm64
Python: 3.9.16
Version: v1.7.3

Additional context
This happens when trying to "joblib.load()"

libomp/libiomp5 conflict macOS

Hello! I am currently getting the following abort in my Python code:

OMP: Error #15: Initializing libomp.a, but found libiomp5.dylib already initialized.
OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, 
since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP 
runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, 
unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow 
the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, 
please see http://openmp.llvm.org/
[1]    24366 abort      python train.py

I believe this is probably caused by a conflict between torch and faiss-cpu since my code relies on both (as well as numpy but numpy is used by faiss). I'm trying to reproduce the issue outside of my code so I can post it here. I tried to compile a version myself by copying the build parameters used in the repository, however, I was unable to get the same performance so I gave up.

I am using faiss-cpu version 1.7.0 on macOS 11.2.3

Support Selector for IDMap

Hi, could you please update the python pip package with the changes in PR #2848 ?
facebookresearch/faiss#2843
facebookresearch/faiss#2848
facebookresearch/faiss#2988

Installing faiss-cpu ,happens error

When I install happens error

ERROR: Command errored out with exit status 1:
   command: /home/sueoka/anaconda3/envs/nemo_v2/bin/python /home/sueoka/anaconda3/envs/nemo_v2/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmps_8f0m_q
       cwd: /tmp/pip-install-kfbf4zqn/faiss-cpu_35de81a7d3fc4ea899e7130f7a81f594
  Complete output (10 lines):
  running bdist_wheel
  running build
  running build_py
  running build_ext
  building 'faiss._swigfaiss' extension
  swigging faiss/faiss/python/swigfaiss.i to faiss/faiss/python/swigfaiss_wrap.cpp
  swig -python -c++ -Doverride= -I/usr/local/include -Ifaiss -doxygen -DSWIGWORDSIZE64 -o faiss/faiss/python/swigfaiss_wrap.cpp faiss/faiss/python/swigfaiss.i
  swig error : Unrecognized option -doxygen
  Use 'swig -help' for available options.
  error: command '/usr/bin/swig' failed with exit code 1
  ----------------------------------------
  ERROR: Failed building wheel for faiss-cpu
Failed to build faiss-cpu
ERROR: Could not build wheels for faiss-cpu which use PEP 517 and cannot be installed directly

my env

SWIG Version 3.0.12
Compiled with g++ [powerpc64le-unknown-linux-gnu]
Configured options: +pcre
gcc (Ubuntu/IBM 5.5.0-12ubuntu1) 5.5.0 20171010 [ibm/gcc-5-branch revision 248295]
Python 3.8.0

pip installs the previous release when tries to build from source

When I'm trying to install faiss-cpu with pip install --no-binary :all: faiss-cpu pip installs 1.7.2 instead of 1.7.3, the latest one. Forcing installing faiss-cpu==1.7.3 shows that 1.7.2 is the latest one and there is no 1.7.3 at all.

The problem live with publishing to PyPi, because there is nothing in Source Distributions block on the PyPi page of the library, see here: https://pypi.org/project/faiss-cpu/#files, that's why pip can't build from the source.

kyamagu / faiss-wheels Goto Github PK

faiss-wheels's People

Stargazers

Watchers

Forkers

faiss-wheels's Issues

File "C:\Users<user>\AppData\Local\Temp\pip-install-nppl0xcx\faiss-gpu_3375d4d88f02454c85440c4a6dcee4e7\setup.py", line 58, in _remove_flag compiler = self.compiler.compiler AttributeError: 'MSVCCompiler' object has no attribute 'compiler'

Problem

Resolution

Recommend Projects

Recommend Topics

Recommend Org

File "C:\Users<user>\AppData\Local\Temp\pip-install-nppl0xcx\faiss-gpu_3375d4d88f02454c85440c4a6dcee4e7\setup.py", line 58, in _remove_flag
compiler = self.compiler.compiler
AttributeError: 'MSVCCompiler' object has no attribute 'compiler'