Slow (?) indexing on Apple m1 about ragatouille

[bclavie]

(Copy/pasting this message in a few related issues)

Hey guys!

Thanks a lot for bearing with me as I juggle everything and trying to diagnose this. It’s complicated to fix with relatively little time to dedicate to it, as it seems like the dependencies causing issues aren’t the same for everyone, with no clear platform pattern as of yet. Overall, the issues center around the usual suspects of faiss and CUDA.

While because of this I can’t fix the issue with PLAID optimised indices just yet, I’m also noticing that most of the bug reports here are about relatively small collections (100s-to-low-1000s). To lower the barrier to entry as much as possible, #137 is introducing a second index format, which doesn’t actually build an index, but performs an exact search over all documents (as a stepping stone towards #110, which would use an HNSW index to be an in-between compromise between PLAID optimisation and exact search).
This approach doesn’t scale, but offers the best possible search accuracy & is still performed in a few hundred milliseconds at most for small collections. Ideally, it’ll also open up the way to shipping lower-dependency versions (#136)

The PR above (#137) is still a work in progress, as it needs CRUD support, tests, documentation, better precision routing (fp32/bfloat16) etc… (and potentially searching only subset of document ids).
However, it’s working in a rough state for me locally. If you’d like to give it a try (with the caveat that it might very well break!), please feel free to install the library directly from the feat/full_vectors_indexing branch and adding the following argument to your index() call:

index(…
index_type=“FULL_VECTORS”,
)

Any feedback is appreciated, as always, and thanks again!

[bclavie]

Hey!

No, this very much isn't expected. Tried it out on M1 and it took <30s to index the Miyazaki example, which seems to be what you're doing here!

Are you running the standard notebook example on the latest version? I can't reproduce the issue on weaker hardware than yours via VSCode Jupyter (M1 Air 8GB, Python 3.11, indexing time ~27s)🤔

[gojira]

Yes - 01-basic_indexing_and_search.ipynb.

I am still seeing same after doing the following:

1. clone from scratch again (I cloned yesterday but already 5 commits behind) + mamba env from scratch again with Python 3.11
2. clone from scratch again + new mamba env with Python 3.9 because repo notebook output showed 3.9

When I interrupt it shows multiprocessing so is this the same issue as if __name__ == "__main__"?

{
	"name": "KeyboardInterrupt",
	"message": "",
	"stack": "---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
Cell In[4], line 1
----> 1 RAG.index(collection=[full_document], index_name=\"Miyazaki\", max_document_length=180, split_documents=True)

File ~/mambaforge/envs/raga/lib/python3.9/site-packages/ragatouille/RAGPretrainedModel.py:125, in RAGPretrainedModel.index(self, collection, index_name, overwrite_index, max_document_length, split_documents, document_splitter_fn, preprocessing_fn)
    123 if overwrite_index:
    124     overwrite = True
--> 125 return self.model.index(
    126     collection,
    127     index_name,
    128     max_document_length=max_document_length,
    129     overwrite=overwrite,
    130 )

File ~/mambaforge/envs/raga/lib/python3.9/site-packages/ragatouille/models/colbert.py:203, in ColBERT.index(self, collection, index_name, max_document_length, overwrite)
    197 self.config.avoid_fork_if_possible = True
    198 self.indexer = Indexer(
    199     checkpoint=self.checkpoint,
    200     config=self.config,
    201     verbose=self.verbose,
    202 )
--> 203 self.indexer.index(
    204     name=self.index_name, collection=collection, overwrite=overwrite
    205 )
    207 index_path = str(
    208     Path(self.run_config.root)
    209     / Path(self.run_config.experiment)
    210     / \"indexes\"
    211     / self.index_name
    212 )
    213 self.config.root = str(
    214     Path(self.run_config.root) / Path(self.run_config.experiment) / \"indexes\"
    215 )

File ~/mambaforge/envs/raga/lib/python3.9/site-packages/colbert/indexer.py:78, in Indexer.index(self, name, collection, overwrite)
     75     self.erase()
     77 if index_does_not_exist or overwrite != 'reuse':
---> 78     self.__launch(collection)
     80 return self.index_path

File ~/mambaforge/envs/raga/lib/python3.9/site-packages/colbert/indexer.py:89, in Indexer.__launch(self, collection)
     87 # Encodes collection into index using the CollectionIndexer class
     88 launcher = Launcher(encode)
---> 89 launcher.launch(self.config, collection, shared_lists, shared_queues, self.verbose)

File ~/mambaforge/envs/raga/lib/python3.9/site-packages/colbert/infra/launcher.py:79, in Launcher.launch(self, custom_config, *args)
     75 print_memory_stats('MAIN')
     77 # TODO: If the processes crash upon join, raise an exception and don't block on .get() below!
---> 79 return_values = sorted([return_value_queue.get() for _ in all_procs])
     80 return_values = [val for rank, val in return_values]
     82 if not self.return_all:

File ~/mambaforge/envs/raga/lib/python3.9/site-packages/colbert/infra/launcher.py:79, in <listcomp>(.0)
     75 print_memory_stats('MAIN')
     77 # TODO: If the processes crash upon join, raise an exception and don't block on .get() below!
---> 79 return_values = sorted([return_value_queue.get() for _ in all_procs])
     80 return_values = [val for rank, val in return_values]
     82 if not self.return_all:

File ~/mambaforge/envs/raga/lib/python3.9/multiprocessing/queues.py:103, in Queue.get(self, block, timeout)
    101 if block and timeout is None:
    102     with self._rlock:
--> 103         res = self._recv_bytes()
    104     self._sem.release()
    105 else:

File ~/mambaforge/envs/raga/lib/python3.9/multiprocessing/connection.py:216, in _ConnectionBase.recv_bytes(self, maxlength)
    214 if maxlength is not None and maxlength < 0:
    215     raise ValueError(\"negative maxlength\")
--> 216 buf = self._recv_bytes(maxlength)
    217 if buf is None:
    218     self._bad_message_length()

File ~/mambaforge/envs/raga/lib/python3.9/multiprocessing/connection.py:414, in Connection._recv_bytes(self, maxsize)
    413 def _recv_bytes(self, maxsize=None):
--> 414     buf = self._recv(4)
    415     size, = struct.unpack(\"!i\", buf.getvalue())
    416     if size == -1:

File ~/mambaforge/envs/raga/lib/python3.9/multiprocessing/connection.py:379, in Connection._recv(self, size, read)
    377 remaining = size
    378 while remaining > 0:
--> 379     chunk = read(handle, remaining)
    380     n = len(chunk)
    381     if n == 0:

KeyboardInterrupt: "
}

[bclavie]

It's very odd that it'd hang on a Mac, I think we've had a few hundred people do it locally already and it never occurred 🤔

Although it does reveal another issue -- @Anmol6 I've pinged you, we need a more reliable way of bypassing forking, currently RAGatouille has no easy way of enforcing it in every case.

@gojira I've tried a new env on 3.9 as well, and everything runs smoothly... I think once we fixed the flawed MP-bypass implementation it'll also solve your problem!

[gojira]

It's been a little while since I've had things break in AI this way - I guess it is truly cutting edge ;)!

[bclavie]

😄

Multiprocessing should now be properly disabled when fewer than 2 GPUs are detected -- let me know if the update fixes your issue!

[gojira]

I installed the following and it still hangs. Now VS Code has problems interrupting the kernel - so I can't get a trace like I did last time

Downloading ragatouille-0.0.4b2-py3-none-any.whl (30 kB)
Downloading colbert_ai-0.2.17-py3-none-any.whl (113 kB)

[bclavie]

This is very strange, especially considering it seems to be dependant on something very specific, but utterly unclear what 🤔 ...

Would you mind posting your pip freeze? I'll mark this issue as Help Wanted and it'd be great if someone managed to figure out exactly what's causing it.

[gojira]

Yes - here you go - I deleted it before

Babel @ file:///home/conda/feedstock_root/build_artifacts/babel_1702422572539/work
Brotli @ file:///Users/runner/miniforge3/conda-bld/brotli-split_1695989934239/work
Deprecated==1.2.14
Flask==3.0.0
GitPython==3.1.41
Jinja2 @ file:///home/conda/feedstock_root/build_artifacts/jinja2_1704966972576/work
MarkupSafe @ file:///Users/runner/miniforge3/conda-bld/markupsafe_1695367493730/work
PySocks @ file:///home/conda/feedstock_root/build_artifacts/pysocks_1661604839144/work
PyYAML @ file:///Users/runner/miniforge3/conda-bld/pyyaml_1695373497981/work
Pygments @ file:///home/conda/feedstock_root/build_artifacts/pygments_1700607939962/work
QtPy @ file:///home/conda/feedstock_root/build_artifacts/qtpy_1698112029416/work
RAGatouille==0.0.4b2
SQLAlchemy==2.0.25
Send2Trash @ file:///Users/runner/miniforge3/conda-bld/send2trash_1682601407921/work
Werkzeug==3.0.1
aiohttp==3.9.1
aiosignal==1.3.1
annotated-types==0.6.0
anyio @ file:///home/conda/feedstock_root/build_artifacts/anyio_1702909220329/work
appnope @ file:///home/conda/feedstock_root/build_artifacts/appnope_1649077682618/work
argon2-cffi @ file:///home/conda/feedstock_root/build_artifacts/argon2-cffi_1692818318753/work
argon2-cffi-bindings @ file:///Users/runner/miniforge3/conda-bld/argon2-cffi-bindings_1695386652227/work
arrow @ file:///home/conda/feedstock_root/build_artifacts/arrow_1696128962909/work
asttokens @ file:///home/conda/feedstock_root/build_artifacts/asttokens_1698341106958/work
async-lru @ file:///home/conda/feedstock_root/build_artifacts/async-lru_1690563019058/work
async-timeout==4.0.3
attrs @ file:///home/conda/feedstock_root/build_artifacts/attrs_1704011227531/work
beautifulsoup4 @ file:///home/conda/feedstock_root/build_artifacts/beautifulsoup4_1680888073205/work
bitarray==2.9.2
bleach @ file:///home/conda/feedstock_root/build_artifacts/bleach_1696630167146/work
blinker==1.7.0
blis==0.7.11
cached-property @ file:///home/conda/feedstock_root/build_artifacts/cached_property_1615209429212/work
catalogue==2.0.10
certifi @ file:///home/conda/feedstock_root/build_artifacts/certifi_1700303426725/work/certifi
cffi @ file:///Users/runner/miniforge3/conda-bld/cffi_1696001750083/work
charset-normalizer @ file:///home/conda/feedstock_root/build_artifacts/charset-normalizer_1698833585322/work
click==8.1.7
cloudpathlib==0.16.0
colbert-ai==0.2.17
comm @ file:///home/conda/feedstock_root/build_artifacts/comm_1704278392174/work
confection==0.1.4
cymem==2.0.8
dataclasses-json==0.6.3
datasets==2.16.1
debugpy @ file:///Users/runner/miniforge3/conda-bld/debugpy_1695534371680/work
decorator @ file:///home/conda/feedstock_root/build_artifacts/decorator_1641555617451/work
defusedxml @ file:///home/conda/feedstock_root/build_artifacts/defusedxml_1615232257335/work
dill==0.3.7
distro==1.9.0
entrypoints @ file:///home/conda/feedstock_root/build_artifacts/entrypoints_1643888246732/work
exceptiongroup @ file:///home/conda/feedstock_root/build_artifacts/exceptiongroup_1704921103267/work
executing @ file:///home/conda/feedstock_root/build_artifacts/executing_1698579936712/work
faiss-cpu==1.7.4
fastjsonschema @ file:///home/conda/feedstock_root/build_artifacts/python-fastjsonschema_1703780968325/work/dist
filelock @ file:///home/conda/feedstock_root/build_artifacts/filelock_1698714947081/work
fqdn @ file:///home/conda/feedstock_root/build_artifacts/fqdn_1638810296540/work/dist
frozenlist==1.4.1
fsspec==2023.10.0
git-python==1.0.3
gitdb==4.0.11
gmpy2 @ file:///Users/runner/miniforge3/conda-bld/gmpy2_1666808750085/work
greenlet==3.0.3
h11==0.14.0
httpcore==1.0.2
httpx==0.26.0
huggingface-hub==0.20.2
idna @ file:///home/conda/feedstock_root/build_artifacts/idna_1701026962277/work
importlib-metadata @ file:///home/conda/feedstock_root/build_artifacts/importlib-metadata_1703269254275/work
importlib-resources @ file:///home/conda/feedstock_root/build_artifacts/importlib_resources_1699364556997/work
ipykernel @ file:///Users/runner/miniforge3/conda-bld/ipykernel_1703631823913/work
ipython @ file:///home/conda/feedstock_root/build_artifacts/ipython_1701831663892/work
ipywidgets @ file:///home/conda/feedstock_root/build_artifacts/ipywidgets_1694607144474/work
isoduration @ file:///home/conda/feedstock_root/build_artifacts/isoduration_1638811571363/work/dist
itsdangerous==2.1.2
jedi @ file:///home/conda/feedstock_root/build_artifacts/jedi_1696326070614/work
joblib==1.3.2
json5 @ file:///home/conda/feedstock_root/build_artifacts/json5_1688248289187/work
jsonpatch==1.33
jsonpointer @ file:///Users/runner/miniforge3/conda-bld/jsonpointer_1695397322962/work
jsonschema @ file:///home/conda/feedstock_root/build_artifacts/jsonschema-meta_1700159890288/work
jsonschema-specifications @ file:///tmp/tmpkv1z7p57/src
jupyter @ file:///home/conda/feedstock_root/build_artifacts/jupyter_1696255489086/work
jupyter-console @ file:///home/conda/feedstock_root/build_artifacts/jupyter_console_1678118109161/work
jupyter-events @ file:///home/conda/feedstock_root/build_artifacts/jupyter_events_1699285872613/work
jupyter-lsp @ file:///home/conda/feedstock_root/build_artifacts/jupyter-lsp-meta_1701091994466/work/jupyter-lsp
jupyter_client @ file:///home/conda/feedstock_root/build_artifacts/jupyter_client_1699283905679/work
jupyter_core @ file:///Users/runner/miniforge3/conda-bld/jupyter_core_1704727402891/work
jupyter_server @ file:///home/conda/feedstock_root/build_artifacts/jupyter_server_1704986851950/work
jupyter_server_terminals @ file:///home/conda/feedstock_root/build_artifacts/jupyter_server_terminals_1703611053195/work
jupyterlab @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_1703881392925/work
jupyterlab-widgets @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_widgets_1694598704522/work
jupyterlab_pygments @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_pygments_1700744013163/work
jupyterlab_server @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_server-split_1700310846957/work
langchain-community==0.0.12
langchain-core==0.1.10
langchain==0.1.0
langcodes==3.3.0
langsmith==0.0.80
llama-index==0.9.31
marshmallow==3.20.2
matplotlib-inline @ file:///home/conda/feedstock_root/build_artifacts/matplotlib-inline_1660814786464/work
mistune @ file:///home/conda/feedstock_root/build_artifacts/mistune_1698947099619/work
mpmath @ file:///home/conda/feedstock_root/build_artifacts/mpmath_1678228039184/work
multidict==6.0.4
multiprocess==0.70.15
murmurhash==1.0.10
mypy-extensions==1.0.0
nbclient @ file:///home/conda/feedstock_root/build_artifacts/nbclient_1684790896106/work
nbconvert @ file:///home/conda/feedstock_root/build_artifacts/nbconvert-meta_1704986998851/work
nbformat @ file:///home/conda/feedstock_root/build_artifacts/nbformat_1690814868471/work
nest-asyncio @ file:///home/conda/feedstock_root/build_artifacts/nest-asyncio_1697083700168/work
networkx @ file:///home/conda/feedstock_root/build_artifacts/networkx_1698504735452/work
ninja==1.11.1.1
nltk==3.8.1
notebook @ file:///home/conda/feedstock_root/build_artifacts/notebook_1697550696415/work
notebook_shim @ file:///home/conda/feedstock_root/build_artifacts/notebook-shim_1682360583588/work
numpy @ file:///Users/runner/miniforge3/conda-bld/numpy_1704280477691/work/dist/numpy-1.26.3-cp39-cp39-macosx_11_0_arm64.whl#sha256=b4904f3177fc1d83f3a54b101f5f80740cd57b4e131aaed5716bf3488e4de663
onnx==1.15.0
openai==1.7.2
overrides @ file:///home/conda/feedstock_root/build_artifacts/overrides_1691338815398/work
packaging @ file:///home/conda/feedstock_root/build_artifacts/packaging_1696202382185/work
pandas==2.1.4
pandocfilters @ file:///home/conda/feedstock_root/build_artifacts/pandocfilters_1631603243851/work
parso @ file:///home/conda/feedstock_root/build_artifacts/parso_1638334955874/work
pexpect @ file:///home/conda/feedstock_root/build_artifacts/pexpect_1667297516076/work
pickleshare @ file:///home/conda/feedstock_root/build_artifacts/pickleshare_1602536217715/work
pillow @ file:///Users/runner/miniforge3/conda-bld/pillow_1704252271887/work
pkgutil_resolve_name @ file:///home/conda/feedstock_root/build_artifacts/pkgutil-resolve-name_1694617248815/work
platformdirs @ file:///home/conda/feedstock_root/build_artifacts/platformdirs_1701708255999/work
preshed==3.0.9
prometheus-client @ file:///home/conda/feedstock_root/build_artifacts/prometheus_client_1700579315247/work
prompt-toolkit @ file:///home/conda/feedstock_root/build_artifacts/prompt-toolkit_1702399386289/work
protobuf==4.25.2
psutil @ file:///Users/runner/miniforge3/conda-bld/psutil_1702833188708/work
ptyprocess @ file:///home/conda/feedstock_root/build_artifacts/ptyprocess_1609419310487/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl
pure-eval @ file:///home/conda/feedstock_root/build_artifacts/pure_eval_1642875951954/work
pyarrow-hotfix==0.6
pyarrow==14.0.2
pycparser @ file:///home/conda/feedstock_root/build_artifacts/pycparser_1636257122734/work
pydantic==2.5.3
pydantic_core==2.14.6
pyobjc-core @ file:///Users/runner/miniforge3/conda-bld/pyobjc-core_1702119468109/work
pyobjc-framework-Cocoa @ file:///Users/runner/miniforge3/conda-bld/pyobjc-framework-cocoa_1702125476861/work
python-dateutil @ file:///home/conda/feedstock_root/build_artifacts/python-dateutil_1626286286081/work
python-dotenv==1.0.0
python-json-logger @ file:///home/conda/feedstock_root/build_artifacts/python-json-logger_1677079630776/work
pytz @ file:///home/conda/feedstock_root/build_artifacts/pytz_1693930252784/work
pyzmq @ file:///Users/runner/miniforge3/conda-bld/pyzmq_1701783274137/work
qtconsole @ file:///home/conda/feedstock_root/build_artifacts/qtconsole-base_1700168901209/work
referencing @ file:///home/conda/feedstock_root/build_artifacts/referencing_1704489226496/work
regex==2023.12.25
requests @ file:///home/conda/feedstock_root/build_artifacts/requests_1684774241324/work
rfc3339-validator @ file:///home/conda/feedstock_root/build_artifacts/rfc3339-validator_1638811747357/work
rfc3986-validator @ file:///home/conda/feedstock_root/build_artifacts/rfc3986-validator_1598024191506/work
rpds-py @ file:///Users/runner/miniforge3/conda-bld/rpds-py_1705159905602/work
ruff==0.1.13
safetensors==0.4.1
scikit-learn==1.3.2
scipy==1.11.4
sentence-transformers==2.2.2
sentencepiece==0.1.99
six @ file:///home/conda/feedstock_root/build_artifacts/six_1620240208055/work
smart-open==6.4.0
smmap==5.0.1
sniffio @ file:///home/conda/feedstock_root/build_artifacts/sniffio_1662051266223/work
soupsieve @ file:///home/conda/feedstock_root/build_artifacts/soupsieve_1693929250441/work
spacy-legacy==3.0.12
spacy-loggers==1.0.5
spacy==3.7.2
srsly==2.4.8
stack-data @ file:///home/conda/feedstock_root/build_artifacts/stack_data_1669632077133/work
sympy @ file:///home/conda/feedstock_root/build_artifacts/sympy_1684180540116/work
tenacity==8.2.3
terminado @ file:///Users/runner/miniforge3/conda-bld/terminado_1699810180257/work
thinc==8.2.2
threadpoolctl==3.2.0
tiktoken==0.5.2
tinycss2 @ file:///home/conda/feedstock_root/build_artifacts/tinycss2_1666100256010/work
tokenizers==0.15.0
tomli @ file:///home/conda/feedstock_root/build_artifacts/tomli_1644342247877/work
torch==2.1.2
torchaudio==2.1.2
torchvision==0.16.2
tornado @ file:///Users/runner/miniforge3/conda-bld/tornado_1695373661905/work
tqdm==4.66.1
traitlets @ file:///home/conda/feedstock_root/build_artifacts/traitlets_1704212992681/work
transformers==4.36.2
typer==0.9.0
types-python-dateutil @ file:///home/conda/feedstock_root/build_artifacts/types-python-dateutil_1704512562698/work
typing-inspect==0.9.0
typing-utils @ file:///home/conda/feedstock_root/build_artifacts/typing_utils_1622899189314/work
typing_extensions @ file:///home/conda/feedstock_root/build_artifacts/typing_extensions_1702176139754/work
tzdata==2023.4
ujson==5.9.0
uri-template @ file:///home/conda/feedstock_root/build_artifacts/uri-template_1688655812972/work/dist
urllib3 @ file:///home/conda/feedstock_root/build_artifacts/urllib3_1699933488691/work
voyager==2.0.2
wasabi==1.1.2
wcwidth @ file:///home/conda/feedstock_root/build_artifacts/wcwidth_1704731205417/work
weasel==0.3.4
webcolors @ file:///home/conda/feedstock_root/build_artifacts/webcolors_1679900785843/work
webencodings @ file:///home/conda/feedstock_root/build_artifacts/webencodings_1694681268211/work
websocket-client @ file:///home/conda/feedstock_root/build_artifacts/websocket-client_1701630677416/work
widgetsnbextension @ file:///home/conda/feedstock_root/build_artifacts/widgetsnbextension_1694598693908/work
wrapt==1.16.0
xxhash==3.4.1
yarl==1.9.4
zipp @ file:///home/conda/feedstock_root/build_artifacts/zipp_1695255097490/work

[Anmol6]

@gojira can you provide the exact code you're running via a script file/gist + how you're creating your environment?

[ten2mushi]

Hi guys,
first and foremost, thanks a bunch for this lib !

I think I have the same issue (M3 pro 64gb)

Was running python3.10, then tried with 3.9 with no change in the output (didn't wait for the program to finish but let it run > 15 min a couple of times)

install
python3.9 -m venv ragvenv
pip install ragatouille
(then running the Miyazaki example (01))

pip freeze output:
aiohttp==3.9.1
aiosignal==1.3.1
annotated-types==0.6.0
anyio==4.2.0
async-timeout==4.0.3
attrs==23.2.0
bitarray==2.9.2
blinker==1.7.0
catalogue==2.0.10
certifi==2024.2.2
charset-normalizer==3.3.2
click==8.1.7
colbert-ai==0.2.19
dataclasses-json==0.6.4
datasets==2.17.0
Deprecated==1.2.14
dill==0.3.8
dirtyjson==1.0.8
distro==1.9.0
exceptiongroup==1.2.0
faiss-cpu==1.7.4
filelock==3.13.1
Flask==3.0.2
frozenlist==1.4.1
fsspec==2023.10.0
git-python==1.0.3
gitdb==4.0.11
GitPython==3.1.41
greenlet==3.0.3
h11==0.14.0
httpcore==1.0.2
httpx==0.26.0
huggingface-hub==0.20.3
idna==3.6
importlib-metadata==7.0.1
itsdangerous==2.1.2
Jinja2==3.1.3
joblib==1.3.2
jsonpatch==1.33
jsonpointer==2.4
langchain==0.1.7
langchain-community==0.0.20
langchain-core==0.1.23
langsmith==0.0.87
llama-index==0.9.48
MarkupSafe==2.1.5
marshmallow==3.20.2
mpmath==1.3.0
multidict==6.0.5
multiprocess==0.70.16
mypy-extensions==1.0.0
nest-asyncio==1.6.0
networkx==3.2.1
ninja==1.11.1.1
nltk==3.8.1
numpy==1.26.4
onnx==1.15.0
openai==1.12.0
packaging==23.2
pandas==2.2.0
pillow==10.2.0
protobuf==4.25.2
pyarrow==15.0.0
pyarrow-hotfix==0.6
pydantic==2.6.1
pydantic_core==2.16.2
python-dateutil==2.8.2
python-dotenv==1.0.1
pytz==2024.1
PyYAML==6.0.1
RAGatouille==0.0.7.post2
regex==2023.12.25
requests==2.31.0
ruff==0.1.15
safetensors==0.4.2
scikit-learn==1.4.0
scipy==1.12.0
sentence-transformers==2.3.1
sentencepiece==0.1.99
six==1.16.0
smmap==5.0.1
sniffio==1.3.0
SQLAlchemy==2.0.27
srsly==2.4.8
sympy==1.12
tenacity==8.2.3
threadpoolctl==3.2.0
tiktoken==0.6.0
tokenizers==0.15.2
torch==2.2.0
tqdm==4.66.2
transformers==4.37.2
typing-inspect==0.9.0
typing_extensions==4.9.0
tzdata==2024.1
ujson==5.9.0
urllib3==2.2.0
voyager==2.0.2
Werkzeug==3.0.1
wrapt==1.16.0
xxhash==3.4.1
yarl==1.9.4
zipp==3.17.0

Let me know if I can provide more information !

[ten2mushi]

Hey Benjamin,

Thanks a bunch for your time! Just tried it and it went smoothly.

Installed the feat/full_vectors_indexing branch then tried with and without 'index_type=“FULL_VECTORS”,' both runs where a success!

[bclavie]

Hi @gojira. This should FINALLY be fixed by the new experimental default indexing in 0.0.8, which skips using faiss (does K-means in pure pytorch) as long as you're indexing fewer than ~100k documents!