How to do Indexing using from_index() on CPU only? about ragatouille

[bclavie]

Hey! To run on CPU, you don't need to actually set the n_gpu parameter yourself, it should default to 1 (and then ignore it) or 0... Have you tried this and ran into any issue?

[turjo-001]

Hey! To run on CPU, you don't need to actually set the n_gpu parameter yourself, it should default to 1 (and then ignore it) or 0... Have you tried this and ran into any issue?

Hi @bclavie. I've tried using -1, 0, 1 but nothing seems to work. It keeps loading the model and index to my GPU RAM.

It should be mentioned that till now I had used GPU to run indexing flawlessly. To complete my project I now need to run an LLM model which leaves barely any VRAM on my GPU to run the retriever model and the Index.

I have my conda ragatouille environment setup to utilize the GPU packages. That's why i was wondering if there was any parameter i could change to force the retriever and index to run on using my CPU.

Thanks.

[bclavie]

Oh my bad, I was under the impression that you wee on a GPU-less machine!

I think the easiest way to get the retriever to not use the GPU at all would be to hide it from the script. Are you running it in a way where it's easy for you to set CUDA_VISIBLE_DEVICES="" and/or tell torch to ignore it? There is room for change here but colbert-ai still has a lot of hardcoded .cuda() calls :/

[turjo-001]

Thanks. I'll try setting CUDA_VISIBLE_DEVICES="" and look into the torch thing. Will update here if i can do something.