Model memory usage / quantization about refact

[mitya52]

@coder543 hi!

We're using auto-gptq backend for most of the models (except of Refact, CONTRASTcode and codellama). They are 4bit quantized and should work with you setup (not sure about 15b). Required memory exceeds is just a warning and may be confusing. It only tells that you can get OOM with large file/chat context.

CodeLLama is 8bit quantized dynamically with bitsandbytes. I think we'll move models from auto-gptq to bitsandbytes or ggml backend and add quiantization option.

Refact model shouldn't use too much memory, your estimation is close to our. I should admit it as bug.

Thanks for your report!

[olegklimov]

We have sharding, should be solved! (not yet in docker today)