Problem with quantize model about lorax

[tgaddair]

Glad you got it working! We'll be adding some better docs soon to make these parameters easier to find. Closing this issue for now.

[flozi00]

Do you measure before or after warmup?

With the start up the kv cache gets reserved, with quantited model there is more memory for cache, but the total used memory is the same.
You could limit the available memory by an argument

[tgaddair]

Hey @prd-tuong-nguyen, as @flozi00 said this is likely due to the warmup phase, where we allocate additional memory in advance for batching to avoid having to allocate it on the fly during inference.

For example, here's the memory usage reported by nvidia-smi when running with nf4 quantization using mistral-7b before warmup:

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.129.03             Driver Version: 535.129.03   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA A100-PCIE-40GB          On  | 00000000:11:00.0 Off |                    0 |
| N/A   27C    P0              54W / 250W |   5011MiB / 40960MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+

And here are the results after warmup:

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.129.03             Driver Version: 535.129.03   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA A100-PCIE-40GB          On  | 00000000:11:00.0 Off |                    0 |
| N/A   27C    P0              54W / 250W |  38911MiB / 40960MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+

As you can see, lorax will use as much memory as it can get away to maximize the batch size.

[prd-tuong-nguyen]

@flozi00 , @tgaddair Oh thanks for your fast reply, I see that the memory before warming up is lower then after that.
Can I reduce the additional memory for batching , in that way I can serve multi-instance in the same GPU.

[prd-tuong-nguyen]

Resolved by edit cuda_memory_fraction, thank you <3