P40 with USE_FLASH_ATTENTION=False about text-generation-inference

[LysandreJik]

Hey @ltm920716, thanks for the report!

I believe this was fixed by the following PR #1986 from @Narsil.

You'll now get this instead:

WARN text_generation_launcher: Could not import Flash Attention enabled models: `USE_FLASH_ATTENTION` is false.

I believe this hasn't been released in a version yet, but you can use the docker image that was released with the commit of the PR above:

ghcr.io/huggingface/text-generation-inference:sha-06edde9

The resulting command would be:

sudo nerdctl run -it --gpus '"device=1"' -e USE_FLASH_ATTENTION=False -p 8080:80 -v /data/models:/data ghcr.io/huggingface/text-generation-inference:sha-06edde9 --model-id /data/Meta-Llama-3-8B-Instruct

[ltm920716]

thanks a lot, I will have a try!