docker-compose throws `flash attention is not installed` error about text-generation-inference

[retzero]

@LysandreJik
Thank you for the feedback.
This works for me.

+    environment:
+      CUDA_VISIBLE_DEVICES: "0,1"

//

-            device_ids: ['0', '1']
+            count: all

I'm closing this ticket.

[LysandreJik]

Hey! I've tried reproducing it locally, but unfortunately, I can't manage to do so. I don't have you model ID, but I see it seems to fail on the mistral import, so I designed the following docker compose file with mistral-community/Mistral-7B-v0.2:

version: '3.7'
services:
  inference-chat:
    image: ghcr.io/huggingface/text-generation-inference:2.0
    ports:
      - 8080:80
    volumes:
      - /home/ubuntu/data:/data
    command:
      - "--model-id"
      - "mistral-community/Mistral-7B-v0.2"
    deploy:
      resources:
        reservations:
          devices:
          - driver: nvidia
            device_ids: ['0']
            capabilities: [gpu]
networks:
  default:
    driver: bridge

I don't have any difference between using raw docker and docker compose, and it doesn't fail at the flash attention step. I'm using an A10G GPU which is the same generation as your A100 GPU.

Can you maybe try with the checkpoint above to see if it fails for you, to see if we identify whether the problem comes from a specific architecture or not?

[LysandreJik]

Awesome 🙌