To use ๐ค text-embeddings-inference on Habana Gaudi/Gaudi2, follow these steps:
- Pull the official Docker image with:
docker pull ghcr.io/huggingface/tei-gaudi:latest
Note
Alternatively, you can build the Docker image using Dockerfile-hpu
located in this folder with:
docker build -f Dockerfile-hpu -t tei_gaudi .
- Launch a local server instance on 1 Gaudi card:
model=BAAI/bge-large-en-v1.5 volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run docker run -p 8080:80 -v $volume:/data --runtime=habana -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_vader_single_copy_mechanism=none -e MAX_WARMUP_SEQUENCE_LENGTH=512 --cap-add=sys_nice --ipc=host ghcr.io/huggingface/tei-gaudi:latest --model-id $model --pooling cls
- You can then send a request:
curl 127.0.0.1:8080/embed \ -X POST \ -d '{"inputs":"What is Deep Learning?"}' \ -H 'Content-Type: application/json'
For more information and documentation about Text Embeddings Inference, checkout the README of the original repo.
Not all features of TEI are currently supported as this is still a work in progress.
The license to use TEI on Habana Gaudi is the one of TEI: https://github.com/huggingface/text-embeddings-inference/blob/main/LICENSE
Please reach out to [email protected] if you have any question.