Text-Generation-Inference Docker container not working with quantize parameter about runpod-python

[chris-aeviator]

I was unable to reproduce your issue using SECURE cloud and Llama 2 - I can get TGI (lastest) to work with quantize=bitsandbytes

   6  docker_args = "--model-id meta-llama/Llama-2-7b-hf --num-shard 1 --quantize bitsandbytes"
   1  pod = runpod.create_pod(
   2      name="Llama-2-7b-tgi",
   3      image_name="ghcr.io/huggingface/text-generation-inference",
   4      gpu_type_id="NVIDIA RTX A4000",
   5      cloud_type="SECURE",
   6      docker_args=docker_args, # SEE BELOW
   7      gpu_count=1,
   8      volume_in_gb=30,
   9      container_disk_in_gb=5,
  10      ports="80/http",
  11      volume_mount_path="/data",
  12      env={
  13          "HUGGING_FACE_HUB_TOKEN": HUGGING_FACE_HUB_TOKEN
  14      }
  15  )

❯ curl https://xxxxxxxxxxx-80.proxy.runpod.net/generate \
    -X POST \
    -d '{"inputs":"What is Deep Learning?","parameters":{"max_new_tokens":200}}' \
    -H 'Content-Type: application/json'
{"generated_text":"\nWhat is Deep Learning? Deep learning is a subset of machine learning that is based on artificial neural networks. Neural networks are algorithms that are inspired by the structure and function of the human brain. Deep learning is a subset of machine learning that is based on artificial neural networks. Neural networks are algorithms that are inspired by the structure and function of the human brain.\nDeep learning is a subset of machine learning that is based on artificial neural networks. Neural networks are algorithms that are inspired by the structure and function of the human brain. Deep learning is a subset of machine learning that is based on artificial neural networks. Neural networks are algorithms that are inspired by the structure and function of the human brain.\nDeep learning is a subset of machine learning that is based on artificial neural networks. Neural networks are algorithms that are inspired by the structure and function of the human brain. Deep learning is a subset of machine learning that is based on artificial neural networks. Neural networks are"}%

[justinmerrell]

Thank you for the input @chris-aeviator

@solarslurpi I am not sure this is the best location for this issue unless you think there is a fix within this repo that would resolve things for you.

[justinmerrell]

Closing for now, not related to this repo.