InternalServerException at runtime about sagemaker-huggingface-inference-toolkit

[philschmid]

@krokoko LLama got added with Transformers 4.28 which is not yet available as DLC but should come in the coming weeks. In the meantime you could create a custom inference script and a requirements.txt to add the latest versions. See here for an example: https://www.philschmid.de/custom-inference-huggingface-sagemaker

Additionally i should mentioned that the default inference container is not doing any form of model parallelism so if the model is not fitting on single GPU you also need a custom script to do this. Here is an example for how we did this for flan-ul2 https://www.philschmid.de/deploy-flan-ul2-sagemaker

[krokoko]

@philschmid thanks for the quick reply ! Sorry I wasn't clear, I extended the latest version of the DLC container to install Transformers v4.28.1 and v4.28.0, then uploaded the new containers to my private ECR repo. Something like:

FROM 763104351884.dkr.ecr.us-east-1.amazonaws.com/huggingface-pytorch-inference:1.13.1-transformers4.26.0-gpu-py39-cu117-ubuntu20.04
RUN pip install --upgrade 'transformers==4.28.1'

That makes things easier since I don't need to create a custom inference script. I use the custom container when instantiating the model (first parameter):

huggingface_model_snoozy = HuggingFaceModel(
        image_uri=ecr_image,
	transformers_version='4.28.0',
	pytorch_version='1.13.1',
	py_version='py39',
	env=hub_snoozy,
	role=role, 
)

Regarding the default inference container, you mean that in that case the custom inference.py script is needed is that correct ?

[philschmid]

Most likely yes. But since you update the image there also might be an issue with the model weights. Normally you should be able to load it with the llama class.