InternalServerException while deploying HuggingFace model on SageMaker about sagemaker-huggingface-inference-toolkit

[philschmid]

Hello @KartikKannapur,

Thank you for opening the issue. The snippet ist current not up to date regarding the versions. Could you try with transformers version 4.26?

[KartikKannapur]

Hey @philschmid

Thank you for looking into this.

I updated the transformers version to 4.26
The SageMaker SDK version is 2.132.0

and I received the following error message

ValueError: Unsupported huggingface version: 4.26.0. You may need to upgrade your SDK version (pip install -U sagemaker) for newer huggingface versions. Supported huggingface version(s): 4.6.1, 4.10.2, 4.11.0, 4.12.3, 4.17.0, 4.6, 4.10, 4.11, 4.12, 4.17.

So, I updated the SageMaker SDK to version 2.140.1
and when I ran the predictor.predict for model inference, I ran into the same error as before.

Any suggestions?

[philschmid]

Could you share your code how you deployed? i will try to reproduce it then.

[KartikKannapur]

from sagemaker.huggingface import HuggingFaceModel
import sagemaker

role = sagemaker.get_execution_role()
# Hub Model configuration. https://huggingface.co/models
hub = {
	'HF_MODEL_ID':'allenai/cosmo-xl',
	'HF_TASK':'conversational'
}

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
	transformers_version='4.26.0',
	pytorch_version='1.13.1',
	py_version='py39',
	env=hub,
	role=role, 
)

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
	initial_instance_count=1, # number of instances
	instance_type='ml.m5.xlarge' # ec2 instance type
)

predictor.predict({
	'inputs': {
		"past_user_inputs": ["Which movie is the best ?"],
		"generated_responses": ["It's Die Hard for sure."],
		"text": "Can you explain why ?"
	}
})

Thanks @philschmid
Appreciate your support on this.

[philschmid]

Ah i see. the task conversation is not correct here. Not sure why they added it manually. Can you switch the task to text2text-generation? then it should work.

Also from reading the model card: https://huggingface.co/allenai/cosmo-xl#how-to-use it might make sense if you create a custom infernece.py to get the "conversational" flow in to the generation method.

[KartikKannapur]

Understood. Thanks @philschmid

I'm going to go the custom inference.py route.