openchat_v3.2_super deployment to SageMaker doesn't work about openchat

[imoneoi]

This is because of some optimizations for padding-free training, the missing weights can be safely ignored and just let HF initialize them. You can set the ignore missing weights flag.

[ErdincWand]

You can set the ignore missing weights flag.

Thanks for your reply. I am really new to this world. Can you explicitly help what parameter I should set?

[ErdincWand]

Just for reference, below code worked in Sagemaker. Thanks for your help!

import json
import sagemaker
import boto3
from sagemaker.huggingface import HuggingFaceModel, get_huggingface_llm_image_uri

try:
	role = sagemaker.get_execution_role()
except ValueError:
	iam = boto3.client('iam')
	role = iam.get_role(RoleName='role-name')['Role']['Arn']

# Hub Model configuration. https://huggingface.co/models
hub = {
	'HF_MODEL_ID':'openchat/openchat_v3.2_super',
	'SM_NUM_GPUS': json.dumps(4)
}



# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
        image_uri="763104351884.dkr.ecr.us-east-1.amazonaws.com/huggingface-pytorch-tgi-inference:2.0.1-tgi1.0.3-gpu-py39-cu118-ubuntu20.04",
	env=hub,
	role=role, 
    
)

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
	initial_instance_count=1,
	instance_type="ml.g5.12xlarge",
    endpoint_name="openchat-v3-2-super",
	container_startup_health_check_timeout=600,
  )
  
# send request
predictor.predict({
	"inputs": "My name is Julien and I like to",
})