Where should we store config.json? about sagemaker-huggingface-inference-toolkit

[philschmid]

Thats because your structure is not correct. It need to be

model.tar.gz
    code/
        inference.py
    config.json
    tf_model.h5
    special_tokens_map.json
    tokenizer.json
    tokenizer_config.json
    vocab.txt

You can take a look here

[la-cruche]

ok nice! we should write that in the doc no? to have all model artifacts directly in the archive without folder hierarchy

[philschmid]

It is actually documented here.
But I agree with you since you already into trouble we should make it more clear.

The Hugging Face Inference Toolkit allows user to override the default methods of the HuggingFaceHandlerService. Therefor the need to create a named code/ with a inference.py file in it. For example:

model.tar.gz/
|- pytroch_model.bin
|- ....
|- code/
|- inference.py
|- requirements.txt
In this example, pytroch_model.bin is the model file saved from training, inference.py is the custom inference module, and requirements.txt is a requirements file to add additional dependencies. The custom module can override the following methods:

[la-cruche]

yes, what I meant is that the doc didn't say "don't create folders, have every file in the archive without any folder" right?

[philschmid]

added this to the documentation.

Model artificat structure model.tar.gz

The model.tar.gz contains all required files to run your model including your model file either pytorch_model.bin, tf_model.h5, tokenizer.json , tokenizer_config.json etc. All model artifacts need to be directly in the archive without folder hierarchy.

examples for PyTorch:

model.tar.gz/
|- pytroch_model.bin
|- vocab.txt
|- tokenizer_config.json
|- config.json
|- special_tokens_map.json

[la-cruche]

nice! but we agree that with custom model_fn the archive structure can be anything right? people would be responsible to write code that can parse it

[philschmid]

Yes