LoRA Adapter from local model are leading to error about text-generation-inference

[Narsil]

Your example doesn't showcase the issue since you're passing the HFValidator, try using /data/test/tmp/xx/ or something.

[Narsil]

Issue is in Peft I believe ?

[philschmid]

It works when using peft with version 0.10.0.

1. download model

huggingface-cli download alignment-handbook/zephyr-7b-sft-qlora --exclude "*.bin" "*.pth" "*.gguf" --local-dir ./tmp
rm tmp/config.json

2. load peft model

from peft import AutoPeftModelForCausalLM

m = AutoPeftModelForCausalLM.from_pretrained("./tmp")

correctly works

[philschmid]

Thanks this leads to the issue.

Steps:

1. download model

huggingface-cli download alignment-handbook/zephyr-7b-sft-qlora --exclude "*.bin" "*.pth" "*.gguf" --local-dir ./tmp
rm tmp/config.json

2. run pytorch container and mount under /opt/ml/model

docker run --gpus all -it --rm \
-v $(pwd)/tmp/:/opt/ml/model \
-e HUGGING_FACE_HUB_TOKEN=$(cat ~/.cache/huggingface/token) \
-e HF_TOKEN=$(cat ~/.cache/huggingface/token) \
 --entrypoint /bin/bash nvcr.io/nvidia/pytorch:24.01-py3

3. install peft & run python

pip3 install peft && python3

4. run peft

from peft import AutoPeftModelForCausalLM

m = AutoPeftModelForCausalLM.from_pretrained("/opt/ml/model")

Error

>>> m = AutoPeftModelForCausalLM.from_pretrained("/opt/ml/model")
config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 571/571 [00:00<00:00, 9.28MB/s]
model.safetensors.index.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 25.1k/25.1k [00:00<00:00, 19.0MB/s]
model-00001-of-00002.safetensors: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████| 9.94G/9.94G [00:18<00:00, 549MB/s]
model-00002-of-00002.safetensors: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4.54G/4.54G [00:08<00:00, 553MB/s]
Downloading shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:26<00:00, 13.22s/it]
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:01<00:00,  1.33it/s]
generation_config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 116/116 [00:00<00:00, 1.14MB/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.10/dist-packages/peft/auto.py", line 128, in from_pretrained
    return cls._target_peft_class.from_pretrained(
  File "/usr/local/lib/python3.10/dist-packages/peft/peft_model.py", line 430, in from_pretrained
    model.load_adapter(model_id, adapter_name, is_trainable=is_trainable, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/peft/peft_model.py", line 984, in load_adapter
    adapters_weights = load_peft_weights(model_id, device=torch_device, **hf_hub_download_kwargs)
  File "/usr/local/lib/python3.10/dist-packages/peft/utils/save_and_load.py", line 415, in load_peft_weights
    has_remote_safetensors_file = file_exists(
  File "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_validators.py", line 106, in _inner_fn
    validate_repo_id(arg_value)
  File "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_validators.py", line 154, in validate_repo_id
    raise HFValidationError(
huggingface_hub.errors.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/opt/ml/model'. Use `repo_type` argument if needed.

Will open an issue in peft, once fixed we should add >= for the version.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.