Reload the prunered Model failed about llm-pruner

[horseee]

And thanks for your reminder =v=
We updated the README.md about how to load the model.

[horseee]

Hi.
One simple way to fix it is not to save the tokenizer. You can delete Line 217 in hf_prune.py, which will become:

if args.save_model:
        model.half()
        torch.save({
            'model': model, 
        }, logger.best_checkpoint_path)

And after that, if you load your model on the local machine, you will encounter another problem that says LLMPruner not found or something like that (I don't remember the exact error message). The way to solve it is to put the folder LLMPruner in the same directory you execute the loading script.

And, that is the problem of using torch.load() and torch.save() to load and restore the model, which will cause a lot of problems if switching to a new machine or even changing to a new folder 😔

[horseee]

Hi.

We provide a script 'generate.py' to use the pruned model. The instruction can be found in README.md. Another simple way to load the model is :

checkpoint = torch.load('pytorch_model.bin', map_location = 'cpu')
model, tokenizer = checkpoint['model'], checkpoint['tokenizer']

Since the model does not have a consistent configuration for all layers (like some layers might be wider than others), we cannot use the model.from_pretrained to load the model. We are still working on this to make it possible.

[Zhuqln]

thanks！appreciate that!
i just tried but i get a wired issue, i pruner the base model on my server ,then i took the state_dict down to my local machine to do some test ,but i found that the state_dict cached the server_file_path

i tried:

import torch

pruned_dict = torch.load('/home/usr-fy/vicuna7b_pru/pytorch_model.bin', map_location='cpu')
tokenizer, model = pruned_dict['tokenizer'], pruned_dict['model']

'/home/usr-fy/vicuna7b_pru/pytorch_model.bin' is my local machine path
then i get :

  File "/home/usr-fy/anaconda3/envs/LLM-Pruner/lib/python3.10/site-packages/sentencepiece/__init__.py", line 310, in LoadFromFile
    return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
OSError: Not found: "/data/app/vicuna-7b-v1.1/tokenizer.model": No such file or directory Error #2

the "/data/app/vicuna-7b-v1.1/tokenizer.model"is the base_model_path on my server

i dont know how to modify the path, it seems contained by the model.bin file

[Zhuqln]

thanks a lot for your reply! i get your point
i'll try it later :)

[VainF]

Hi @Zhuqln , we just implemented a new feature to save & load pruned models https://github.com/VainF/Torch-Pruning#3-save--load. This feature allows us to re-create a pruned model from unpruned ones.

We will test it on LLM-Pruner and get back to you if there are any updates.

[Zhuqln]

appreciate that ，i'm looking forward to your achievements！ @VainF
by the way, i'm using LLM-Pruner for inference now and its goes well @horseee ，thanks!

[horseee]

Glad it's working well for you!