llama 13b as evaluate_from_model does not use GPU about alpaca_eval

[rtaori]

Hi @RealTong, just following up on this.
I've fixed the issue with falcon #47 and it's now able to run decoding (there still be a small issue #46 we're looking into).

Regardless, could you please pull the latest from main and see if your original GPU issue has been resolved? Just to clarify, I've tested alpaca_eval evaluate_from_model --model_configs 'falcon-7b-instruct' on my system and have verified that the GPU is being utilized.
If you're still running into a GPU issue, please feel free to open another issue and post the relevant details.

[rtaori]

Hi @RealTong ,
This seems to be a runtime issue instead of a gpu not in use issue. The error stacktrace seems to suggest a recursion error during tokenization. What is your transformers version? Could you upgrade to transformers>=4.29.2 and try again?

[RealTong]

I've tried transformers>=4.29.2 and it still doesn't work. It starts reporting RecursionError: maximum recursion depth exceeded while calling a Python object after 3 minutes of running.

Is there any other useful information I can provide?

[rtaori]

I see. The issue could be due to some issue in your local checkpoint, which would be hard for me to help you debug. As a sanity check, could you try running a model that we've already integrated/validated? For example, falcon-7b-instruct?

alpaca_eval evaluate_from_model --model_configs 'falcon-7b-instruct'

[RealTong]

falcon-7b-instruct:
  prompt_template: "/root/code/eval/models/falcon-7B/prompt.txt"
  fn_completions: "huggingface_local_completions"
  completions_kwargs:
    model_name: "/root/.cache/LLM-Repo/model/falcon-7b-instruct/"
    model_kwargs:
      torch_dtype: 'bfloat16'
    max_new_tokens: 2048
    temperature: 0.7
    top_p: 1.0
    do_sample: True
    trust_remote_code: True
  pretty_name: "Falcon 7B Instruct"
  link: ""

I run alpaca_eval evaluate_from_model --model_configs 'falcon-7b-instruct' and I am prompted with ValueError: Loading /root/.cache/LLM-Repo/model/falcon-7b -instruct/ requires you to execute the configuration file in that repo on your local machine. Make sure you have read the code there to avoid malicious use. Then set the option trust_remote_code=True to remove this error.

Why do I load the local model and execute the code from the network. I have added trust_remote_code: True to the proxy configs.yaml, but it still gives me this error, how should I fix it?

[rtaori]

Ah great point! Apologize for the oversight. Just patched the configs with the proper flag. Could you try again?

[RealTong]

Not working :).

Reported error: TypeError: transformers.pipelines.base.infer_framework_load_model() got multiple values for keyword argument 'trust_remote_code'

Here is my configuration

falcon-7b-instruct:
  prompt_template: "/root/code/eval/models/falcon-7B/prompt.txt"
  fn_completions: "huggingface_local_completions"
  completions_kwargs:
    model_name: "/root/.cache/LLM-Repo/model/falcon-7b-instruct/"
    model_kwargs:
      torch_dtype: 'bfloat16'
      trust_remote_code: True
    max_new_tokens: 2048
    temperature: 0.7
    top_p: 1.0
    do_sample: True
  pretty_name: "Falcon 7B Instruct"
  link: ""

[rtaori]

Ok very interesting, I'll need to investigate this issue more when I get my GPU machines back. Currently our cluster is down for the week, so I'll provide an update next week.

Since this issue is specific to falcon, let's try another model that doesn't require the trust_remode_code Flag. Can you try

alpaca_eval evaluate_from_model --model_configs 'oasst-sft-pythia-12b'

Apologize for the back and forth on resolving this.