serp-ai / llama-8bit-lora

Repository for Chat LLaMA - training a LoRA for the LLaMA (1 or 2) models on HuggingFace with 8-bit or 4-bit quantization. Research only.

Chat LLaMA

8bit-LoRA or 4bit-LoRA

Repository for training a LoRA for the LLaMA (1 and 2) models on HuggingFace with 8-bit or 4-bit quantization. Research only for LLaMA 1, LLaMA 2 is open commercially.


👉 Join our Discord Server for updates, support & collaboration

Dataset creation, training, weight merging, and quantization instructions are in the docs.

Check out our trained LoRAs on HuggingFace

Anthropic's HH

• 7B
• 13B
• 30B

llama-8bit-lora's Issues

Error on "unk_token": tokenizer.convert_ids_to_tokens

This issue was created on DagsHub by:
Tizzzzy

I am getting this error

Traceback (most recent call last):

File "//train/LLaMA-8bit-LoRA/finetune_peft_8bit.py", line 261, in main() File "//train/LLaMA-8bit-LoRA/finetune_peft_8bit.py", line 188, in main "unk_token": tokenizer.convert_ids_to_tokens( File "/opt/conda/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 388, in convert_ids_to_tokens for index in ids:

TypeError: 'NoneType' object is not iterable

I see you using a tool-following approach on gpt-3.5 to generate a dataset for the Lora training, would you be willing to put your weights up somewhere?

This happens to be the exact problem I can here trying to solve and it seems like you've already been at it.

This is very cool. Would you happen to be willing to share the Lora weights that you've trained based on that? Perhaps they are already up somewhere. Maybe you already have some stuff on HuggingFace. Could be cool to link to from the readme.

By the way, using the Lora approach is just great. It avoids the licensing problems very deftly, and its just a more efficient and hot-swappable way to go. Really impressed.

Flash Attention

With Flash Attention, can we fine-tune model for seq length to say 8k? What code changes needs to be done in your repo for this? Can you help us with this. Thanks!

The 7B LoRA downloads the 13B foundational model instead of the 7B one

Why does the LoRA serpdotai/llama-hh-lora-7B config file point to decapoda-research/llama-13b-hf instead of decapoda-research/llama-7b-hf?

ValueError: Tokenizer class LLaMATokenizer does not exist or is not currently imported. After 7B foundational model download.

After the download of the 7B foundational model, the merge_adapter_weights.py script crashes with the error Tokenizer class LLaMATokenizer does not exist or is not currently imported.

I found that the tokenizer class should be LlamaTokenizer instead of LLaMATokenizer.

Traceback (most recent call last):
  File "/Users/.../LLaMA-8bit-LoRA/merge_adapter_weights.py", line 35, in <module>
    tokenizer = AutoTokenizer.from_pretrained(peft_config.base_model_name_or_path)
  File "/Users/.../venv/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 677, in from_pretrained
    raise ValueError(
ValueError: Tokenizer class LLaMATokenizer does not exist or is not currently imported.

How to finetune a customize data

Hi,
Huge fan of your work. I see right now the code is using a dataset from HF. I was wondering if I can change the dataset to my own dataset and lora llama2 with 8bit quantize. How should I change the code?
Thank you