Training time is unexpectedly very slow compared to lit-llama about litgpt

[iskandr]

I'm also hitting some CUDA out of memory errors on models + data that I expect to more easily fit on a 40GB A100 MiG.

I'm not familiar with the lit-llama codebase, so I'm not sure what's potentially different in lit-parrot but wanted to note that I'm seeing something similar.

[carmocca]

Do you still see this behaviour, and if so, can you share exactly the code you ran and the arguments passed?

[carmocca]

This is because LLaMA fine-tuning is hardcoded to use 256 max_seq_length:
https://github.com/Lightning-AI/lit-llama/blob/main/scripts/prepare_alpaca.py#L26
https://github.com/Lightning-AI/lit-llama/blob/main/finetune/adapter.py#L52

Whereas this repository is configured to use the longest sequence length in alpaca: 1037. If you override it to 256 in https://github.com/Lightning-AI/lit-gpt/blob/main/finetune/adapter.py#L30, you should see the times match.

[LamOne1]

Actually I was using the pretrain script, and I think the max token length is fixed in both lit-llama and lit-gpt?