AttributeError: 'NoneType' object has no attribute 'to_legacy_cache' when training about infini-attention

[jlamprou]

You are right, I tend to train with use_cache=False and didnt see it, It must be fixed now with latest commit, do a git pull

[ckybit]

thanks for your reply. I use two 40G A100 and got cuda out of memory...
i am wondering how many gpu you used when training?

[jlamprou]

Refer to #4 , The pre-trained model may barely fit in an A100 40GB with 1 batch size , bf16 and an 8-bit optimizer. Otherwise if you have multiple A100s in parallel i suggest either mapping the model on the GPU or using FSDP with Accelerate. Lastly you could create a QwenConfig with less parameters but you will have to pretrain it on billions of tokens before you get some good performance. You can use the 🤗 Model Memory Calculator to check how much vram the model needs.