princeton-nlp / cepe Goto Github PK

Hello, I'm processing redpajama data and it's unacceptably slow, especially processing book domain, any suggestions please?
Or can you share a copy of your processed training data, thanks a lot!

Congratulations on your excellent work. Intuitively, introducing new parameters for cross-attention may lead to a high loss. Could you please share your loss curve? Thanks a lot!

Congratulations on your excellent work! I attempted to run bash scripts/run_streamingllm_lm.sh to reproduce the results of streaming_llm, but I encountered the following error:

TypeError: llama_pos_shift_attention_forward() got an unexpected keyword argument 'padding_mask'

It seems that the transformers library version needs to be below 4.34.0, but this project requires version 4.34.1. Have you encountered the same issue? If so, do you have any solutions? Thank you very much!

I'm curious about the discrepancies between my results (in red font) and the results presented in your paper (in black font), both obtained using the default parameters with the run_qa.sh script.

Could there be any potential errors on my end that could explain these differences?

Hi, thanks for the awesome work. However, during dataset collection, I found Books3 in RP has already been removed due to copyright issue. Is there any other possible resource for this dataset? I thought it was indispensible and significant for the training process as you mentioned in your paper.