More than 4096 context length? about stablelm

[jon-tow]

Nope; see https://github.com/kyleliang919/Long-context-transformers

[jon-tow]

Sure; You just need to fine-tune to a longer context :)

[StoyanStAtanasov]

@jon-tow You are joking right?

[NPap0]

1. Training code has to change
2. Data that you fine tune the model with after training has to change

So no, nothing can be done user-side to change attention span. (But maybe you can summarize blocks of text with the model so you can then feed the already summarized text as a whole to do your thing)

[mallorbc]

Pretty sure the answer is no due to how positional encoding is done.

[jon-tow]

@mallorbc The model uses RoPE - a relative attention mechanism. See https://blog.eleuther.ai/rotary-embeddings/

[mallorbc]

@jon-tow So models like GPTJ can be finetuned and generate more than their sequence length? Whenever I try to generate sequences for GPTJ I have issues. Maybe that is something else unrelated.

[Ph0rk0z]

A lora will not work for this? Would you have to retrain the model from scratch on long context as you have done?