bojone / rerope

Rectified Rotary Position Embeddings

Rectified Rotary Position Embeddings (ReRoPE)

Using ReRoPE, we can more effectively extend the context length of LLM without the need for fine-tuning.

Blog

• https://kexue.fm/archives/9706 (Chinese)
• https://kexue.fm/archives/9708 (Chinese)
• https://normxu.github.io/Rethinking-Rotary-Position-Embedding-2/ (English by @NormXU)
• https://normxu.github.io/Rethinking-Rotary-Position-Embedding-3/ (English by @NormXU)

Idea

Results

Calculated the loss on llama2-13b with samples_15k.jsonl:

Method	loss
RoPE-4k(original llama2-13b)	1.4967
RoPE-8k(original llama2-13b)	8.8615
NTK-RoPE-4k(not dynamic)	1.6081
NTK-RoPE-8k(not dynamic)	1.5417
NTK-RoPE-16k(not dynamic)	1.5163
ReRoPE-w1024-4k	1.4996
ReRoPE-w1024-8k	1.4267
ReRoPE-w1024-16k	1.4001

ReRoPE's performance at training length (4k) has hardly decreased, and it possesses the ideal property of "longer context, lower loss".

Usage

Dependency: transformers 4.31.0

Run python test.py to test chatting or run python eval_loss.py to calculate loss with llama2.

From here and here, we can see what modifications ReRoPE/Leaky ReRoPE has made compared to the original llama implementation.

Other

Triton Implementation of ReRoPE: https://gist.github.com/chu-tianxiang/4307937fd94b49c75b61a6967716bae9

Cite

@misc{rerope2023,
  title={Rectified Rotary Position Embeddings},
  author={Jianlin Su},
  year={2023},
  howpublished={\url{https://github.com/bojone/rerope}},
}

Communication

QQ discussion group: 67729435, for WeChat group, please add the robot WeChat ID spaces_ac_cn

rerope's Issues

problem with rerope_patch

请问rerope的实验结果是基于这个仓库的pytorch代码吗？我本地无法直接正确运行

Blogs in English

Thank you very much for sharing your awesome work!

As the blogs mentioned in README are in Chinese, I am working on translating them into English. I guarantee you I am not using any AI translations. I believe people working on expanding context length will love these blogs and draw inspirations from them.

I have finished some parts, and for people who can't read Chinese, please check

• part1
• part2
• part3

Generating same token

I was running the code using your Rerope implementation with vicuna-7b for Code completion task but each time it is producing the same sequence of tokens like nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody nobody'. I am using transformer 4.41 can you please inform me what could be the issue behind the generation. Dataset that I am using here is https://huggingface.co/datasets/microsoft/LCC_python. I was actually implementing this to reproduce the baseline results of Hirope..

为什么ntk rope那里要乘以-a

#ntk rope实现
inv_freq *= (-a * torch.arange(1, dim // 2 + 1).float().to(device)**b).exp()
##文章中给的公式是这样的
λ1λ2⋯λm=exp(amb)(12) #https://kexue.fm/archives/9706

<q_len=1> question

rerope/rerope_patch.py

Lines 68 to 74 in 3710a35

	if q_len == 1:
	cos, sin = self.rotary_emb(value_states, seq_len=kv_seq_len)
	position_ids = (position_ids[:, -1] - position_ids).clip(max=window)
	_, key_states = apply_rotary_pos_emb(None, key_states, cos, -sin, position_ids)
	key_states = repeat_kv(key_states, self.num_key_value_groups)
	value_states = repeat_kv(value_states, self.num_key_value_groups)
	attn_weights = torch.matmul(query_states, key_states.transpose(2, 3)) / math.sqrt(self.head_dim)

In line 71, can you explain why sin is negated? And also why does it have nothing to with window size?

rerope：为什么对大于train_seqlength的query token要稍微放大一点

https://github.com/bojone/rerope/blob/e30e7f6fec2d11db17211e164f3b5f96957e19d1/rerope_patch.py#L56C5-L56C122
作者您好～我想请教一下上面这一行代码起着什么作用。。谢谢🙏

[BUG] rope implementation question

I filed a bug report on huggingface, huggingface/transformers#30900, @bojone can you confirm if it makes sense?

Monkey patch for original LLaMA2 code.

作者您好，感谢您的分享！
目前我的工作基于meta发布的原始llama2的代码和权重上，不知道本仓库是否有更新基于原始llama2仓库的rerope的patch的实现方式的计划呢？

Excellent Idea！！！！

Could you support llama2 with rerope?

position_ids_q - position_ids_k < window

数据集samples_15k

请问苏神所用的数据集samples_15k.jsonl的来源是什么？

[BUG] relative position ids seems wrong

rerope/rerope_patch.py

Line 70 in 3710a35

position_ids = (position_ids[:, -1] - position_ids).clip(max=window)

should this be
position_ids = (position_ids[:, -1:] - position_ids).clip(max=window) with ":" behind -1, just like leaky-rerope?

rerope/leaky_rerope_patch.py

Line 77 in 3710a35

position_ids = position_ids[:, -1:] - position_ids

"position_ids_q - position_ids_k < window" without abs()

rerope/leaky_rerope_patch.py

Line 96 in acfe788

rectified_mask = (position_ids[:, -q_len:, None] - position_ids[:, None]).abs() < window

I think abs() is not needed here, right? @bojone

Dataset of ReROPE eval

Hi I was going through a paper on the topic of length extrapolation on code models namely "HiRoPE: Length Extrapolation for Code Models" (https://arxiv.org/pdf/2403.19115v1). In this paper authors have mentioned about the ReROPE-eval dataset. Can you please help me to get the ReRoPE-eval dataset because I have not found any dataset in this repository.

LlamaLinearScalingRotaryEmbedding接口参数问题

rerope/leaky_rerope_patch.py

Line 75 in 3710a35

cos2, sin2 = self.rotary_emb2(value_states, seq_len=kv_seq_len + offset)

它的行参是postion_ids，不是seq_len，很困惑这里到底调用的是哪个接口。

https://github.com/huggingface/transformers/blob/745bbfe4bb2b61491dedd56e1e8ee4af8ef1a9ec/src/transformers/models/llama/modeling_llama.py#L148-L155

求助。

cos and sin dimension

rerope/leaky_rerope_patch.py

Lines 78 to 79 in 3710a35

	cos = torch.cat([cos[:, :, :window], cos2[:, :, window + offset:]], axis=2)
	sin = torch.cat([sin[:, :, :window], sin2[:, :, window + offset:]], axis=2)

cos and sin tensors are two dimensional, but here it is indexed as 3-d, is this a problem?

请问在推理阶段，为什么需要对input部份计算注意力呢？

https://github.com/bojone/rerope/blob/3710a359ae0a80aad98312c9194898fb7312dcf3/rerope_patch.py#L75C5-L75C10

请问在这段代码中，为什么在推理阶段需要对input部份计算注意力，为什么不能直接使用 if q_len == 1 中的逻辑呢？

ntk_rope_mixed_init 中old_init是否可以简化，省略inv_freq、_set_cos_sin_cache()步骤

作者您好～
在ntk_rope_mixed_init的实现中，首先计算了 old_init(self, dim, max_position_embeddings, base, device)，但在我理解中old_init除了定义下面的值外，
super().__init__()
self.dim = dim
self.max_position_embeddings = max_position_embeddings
self.base = base

计算inv_freq，self.register_buffer("inv_freq", inv_freq, persistent=False)，self._set_cos_sin_cache()这几步都会在ntk_rope_mixed_init接下来的计算中被替换掉，因此是否可以将old_init简化为仅下式呢？这样可以节省很多的显存
super().__init__()
self.dim = dim
self.max_position_embeddings = max_position_embeddings
self.base = base

运行 test.py 显存爆了

请问苏神的GPU配置是什么？ 我两个A100 80G 直接运行python test.py 显存爆了， 也没找出原因

测了一下千问

前几个问题可以，后面就坍塌了，ntk表现也差不多。可能是模型的问题·。