What is the performance of PoSE without using interpolations like linear / NTK / YaRN? about pose

[dwzhu-pku]

Hi, many thanks for your attention @66RING ! Across all our experiments, unless explicitly stated, linear interpolation is used by default. So Figure 3(c) are results of Full-length / PoSE using linear interpolation. We did not attempt Full-length / PoSE fine-tuning without position interpolation (PI), as previous investigations have demonstrated that such naive training approaches without PI tend to progress very slow, hence being very costly.

[1] PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training.

[dwzhu-pku]

Hi, @SkyAndCloud , we sincerely appreciate your attention to our work! PoSE can indeed be applied to NTK and YaRN, and for more details, please refer to Figure 4 in our paper (link: https://arxiv.org/pdf/2309.10400.pdf). However, it's important to note that PoSE doesn't improve PI/NTK/YaRN; rather, it serves as a memory and time-efficient alternative to full-length fine-tuning.

[SkyAndCloud]

@dwzhu-pku Hi, thanks for the excellent work. As you mentioned, PoSE can improve the PI method? Can PoSE also enhance the effects of NTK/YaRN?