dwzhu-pku / pose Goto Github PK
View Code? Open in Web Editor NEWPositional Skip-wise Training for Efficient Context Window Extension of LLMs to Extremely Length (ICLR 2024)
Home Page: https://arxiv.org/abs/2309.10400
License: MIT License
Positional Skip-wise Training for Efficient Context Window Extension of LLMs to Extremely Length (ICLR 2024)
Home Page: https://arxiv.org/abs/2309.10400
License: MIT License
Thank you so much for your work and your open source code.
But some problems occurred when running lm_eval. I guess it is a version problem.
Hi, this seems like an amazing breakthrough to me, and I'm not sure how it isn't getting more attention.
Anyway, I was looking at your docs and particular your scripts. Do you all have links to the training/testing jsonl files you used? I wanted to look at it to get an idea of what to pass in.
bash script/run_train_baichuan.sh 64 yarn
factor=$1
rope_type=$2
debug_mode="-m debugpy --listen 127.0.0.1:6679 --wait-for-client"
deepspeed src/train_baichuan.py
--model_name_or_path ./baichuan2-7b-base
--train_data_path ./data/pile/train_00_long_10w.jsonl
--valid_data_path ./data/pile/val_long.jsonl
--test_data_path ./data/pile/test_pg19.jsonl
--output_dir ./skipos/results/baichuan2/4k-$((factor*4))k-${rope_type}
--max_steps 1000
--model_max_position_embeddings 4096
--rope_scaling_type ${rope_type}
--rope_scaling_factor $factor
--inference_length 16384
--per_device_train_batch_size 4
--per_device_eval_batch_size 1
--gradient_accumulation_steps 2
--do_train True
--do_eval True
--do_predict True
--evaluation_strategy "steps"
--eval_steps 50
--save_strategy "steps"
--save_steps 500
--warmup_steps 0
--learning_rate 2e-5
--logging_steps 10
--report_to "tensorboard"
--gradient_checkpointing True
--fp16 True
--deepspeed src/configs/deepspeed_config.json \
hi, @dwzhu-pku , have you ever tried performing PI directly on a length of 2k, and then compared using PoSE to perform PI on a length of 2k? Also fine-tune 1000 steps with the same parameters.
My results show that there is not much difference between the two, but I did not find any relevant comparative experiments in the paper.
Please correct me if I'm wrong.
能支持LLaMA3的上下文扩展吗,效果怎么样
“input_ids” is the same as "labels" in train_preprocess_function_pose function ?
model_inputs = {"input_ids": input_ids, "position_ids": position_ids, "labels": input_ids}
将训练数据集替换成中文的,代码需要修改吗
Hi, thanks for the nice work! I see you mentioned you use 8x V100 for training. I wonder how long it takes for training for e.g., Llama2? And is there any modification I need to make if I want to use flash attention during training with A100 GPUs? Many thanks!
Hi nice work on the paper. I pulled your code and model locally (LLaMA-7B-PoSE-YaRN-128k).
I run it against the LongChat evaluation "topic" retrieval but model does very very poorly > 4096 tokens
I noticed that your model does well on pass-key, I run it locally and it performs well. It seems that maybe pass-key is not a good benchmark?
LongChat benchmarks can be found here: https://github.com/DachengLi1/LongChat
您好,我理解的是如果想拓展到4096的长度,那就把4096的example分成两个2048长度的example,那样一个example就变成两个了(当然会加一些term去链接两个example)。但是训练的时候,单个样本还是2048的长度呀,他并没有真正的见到4096。那怎么保证可以拓展到无限长度呢?
Line 172 in e0eb5c7
Hi,in line 172 of the file, it appears that two non-adjacent fragments have been cut from the original training data, and each of these fragments has been assigned a continuous positional code in subsequent processing. Why is this done?
In a real SFT scenario, the raw data input is often continuous fragments. What is the problem with using continuous fragments and applying pose?
My intention is to use the PoSE to pre/fine-train the LLM with diverse structured data like JSON and XML documents.
Unlike natural language text, structured data doesn't meaningful units sentences. Instead it consists of key-value pairs, nested objects, arrays, and etc. Besides that I am dealing with a very large documents like XML/JSONs with 100k+ tokens.
Do you think I can use PoSE for that?
Regarding the script code: I am struggling to match the code with the paper.
Figure 6: Python Code used for calculating coverage probability of each relative position in Figure 5.
could you brief explain what these methods is doing?
train_pose.py
Thank you in advance.
建议把数据上传到国内的魔塔社区一份
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.