yangjianxin1 / balanced-dataparallel

这里是改进了pytorch的DataParallel, 用来平衡第一个GPU的显存使用量

Hi there 👋, I'm Yang Jianxin

I'm a NLPer interested in Large Language Model and graduated from SYSU with a master's degree.

In my free time, I like to write technical blogs on [Wechat Official Accounts: YeungNLP] and [Zhihu: 红雨瓢泼]

🔭 Experiences:

• Shopee, responsible for building NLP algorithm ability about Customer Service. (from 2022-04 to now)
• Tencent, responsible for building NLP algorithm ability about Product Understanding. (from 2021-06 to 2022-04)
• Alibaba, Internship at Alibaba (from 2020-06 to 2020-09).

⚙ Here are some my public projects:

Project	Description	Code
Firefly	One-stop training for LLMs. Some achievements: 1. firefly-llama2-13b ranked 3rd among all 13B models on Open LLM Leaderboard, only 0.5 points less than 1st. 2. firefly-llama-30b ranked 10th among all 30B models on Open LLM Leaderboard trained with single V100. 3. firefly-baichuan-13b achieves over 1.63 million downloads. 4. firefly-qwen1.5-en-7b-dpo improves 7.21 points compared with the official chat model. 5. firefly-gemma-7b improves 9.37 points compared with the official chat model.
GPT2-chitchat	Chinese GPT2 for chitchat
Firefly-LLaMA2-Chinese	Chinese Llama2 with efficient and effective training method.
LongQLoRA	Efficient and Effective method for extending context length of Llama2 to 8192 with single V100. Technical Report
CPM	Chinese composition model based on CPM
CLIP-Chinese	Chinese CLIP model trained with 1.4 million image-text pairs
ClipCap-Chinese	Chinese image caption model based on clip and mengzi
OFA-Chinese	Chinese multi-modal unified pre-training model
LLMPruner	Prune vocabulary of LLMs to save memory in training.

📁 Here are some my technical blogs:

• 📝 使用Firefly在单卡V100上对Qwen1.5进行SFT和DPO，大幅超越Qwen1.5和Gemma
• 📝 图解大模型推理优化之KV Cache
• 📝 Mixtral-8x7B MoE大模型微调实践，超越Llama2-65B
• 📝 LongQLoRA：单卡高效扩展LLaMA2-13B的上下文长度
• 📝 详解基于调整RoPE旋转角度的大模型长度外推方法
• 📝 图解RoPE旋转位置编码及其特性
• 📝 QLoRA轻量级增量预训练方案，及汉化Llama2的实践
• 📝 源码解析ChatGLM2多轮对话训练方法的不足，以及改进方法
• 📝 微调百川Baichuan-13B保姆式教程，手把手教你训练百亿大模型
• 📝 QLoRA文章解读 & 单卡高效微调bloom-7b1
• 📝 Firefly(流萤): 中文对话式大语言模型
• 📝 LLMPruner：大语言模型裁剪工具