Comments (2)
Hi! We faced this problem, so there are several things you can do:
- make lora finetune params as "small" as they can be, use finetune settings section for that (Lora R, Lora Alpha)
- try different checkpoints (from early steps)
Combining these two, you can find a balance between performance on humaneval / your codebase
However, you can never completely beat that problem while you're using the finetune. There are a couple of methods to prepare data to make the problem less visible though, https://arxiv.org/abs/2312.05934. I guess we'll revisit this in some time, but you're welcome to contribute if you have some ideas
from refact.
Hi! We faced this problem, so there are several things you can do:
- make lora finetune params as "small" as they can be, use finetune settings section for that (Lora R, Lora Alpha)
- try different checkpoints (from early steps)
Combining these two, you can find a balance between performance on humaneval / your codebase
However, you can never completely beat that problem while you're using the finetune. There are a couple of methods to prepare data to make the problem less visible though, https://arxiv.org/abs/2312.05934. I guess we'll revisit this in some time, but you're welcome to contribute if you have some ideas
Thx, i will try. I have two questions:
(1) My code-dataset is relatively large(≈1G),may be better to fine-tune on full parameter? but full parameter fine-tuning is more prone to catastrophic forgetting than lora, is that right ?
(2) I find very few project use FIM to fine-tune codellama, but use instruction, but you use it here.
My current task is to fine-tune and implement performance(Generation && FIM) enhancements based on our internal code, do you have any better suggestions ?
Thx again !
from refact.
Related Issues (20)
- docker image fails to start on mac m3 HOT 8
- Database not starting? HOT 2
- Finetune of deepseek-coder fails HOT 7
- refact refuses to finetune if finds weird bytes in files HOT 2
- more files to process than processes HOT 4
- Self host v1.4.0 MODEL always /infengine-v1/completions-wait-batch WAIT time out HOT 10
- Latest lora checkpoints for deepseek-coder/5.7b/mqa-base only generate 1 token to some requests HOT 1
- [VS Code] Multiline completion not working in some cases
- VRAM memory leak for Refact.AI 1.6B HOT 10
- stats problem HOT 1
- run without database oss HOT 1
- error running docker on wsl with cuda HOT 1
- Llama2 chat model times out HOT 1
- VSCode plugin broken by "Cannot reach the server:..." HOT 1
- Missing link on page https://docs.refact.ai/faq/ HOT 2
- Maybe hide popup when stats is empty? HOT 2
- how add local model to mapping in docker-compse mount
- Add stablelm models
- docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]]. HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from refact.