Comments (3)
thanks for taking the time to open this ticket @neel-desh - sorry for the slow response.
In order to add new languages the model needs to be fine-tuned and updated as I mentioned in #16
It's currently not possible to train or fine tune the quantized models so we'd need to take the original unquantized models and fine tune those.
Of course the other challenge is finding an appropriate corpus to fine-tune the model on. You'd need a suitably large set of Elixir lang repos with which to train it and then apply the training procedure.
I'm definitely interested in helping to support this if you're interested in trying to build parts of it yourself. I am also very interested in supporting LORA as I can see a use case for allowing users (or companies even) to fine tune turbopilot on their own historical code repos and have it produce suggestions in their style
from turbopilot.
Yes, I'm so sorry I missed the email notification,
Im free and Ill be more than happy to contribute not only for elixir but any other languages that is required
from turbopilot.
hey @neel-desh - sorry I've been pretty busy recently too, I've only just remembered that you replied.
Since you opened this thread a few new models have come out that I want to add support for in TurboPilot but the only one that I can find which already supports Elixir is StarCoder which requires at least 20GB RAM to run in pytorch and would probably still need ~10-12GB RAM to run as a ggml model.
Therefore, I still think there is some merit in trying to add support for Elixir to smaller models and using that as a learning exercise for adding other languages.
I'm going to have a go at scoping out all the necessary steps here. I still think that a LoRA adapter is probably the way to go (basically we plug in some additional layers to the model to give it the ability to deal with a specific language).
In order to train a LoRA adapter we need to gather training data - you'll need a whole bunch of elixir code. The training set that StarCoder was trained on is the stack which is a huge dataset with 300+ different languages in it. I have just verified that it also has a subset of elixir files in it so you could try extracting this as a first step although you will need to pay attention to the license notices when you first log in.
The next steps will be to train a LoRA adapter model, add support for LoRA models to turbopilot based on existing Llama support for them, quantise and integrate the two models. I'll do some more research on what is involved.
from turbopilot.
Related Issues (20)
- Local build failing to run (NO AVX2) HOT 3
- use WebSocket for Real-time reception
- terminated by signal SIGABRT (Abort)
- Fauxpilot client does not communicate with TurboPilot πserver
- How to use it with cuda in v0.0.5 HOT 2
- Any chance for cuda 12 support? HOT 1
- Is there any roadmap to add support for replit models? HOT 2
- Add support for StableCode
- Support Huggingface Code plugin
- "symbol not found" error in docker image running under ARM64 HOT 3
- How to build for Mac OS Apple Silicon? HOT 2
- ggml_new_tensor_impl: not enough space in the context's memory pool HOT 7
- docker turbopilot:v0.1.0-cuda12 not using gpu HOT 3
- OOM - Segmentation fault (core dumped) HOT 3
- CAN NOT RUN TURBOPILOT USING DOCKER HOT 5
- Only huggingface client works, and crashes server HOT 3
- Support for Code Llama HOT 1
- Docker Image Fail to load model
- Failed to load model wizardcoder (Illegal Instruction)
- [Feature request] Add refact model
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from turbopilot.