Comments (1)
Hey @thincal, definitely we plan on supporting DoRA. I think the main thing that needs to be figured out is how to efficiently serve DoRA without merging back the weights into the base model, while still achieving good throughput and latency. This will require some experimentation on our side.
from lorax.
Related Issues (20)
- AutoTokenzier.from_pretrains needs setting with `trust_remote_code` inside `load_module_map` HOT 2
- Ensure api_token is not included in the response on error HOT 3
- [QUESTION] How to change HuggingFace model download Path in Lorax When deployed to Kubernetes through HelmChart HOT 1
- Bug Report: lorax-launcher failed with --source "s3" for model_id "mistralai/Mistral-7B-Instruct-v0.2" HOT 1
- Improve warmup checking for max new tokens when using speculative decoding
- Support inference on INF2 instance
- Reject unknown fields from API requests
- When caching adapters, cache the adapter ID + the API token pair HOT 4
- Add HTTP status codes to docs HOT 1
- Quantized KV Cache
- `make install` insufficient for running llama3-8B-Instruct HOT 4
- Fail to run Phi-3 HOT 9
- Quickstart example not working HOT 3
- AssertionError when using model "google/gemma-2b" with multi-gpus
- can't run lorax with docker. HOT 1
- Why are qlora (4bit) and lora (16bit) adapter file sizes the same?
- Fail to load special token in phi-3
- Add Support for AutoModelForSequenceClassification Models
- can't start my local llama3 model server with docker
- Important: In latest main, the server can not serve more than 1 user HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from lorax.