Comments (10)
Hi @sasha0552,
Thank you for bring this up. For now, would you mind maintain this in your fork? There are few reasons that we are hesitant to include support for Pascals:
- Aside from Triton, we are continuously relying on Cutlass, FlashAttention, and FlashInfer which all seems to dropped Pascal.
- It is sufficiently easy to build from source in vLLM with Pascal support.
- As we add more features and performance optimizations, we are afraid we can no longer test and maintain the support for support for Pascal due to added complexity.
from vllm.
Hello. Can #4409 be included in any of the next releases? Or at least, can I get an explanation as to why it can't be included (maybe I can help in some way?).
Once the wheel size limit increase was approved, and #6394 was merged, wheel size should not be an issue.
I am currently waiting for PyPI staff to approve the wheel size limit increase request to publish the patched triton
to PyPI (pypi/support#4295).
It would be nice to see support for Pascal GPUs in vLLM. Many people use them because they are cheap.
from vllm.
July 23rd is Tuesday. Do you mean July 24th?
from vllm.
hello, when will v0.6.0 release? I'm looking forward to #5036 and MiniCPM-Llama3-V-2_5
from vllm.
v0.5.2 has been released: https://github.com/vllm-project/vllm/releases/tag/v0.5.2
from vllm.
Can this PR be added in v0.5.3?
#5036
from vllm.
@AlphaINF unlikely given the current state of the PR at the moment (still being reviewed). but I'm looking very much forward to this PR as well!
from vllm.
@simon-mo thanks!
from vllm.
@simon-mo for this "async scheduling to overlap scheduling" do we have a plan๏ผ
from vllm.
Would it be possible to get #6594 merged in before the next release is due? @joerunde @Yard1
from vllm.
Related Issues (20)
- [Usage]: Qwen2 GGUF model can't run successfully HOT 7
- [Feature]: Overlap model weight loading and model prefill HOT 2
- [RFC]: Add Ascend NPU as a new backend
- [Bug]: Inconsistent Output Behavior with and without tools and tool_choice Parameters
- [Usage]: Periodic snapshots for spot instances HOT 1
- [RFC]: Enable Memory Tiering for vLLM HOT 5
- [Bug]: vLLM inconsistently crashes on startup for multinode cluster HOT 7
- [Usage]: how to abort request? HOT 1
- [Usage]: alignment between trl and llm.generate
- [Bug]: errors when loading mixtral 8x7b
- [Bug]: The MixtralForCausalLM architecture and the mistralai/Mixtral-8x7B-Instruct-v0.1 model are stated to be supported by vLLM, but an error occurs during model loading. HOT 3
- [Bug]: Unable to use fp8 kv cache with neuralmagic quants on ampere HOT 1
- [Doc]: AutoAWQ quantization example fails HOT 3
- [Bug]: Error loading microsoft/Phi-3.5-vision-instruct HOT 1
- [Bug]: torch.OutOfMemoryError: CUDA out of memory HOT 5
- [Bug]: Using CPU for inference, an error occurred. [Engine iteration timed out. This should never happen! ] HOT 4
- [Usage]: How to use FP8 or other quantization algorithms for Minicpmv2_6
- [Bug]: Unexpected non-determinism with vLLM 0.5.4 and Llama 3.1 HOT 3
- [New Model]: MiniCPM-V-2_6-int4 HOT 1
- [Usage]: Potential Hardware Failure when running vllm HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from vllm.