Comments (8)
i do not use ray because i can not sudo to change the progress limit
from vllm.
upgrade to 0.5.1 should help.
from vllm.
i will try
from vllm.
upgrade to 0.5.1 should help.
it fail too:
[rank0]: Traceback (most recent call last):
[rank0]: File "/llama3/mybench/mbxp/my_vllm.py", line 19, in <module>
[rank0]: llm = LLM(model=the_path, gpu_memory_utilization=0.8, tensor_parallel_size = 2)
[rank0]: File "/miniconda/envs/vllama/lib/python3.9/site-packages/vllm/entrypoints/llm.py", line 149, in __init__
[rank0]: self.llm_engine = LLMEngine.from_engine_args(
[rank0]: File "/miniconda/envs/vllama/lib/python3.9/site-packages/vllm/engine/llm_engine.py", line 414, in from_engine_args
[rank0]: engine = cls(
[rank0]: File "/miniconda/envs/vllama/lib/python3.9/site-packages/vllm/engine/llm_engine.py", line 243, in __init__
[rank0]: self.model_executor = executor_class(
[rank0]: File "/miniconda/envs/vllama/lib/python3.9/site-packages/vllm/executor/distributed_gpu_executor.py", line 25, in __init__
[rank0]: super().__init__(*args, **kwargs)
[rank0]: File "/miniconda/envs/vllama/lib/python3.9/site-packages/vllm/executor/executor_base.py", line 42, in __init__
[rank0]: self._init_executor()
[rank0]: File "/miniconda/envs/vllama/lib/python3.9/site-packages/vllm/executor/multiproc_gpu_executor.py", line 78, in _init_executor
[rank0]: self._run_workers("init_device")
[rank0]: File "/miniconda/envs/vllama/lib/python3.9/site-packages/vllm/executor/multiproc_gpu_executor.py", line 130, in _run_workers
[rank0]: driver_worker_output = driver_worker_method(*args, **kwargs)
[rank0]: File "/miniconda/envs/vllama/lib/python3.9/site-packages/vllm/worker/worker.py", line 126, in init_device
[rank0]: init_worker_distributed_environment(self.parallel_config, self.rank,
[rank0]: File "/miniconda/envs/vllama/lib/python3.9/site-packages/vllm/worker/worker.py", line 327, in init_worker_distributed_environment
[rank0]: ensure_model_parallel_initialized(parallel_config.tensor_parallel_size,
[rank0]: File "/miniconda/envs/vllama/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 915, in ensure_model_parallel_initialized
[rank0]: initialize_model_parallel(tensor_model_parallel_size,
[rank0]: File "/miniconda/envs/vllama/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 883, in initialize_model_parallel
[rank0]: _TP = init_model_parallel_group(group_ranks,
[rank0]: File "/miniconda/envs/vllama/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 729, in init_model_parallel_group
[rank0]: return GroupCoordinator(
[rank0]: File "/miniconda/envs/vllama/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 175, in __init__
[rank0]: self.pynccl_comm = PyNcclCommunicator(
[rank0]: File "/miniconda/envs/vllama/lib/python3.9/site-packages/vllm/distributed/device_communicators/pynccl.py", line 89, in __init__
[rank0]: self.comm: ncclComm_t = self.nccl.ncclCommInitRank(
[rank0]: File "/miniconda/envs/vllama/lib/python3.9/site-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 244, in ncclCommInitRank
[rank0]: self.NCCL_CHECK(self._funcs["ncclCommInitRank"](ctypes.byref(comm),
[rank0]: File "/miniconda/envs/vllama/lib/python3.9/site-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 223, in NCCL_CHECK
[rank0]: raise RuntimeError(f"NCCL error: {error_str}")
[rank0]: RuntimeError: NCCL error: unhandled cuda error (run with NCCL_DEBUG=INFO for details)
from vllm.
that's another error you need to check, following https://docs.vllm.ai/en/latest/getting_started/debugging.html
from vllm.
device_communicators/pynccl_wrapper.py", line 223, in NCCL_CHECK
i found why, i use cuda11.8, however the runtime cuda is 11.7.
but i can not change the runtime cuda version, so is there any method to solve it?
init.cc:1674 NCCL WARN Cuda failure 'CUDA driver version is insufficient for CUDA runtime version'
from vllm.
may i use torch 2.0.1 and vllm to offline the llama3 70B or qwen?
from vllm.
Nvidia driver version: 515.43.04
you need to contact your admin to see how to update the driver version.
from vllm.
Related Issues (20)
- [Bug]: OpenGVLab/InternVL-Chat-V1-5 never stops properly HOT 8
- [Bug]: assert num_new_tokens > 0 crashes entire worker instead of just failing single API call HOT 1
- [Feature]: Exit on failures HOT 2
- [Misc]: TTFT profiling with respect to prompt length HOT 3
- [Usage]: The results of the model depend on the number of GPUs. HOT 2
- [Feature]: support chunked_prefill for llava
- [Bug]: Gemma2 models inference using vLLM 0.5.4 produces incorrect responses
- [Bug]: Error happened with Large scale requests based on 0.5.4 vllm HOT 6
- [Misc]: OOM (CUDA Out Of Memory) when running LLMs in WSL using vLLM HOT 4
- [Feature]: json_schema support in OpenAI compat server
- Virtual Office Hours: August 8 and August 21
- [Feature]: GGUF quantization with tensor parallelism HOT 2
- [Bug]: when using llama-3.1-70b-instruct for inference, input with large number of tokens(>8k) will result in endless output
- [Bug]: Mismatch in the number of image tokens and placeholders during batch inference HOT 2
- [Bug]: Streaming API: Abort functionality not working as expected
- [Usage]: how to use guided_decoding_backend? HOT 1
- [Feature]: Please Support FATRelu
- [Usage, bug]: vLLM Docker | ValueError: OpenTelemetry packages must be installed before configuring 'otlp_traces_endpoint' during vLLM startup
- [Performance]: vllm量化和非量化的性能对比 HOT 3
- [Bug]: when `echo=True`, vllm will append chat template(`assistant`) after the last message HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from vllm.