when i set tensor_parallel_size>1(A100 * 4), it does not work about vllm

[cx-hub]

i do not use ray because i can not sudo to change the progress limit

[youkaichao]

upgrade to 0.5.1 should help.

[cx-hub]

i will try

[cx-hub]

upgrade to 0.5.1 should help.

it fail too:

[rank0]: Traceback (most recent call last):
[rank0]:   File "/llama3/mybench/mbxp/my_vllm.py", line 19, in <module>
[rank0]:     llm = LLM(model=the_path, gpu_memory_utilization=0.8, tensor_parallel_size = 2)
[rank0]:   File "/miniconda/envs/vllama/lib/python3.9/site-packages/vllm/entrypoints/llm.py", line 149, in __init__
[rank0]:     self.llm_engine = LLMEngine.from_engine_args(
[rank0]:   File "/miniconda/envs/vllama/lib/python3.9/site-packages/vllm/engine/llm_engine.py", line 414, in from_engine_args
[rank0]:     engine = cls(
[rank0]:   File "/miniconda/envs/vllama/lib/python3.9/site-packages/vllm/engine/llm_engine.py", line 243, in __init__
[rank0]:     self.model_executor = executor_class(
[rank0]:   File "/miniconda/envs/vllama/lib/python3.9/site-packages/vllm/executor/distributed_gpu_executor.py", line 25, in __init__
[rank0]:     super().__init__(*args, **kwargs)
[rank0]:   File "/miniconda/envs/vllama/lib/python3.9/site-packages/vllm/executor/executor_base.py", line 42, in __init__
[rank0]:     self._init_executor()
[rank0]:   File "/miniconda/envs/vllama/lib/python3.9/site-packages/vllm/executor/multiproc_gpu_executor.py", line 78, in _init_executor
[rank0]:     self._run_workers("init_device")
[rank0]:   File "/miniconda/envs/vllama/lib/python3.9/site-packages/vllm/executor/multiproc_gpu_executor.py", line 130, in _run_workers
[rank0]:     driver_worker_output = driver_worker_method(*args, **kwargs)
[rank0]:   File "/miniconda/envs/vllama/lib/python3.9/site-packages/vllm/worker/worker.py", line 126, in init_device
[rank0]:     init_worker_distributed_environment(self.parallel_config, self.rank,
[rank0]:   File "/miniconda/envs/vllama/lib/python3.9/site-packages/vllm/worker/worker.py", line 327, in init_worker_distributed_environment
[rank0]:     ensure_model_parallel_initialized(parallel_config.tensor_parallel_size,
[rank0]:   File "/miniconda/envs/vllama/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 915, in ensure_model_parallel_initialized
[rank0]:     initialize_model_parallel(tensor_model_parallel_size,
[rank0]:   File "/miniconda/envs/vllama/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 883, in initialize_model_parallel
[rank0]:     _TP = init_model_parallel_group(group_ranks,
[rank0]:   File "/miniconda/envs/vllama/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 729, in init_model_parallel_group
[rank0]:     return GroupCoordinator(
[rank0]:   File "/miniconda/envs/vllama/lib/python3.9/site-packages/vllm/distributed/parallel_state.py", line 175, in __init__
[rank0]:     self.pynccl_comm = PyNcclCommunicator(
[rank0]:   File "/miniconda/envs/vllama/lib/python3.9/site-packages/vllm/distributed/device_communicators/pynccl.py", line 89, in __init__
[rank0]:     self.comm: ncclComm_t = self.nccl.ncclCommInitRank(
[rank0]:   File "/miniconda/envs/vllama/lib/python3.9/site-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 244, in ncclCommInitRank
[rank0]:     self.NCCL_CHECK(self._funcs["ncclCommInitRank"](ctypes.byref(comm),
[rank0]:   File "/miniconda/envs/vllama/lib/python3.9/site-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 223, in NCCL_CHECK
[rank0]:     raise RuntimeError(f"NCCL error: {error_str}")
[rank0]: RuntimeError: NCCL error: unhandled cuda error (run with NCCL_DEBUG=INFO for details)

[youkaichao]

that's another error you need to check, following https://docs.vllm.ai/en/latest/getting_started/debugging.html

[cx-hub]

device_communicators/pynccl_wrapper.py", line 223, in NCCL_CHECK

i found why, i use cuda11.8, however the runtime cuda is 11.7.
but i can not change the runtime cuda version, so is there any method to solve it?

init.cc:1674 NCCL WARN Cuda failure 'CUDA driver version is insufficient for CUDA runtime version'

[cx-hub]

may i use torch 2.0.1 and vllm to offline the llama3 70B or qwen?

[youkaichao]

Nvidia driver version: 515.43.04

you need to contact your admin to see how to update the driver version.