Deepspeed on standalone app? about alltalk_tts

[erew123]

Hiya

DeepSpeed needs to be able to find the CUDA toolkit's nvcc to compile. DeepSpeed is not a pre-built bit of code, It actually compiles the code unique to your setup on Linux and it needs to be able to use nvcc --version to confirm the version of CUDA its compiling for. Once it has installed/compiled I believe you wont need to keep the Nvidia toolkit anymore (unless you reinstalled DeepSpeed).

So you would load your custom environment. You can use pip show torch if you need to confirm what version of CUDA you may have installed e.g. cu118 (cuda 11.8) or cu121 (cuda 12.1) etc

Either use your package manager to install the Nvidia CUDA toolkit or alternatively https://developer.nvidia.com/cuda-toolkit-archive (Matching the version of Torch's cuda version).

Once that's installed, confirm the path that CUDA is in... it may be /etc/alternative/cuda (wherever your Linux puts nvcc)

Load your python environment

Set the environment path for CUDA_HOME e.g. export CUDA_HOME=/usr/local/cuda-12.1/bin or export CUDA_HOME=/etc/alternative/cuda

Then you should be able to type nvcc --version which should work and tell you the CUDA version. If it does, go ahead and pip install deepspeed and you are all sorted.

Remove the Nvidia toolkit (unless you want to keep it). Start AllTalk and you will need to activate DeepSpeed in the interface.

That's a loose guide.

[erew123]

Bar starting your Python environment, Steps 1, 3, 7 and 8 are the specific steps. (obviously matching the version of CUDA you have installed to the version of Pytorch).

https://github.com/erew123/alltalk_tts?tab=readme-ov-file#-linux-installation

[q5sys]

Thanks for the response. I'll poke around with it a bit tonight. Assuming I dont run into any complexities introduced by my system, if I can get it working I'll put together a little text block about the steps I went through and file a PR to add that to the docs, so the next person who wants to do this has a clear section dedicated to the stand alone app.
Since I'm using your native UI, I wasn't sure how much I should rely on the instructions for someone using oobabooga.

[q5sys]

So I got it installed, interestingly I ran into a bit of an odd situation.
After setting the env when I tried to install deepspeed, I got the following error:
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/cuda-12.1/bin/bin/nvcc'

I dont know why it was doubling up the bin dir, but I figured instead of altering the path and potentially causing an issue for something else, I'd just set the path as part of PATH since I'll probably be using the CUDA toolkit for other things in the future.

I'm not sure if this was just an oddity about my system (Rocky 9.3) or not.

In either case I got it working so I'm going to close this ticket. Thanks for the help. PR filed with instructions submitted.