Manual Model Execution Error: Troubleshooting GPU Acceleration about localai

[localai-bot]

Hello WuQic, I am an experiment of @mudler and I am here to help you with your issue. I apologize for any inconvenience caused.

Upon reviewing the provided error logs and your configuration details, it seems there might be an issue with the LocalAI configuration or the setup of your GPU. Here are a few points to consider:

1. GPU Setup: It appears that your GPU, NVIDIA GeForce RTX 4090, is recognized by the system. However, it's recommended to ensure that your GPU is compatible with the required CUDA version (12.2) and the LocalAI dependencies. You can verify your GPU compatibility by checking the manufacturer's specifications.

2. Missing Configuration Directory: The error log indicates that there is no LocalAI Configuration Directory, which may cause the configuration directory watcher to fail. Make sure you have created the necessary configuration files in the expected directory (/root/autodl-tmp/local-ai/models). If you have created them, it is possible that they are not named correctly or are located in a different directory.

3. Loading Models with Backends: The error log shows that the models are not being loaded successfully with the specified backends. To troubleshoot this issue, you can try the following:

   • Check the model compatibility with the selected backends. Ensure that the models and backends are properly configured and matched.
   • Verify that the required dependencies for the backends (e.g., stablediffusion, tinydream, piper) are correctly installed and configured.

4. GPU-related Dependencies: The error log also mentions a problem with grpc processes being unavailable for some backends. This could be due to missing dependencies or misconfigured setups. Ensure that all required dependencies for LocalAI and the backends are properly installed.

To resolve the issue, you can try the following steps:

• Verify that your GPU is compatible with the required CUDA version, and all necessary GPU-related dependencies are installed.
• Ensure that the LocalAI Configuration Directory exists and contains the required configuration files.
• Double-check the model compatibility with the selected backends and ensure that the backends' dependencies are correctly configured.
• If the issue persists, please share more details about your system (e.g., operating system, LocalAI version, etc.) for further assistance.

I hope this information is helpful. Let me know if you have any further questions or concerns.

[WuQic]

Could you please share an example demonstrating how to initiate deployment using LocalAI binary + GPU?

[joseluisll]

Can you activate debug log level? Debug messages will help in determining the reason why the backends fail to load the models. Can you paste a debug log? If you do that, you can see is the gpu is used, and how many layers the model and MB of ram is trying to allocate in the GPU... Lunch the binary with --debug

[WuQic]

I solved it with the latest version:

# create dir
mkdir /opt/apps/local-ai /opt/apps/local-ai/models /opt/apps/local-ai/configuration
cd /opt/apps/local-ai
# Download CUDA 12
wget https://github.com/mudler/LocalAI/releases/download/v2.15.0/local-ai-cuda12-Linux-x86_64
# chmod
chmod +x local-ai-cuda12-Linux-x86_64

# run
LOCALAI_LOG_LEVEL=debug
ADDRESS=":6006"
./local-ai-cuda12-Linux-x86_64