Coder Social home page Coder Social logo

vespa-3rdparty-deps's Issues

CUDA in Vespa Docker Container

I am trying to use the onnxruntime with CUDA acceleration as described in this article.
However, after starting the docker container with the following Dockerfile:

FROM vespaengine/vespa

USER root

RUN dnf clean all

RUN dnf -y install 'dnf-command(config-manager)'
RUN dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-rhel8.repo

RUN dnf -y install $(rpm -q --queryformat '%{NAME}-cuda-%{VERSION}' vespa-onnxruntime)

USER vespa

I get the error message (35) that the CUDA driver version is insufficient. This is the log excerpt from vespa-logfmt:

[2024-07-10 10:15:07.835] WARNING container        Container.ai.djl.util.cuda.CudaUtils Failed to detect GPU count: CUDA driver version is insufficient for CUDA runtime version (35)
[2024-07-10 10:15:07.836] WARNING container        Container.ai.djl.util.cuda.CudaUtils Failed to detect GPU count: CUDA driver version is insufficient for CUDA runtime version (35)
[2024-07-10 10:15:07.836] INFO    container        Container.ai.djl.util.Platform       Found matching platform from: bundle://3c149058-7591-414a-a874-e9111ef9eadd_23.0:3/native/lib/tokenizers.properties
[2024-07-10 10:15:09.064] WARNING container        stderr       terminate called after throwing an instance of 'onnxruntime::OnnxRuntimeException'
[2024-07-10 10:15:09.064] WARNING container        stderr         what():  /builddir/build/BUILD/vespa-onnxruntime-1.18.0/onnxruntime/contrib_ops/cuda/bert/tensorrt_fused_multihead_attention/cudaDriverWrapper.cc:42 onnxruntime::contrib::cuda::CUDADriverWrapper::CUDADriverWrapper() handle != nullptr was false.

The container toolkit on the host is installed and CUDA is functional in other images.

(base) ➜  ~ docker run --rm -it --gpus=all pytorch/pytorch:2.2.1-cuda12.1-cudnn8-runtime nvidia-smi
Wed Jul 10 11:14:00 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.90.07              Driver Version: 550.90.07      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4090        Off |   00000000:01:00.0 Off |                  Off |
| 45%   49C    P8             22W /  450W |    6872MiB /  24564MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce RTX 4090        Off |   00000000:03:00.0 Off |                  Off |
|  0%   46C    P8             22W /  450W |    5164MiB /  24564MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
+-----------------------------------------------------------------------------------------+

I tried manually updating the CUDA drivers inside the vespa container but that prevented the vespa application from starting.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.