Modal script - benchmarking, profiling and libraries about llm.c

[gordicaleksa]

I won't be able to help you with Modal, but I'll just say that our goal is ultimately to have custom CUDA kernels that outperform cuDNN. So you should, ideally, be able to just run our code without external dependencies.

One can dream. :)

[awayzjj]

@vyom1611 Hi, I try to run the demo

GPU_MEM=80 modal run benchmark_on_modal.py \
    --compile-command "nvcc -O3 --use_fast_math attention_forward.cu -o attention_forward -lcublas" \
    --run-command "./attention_forward 1

, but failed. Could you please give me some suggestions? Did I miss some steps? Thank you very much!

[vyom1611]

Hi, try running the compile command with the -lcublast option.

[awayzjj]

@vyom1611 Thank you very much! It worked after adding the -lcublasLt option.

nvcc -O3 --use_fast_math attention_forward.cu -o attention_forward -lcublas -lcublasLt

Have you considered adding support for "ncu"? I tried, but encountered an error and no profile file was generated.

[vyom1611]

using ncu is weird, because it likes to profile kernels very deeply. There is a linux_kernel_paranoid level if too high, then nsys cannot profile cpu and os parts during profiling, and for ncu: it seems impossible to fix this on modal containers: ERR_NVGPUCTRPERM The user running <tool_name/application_name> does not have permission to access NVIDIA GPU Performance Counters on the target device.

And to fix it you have to:

Enable access permanently

1. To allow access for any user: create a file with the .conf extension containing options nvidia NVreg_RestrictProfilingToAdminUsers=0 in /etc/modprobe.d.
2. To restrict access to admin users (CAP_SYS_ADMIN capability set), create a file with the .conf extension containing options nvidia NVreg_RestrictProfilingToAdminUsers=1 in /etc/modprobe.d.

which seems impossible on modal containers since you need root privilege to edit modprobe.d and create .conf files in /etc/.

and even if you managed to change it, then you have to reload modrobe using rmmod nvidia and update-initramfs -u which again needs sudo access, and reboot the system,

So currently we cannot run ncu on modal.

This is the link for reference: https://developer.nvidia.com/nvidia-development-tools-solutions-err_nvgpuctrperm-permission-issue-performance-counters

[awayzjj]

using ncu is weird, because it likes to profile kernels very deeply. There is a linux_kernel_paranoid level if too high, then nsys cannot profile cpu and os parts during profiling, and for ncu: it seems impossible to fix this on modal containers: ERR_NVGPUCTRPERM The user running <tool_name/application_name> does not have permission to access NVIDIA GPU Performance Counters on the target device.

And to fix it you have to:

Enable access permanently

1. To allow access for any user: create a file with the .conf extension containing options nvidia NVreg_RestrictProfilingToAdminUsers=0 in /etc/modprobe.d.
2. To restrict access to admin users (CAP_SYS_ADMIN capability set), create a file with the .conf extension containing options nvidia NVreg_RestrictProfilingToAdminUsers=1 in /etc/modprobe.d.

which seems impossible on modal containers since you need root privilege to edit modprobe.d and create .conf files in /etc/.

and even if you managed to change it, then you have to reload modrobe using rmmod nvidia and update-initramfs -u which again needs sudo access, and reboot the system,

So currently we cannot run ncu on modal.

This is the link for reference: https://developer.nvidia.com/nvidia-development-tools-solutions-err_nvgpuctrperm-permission-issue-performance-counters

Got it! Thank you! It is a pity since modal offer some free A100 quota :(