Performance drop using CVMFS through this plugin about cvmfs-csi

[amalthundiyil]

@MohammadAliAmir Thank you for opening this issue. I've tried the resource restriction on an example pod and a VM, and measured the times for ls /cvmfs/alice.cern.ch (first call). This is what I obtained:

real 0m0.22s, user 0m0.00s, sys 0m0.00s # csi
real 0m0.283s, user 0m0.002s, sys 0m0.000s # vm

Could you please let me know how I can reproduce the commands you mentioned locally? IIUC, uptime doesn't seem to be related to CVMFS.

[MohammadAliAmir]

Thanks for responding! uptime is indeed not related to cvmfs, thats why I was so confused that even non-related commands are executed slower. The other command ml is from Lmod, a little bit more complex to set up and I don't want you to go through this entire setup. I am simply looking for clues that could indicate why this is running slower. In the meantime, I also did compare ls from a container to a vm and I see quite normal response:

real 0m0.002s user 0m0.000s sys 0m0.002s # vm
real 0m0.006s user 0m0.001s sys 0m0.004s # container

After loading my profile for lmod which sets some environment variables correctly to start loading modules it still looks good

real 0m0.005s user 0m0.002s sys 0m0.003s

Whenever I load a module effectively, the ls of the repo really drops in performance. I've compared them here for a vm and the container

real 0m1.686s user 0m0.013s sys 0m0.077s # container
real 0m0.035s user 0m0.001s sys 0m0.014s # vm

So loading a module sets alot of environment variables, which in this case many will link to a path of the cvmfs repository.

[MohammadAliAmir]

I have been monitoring the resources for the pods deployed by the daemonset, apparrently it was CPU throttling with these assigned resources. Increasing the resources (upto a core for the CPU limit) on the relevant containers (automount and automount reconciler) did resolve my slow responsiveness from the CVMFS mount.

What I learned:
Depending on your workload, you will need to set the resources for the containers in the cvmfs-csi plugin accordingly.