couldn't allocate MR while test GDR with cuda. about perftest

[derekwin]

sry that i didn't notice this suggestion.
8. If GPUDirect is not working, (e.g. you see "Couldn't allocate MR" error message), consider disabling Scatter to CQE feature. Set the environmental variable MLX5_SCATTER_TO_CQE=0. E.g.:
MLX5_SCATTER_TO_CQE=0 ./ib_write_bw -d ib_dev --use_cuda= -a

[derekwin]

after setting MLX5_SCATTER_TO_CQE=0, the problem still exist.

[Jye-525]

@derekwin Did you solve this problem? I encountered the same error now.

[derekwin]

@derekwin Did you solve this problem? I encountered the same error now.

i still have this error. : (

[Jye-525]

@derekwin I just solved it on my side. I test this across 2 nodes, but I only load the nvidia-peermem on one of the nodes. It is solved by loading nvidia-peermem on all the nodes. Here is the command:
sudo modprobe nvidia-peermem

You can use lsmod|grep nvidia_peermem to check if it is loaded. Hope it works for you too.

[YuMJie]

If only one of the two machines is capable of supporting GDR, and the other is a consumer-grade graphics card, how should the bandwidth of GDR be tested?