jcarreira / disag-firebox Goto Github PK

Dear Sir!

We implement a rdma client at running linux kernel. We found your project. Thanks for your job.

I know that a IB Client for kernel module must call ib_register_client. I could found it from the codes.

The client_module_init() functions never return, But instead endless loop.

If your code is running as successful, you must modify it. I hope so your modification ASAP.

Thanks.

Hello Joao,

I saw your question https://stackoverflow.com/questions/44190665/rdma-between-gpu-and-remote-host. Did you get the answer? I got the normal RDMA mechanics to work with queue pairs, now I'm trying to send data directly to a memory allocated by cudaMalloc so the data would get directly to GPU. Not sure how to do it .

Hello,

You code does not seem to work on the kernel version mentioned in the comments. Could you please recheck the kernel you tested this code on?

Regards

s_ctx.pd = ib_alloc_pd(dev);
CHECK_MSG2(s_ctx.pd != 0, "Error creating pd");

s_ctx.send_cq = ib_create_cq(dev, comp_handler_send, cq_event_handler_send, NULL, 10, 0);
s_ctx.recv_cq = ib_create_cq(dev, comp_handler_recv, cq_event_handler_recv, NULL, 10, 0);
CHECK_MSG2(s_ctx.send_cq != 0, "Error creating CQ");
CHECK_MSG2(s_ctx.recv_cq != 0, "Error creating CQ");

printk(KERN_WARNING "ib_create_cq succ\n");


CHECK2(ib_req_notify_cq(s_ctx.recv_cq, IB_CQ_NEXT_COMP) == 0);_
CHECK2(ib_req_notify_cq(s_ctx.send_cq, IB_CQ_NEXT_COMP) == 0);


//printk(KERN_WARNING "ib_req_notify_cq\n");
// initialize qp_attr
memset(&s_ctx.qp_attr, 0, sizeof(struct ib_qp_init_attr));
s_ctx.qp_attr.send_cq = s_ctx.send_cq;
s_ctx.qp_attr.recv_cq = s_ctx.recv_cq;
s_ctx.qp_attr.cap.max_send_wr  = 10;
s_ctx.qp_attr.cap.max_recv_wr  = 10;
s_ctx.qp_attr.cap.max_send_sge = 1;
s_ctx.qp_attr.cap.max_recv_sge = 1;
s_ctx.qp_attr.cap.max_inline_data = 0;
s_ctx.qp_attr.qp_type = IB_QPT_RC;
s_ctx.qp_attr.sq_sig_type = IB_SIGNAL_ALL_WR;

ERROR : CHECK2(ib_req_notify_cq(s_ctx.recv_cq, IB_CQ_NEXT_COMP) == 0);

[ 95.131893] max_res_rd_atom: 16
[ 95.132512] ib_create_cq succ

[ 95.132923] BUG: unable to handle kernel NULL pointer dereference at (null)
[ 95.132929] IP: [] __mutex_lock_slowpath+0xb0/0x1d0
[ 95.132939] PGD 0
[ 95.132942] Oops: 0002 [#1] SMP**
[ 95.132946] Modules linked in: client(OE+) target_core_mod(E) ib_iser(OE) iscsi_tcp(E) libiscsi_tcp(E) libiscsi(E) scsi_transport_iscsi(E) x86 pkg_temp_thermal(E) intel_powerclamp(E) coretemp(E) kvm_intel(E) kvm(E) crct10dif_pclmul(E) joydev(E) crc32_pclmul(E) aesni_intel(E) aes_x86_64( E) ipmi_devintf(E) lrw(E) gf128mul(E) glue_helper(E) sb_edac(E) ablk_helper(E) edac_core(E) cryptd(E) mei_me(E) mei(E) ipmi_si(E) ipmi_msghandler (E) acpi_power_meter(E) dcdbas(E) acpi_pad(E) wmi(E) shpchp(E) lpc_ich(E) mac_hid(E) knem(OE) lp(E) parport(E) binfmt_misc(E) rdma_ucm(OE) ib_ucm (OE) rdma_cm(OE) iw_cm(OE) configfs(E) ib_ipoib(OE) ib_cm(OE) ib_uverbs(OE) ib_umad(OE) mlx5_ib(OE) mlx5_core(OE) mlx4_ib(OE) ib_sa(OE) ib_mad(OE ) ib_core(OE) ib_addr(OE) ib_netlink(OE) mlx4_en(OE) vxlan(E) ses(E) enclosure(E) hid_generic(E) ahci(E) usbhid(E) hid(E) libahci(E) tg3(E) mlx4 core(OE) ptp(E) pps_core(E) mlx_compat(OE) megaraid_sas(E)