Unable to load module about ksmbd

[namjaejeon]

Have you tried load mlx and rdma core driver using modprobe ?
sudo modprobe mlx5_core
sudo modprobe mlx5_ib

[llamafilm]

Yes, those are loaded already

# lsmod | grep mlx
mlx5_ib               446464  0
ib_uverbs             139264  2 rdma_ucm,mlx5_ib
ib_core               425984  8 rdma_cm,ib_ipoib,iw_cm,ib_umad,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm
mlx5_core            2015232  1 mlx5_ib
mlxdevm               176128  1 mlx5_core
mlxfw                  32768  1 mlx5_core
psample                20480  1 mlx5_core
tls                   114688  1 mlx5_core
mlx_compat             69632  11 rdma_cm,ib_ipoib,mlxdevm,iw_cm,ib_umad,ib_core,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm,mlx5_core

[llamafilm]

BTW, are there any benefits to installing it as part of the kernel instead of as a module?

[namjaejeon]

these are ksmbd module dependancy.
Please try to insmod these drivers.
kernel/fs/ksmbd/ksmbd.ko: kernel/drivers/infiniband/core/rdma_cm.ko kernel/drivers/infiniband/core/iw_cm.ko kernel/drivers/infiniband/core/ib_cm.ko kernel/drivers/infiniband/core/ib_core.ko kernel/lib/crypto/libdes.ko

[namjaejeon]

BTW, are there any benefits to installing it as part of the kernel instead of as a module?

You don't need to insmod kernel module. but kernel image size is increased. So if you don't need to use this module, you can not unload it because it is built-in kernel. If HW has short of memory, Memory usage increases due to the large kernel image size including unused modules.

[llamafilm]

I checked, and those 5 modules are already loaded.

[namjaejeon]

Or you can command in ksmbd source.

sudo make install
sudo modprobe ksmbd

[  201.906576] ksmbd: module verification failed: signature and/or required key missing - tainting kernel
[  201.906612] ksmbd: Unknown symbol ib_mr_pool_destroy (err -2)
[  201.906642] ksmbd: Unknown symbol ib_unregister_client (err -2)
[  201.906667] ksmbd: Unknown symbol rdma_rw_ctx_destroy (err -2)
[  201.906674] ksmbd: Unknown symbol rdma_event_msg (err -2)
[  201.906700] ksmbd: Unknown symbol __ib_alloc_pd (err -2)
[  201.906740] ksmbd: Unknown symbol ib_mr_pool_init (err -2)
[  201.906784] ksmbd: Unknown symbol ib_event_msg (err -2)
[  201.906811] ksmbd: Unknown symbol rdma_disconnect (err -2)
[  201.906858] ksmbd: Unknown symbol __rdma_create_kernel_id (err -2)
[  201.906890] ksmbd: Unknown symbol ib_device_get_by_netdev (err -2)
[  201.906898] ksmbd: Unknown symbol rdma_bind_addr (err -2)
[  201.906905] ksmbd: Unknown symbol ib_register_client (err -2)
[  201.906917] ksmbd: Unknown symbol rdma_create_qp (err -2)
[  201.906930] ksmbd: Unknown symbol rdma_rw_ctx_wrs (err -2)
[  201.906972] ksmbd: Unknown symbol __ib_alloc_cq (err -2)
[  201.906990] ksmbd: Unknown symbol rdma_listen (err -2)
[  201.907013] ksmbd: Unknown symbol ib_destroy_qp_user (err -2)
[  201.907022] ksmbd: Unknown symbol des_encrypt (err -2)
[  201.907050] ksmbd: Unknown symbol ib_destroy_cq_user (err -2)
[  201.907067] ksmbd: Unknown symbol des_expand_key (err -2)
[  201.907078] ksmbd: Unknown symbol ib_wc_status_msg (err -2)
[  201.907091] ksmbd: Unknown symbol ib_dma_virt_map_sg (err -2)
[  201.907107] ksmbd: Unknown symbol ib_free_cq (err -2)
[  201.907116] ksmbd: Unknown symbol rdma_destroy_id (err -2)
[  201.907134] ksmbd: Unknown symbol rdma_accept (err -2)
[  201.907150] ksmbd: Unknown symbol ib_drain_qp (err -2)
[  201.907156] ksmbd: Unknown symbol ib_dealloc_pd_user (err -2)
[  201.907175] ksmbd: Unknown symbol ib_device_put (err -2)
[  201.907200] ksmbd: Unknown symbol rdma_rw_ctx_init (err -2)
linkinjeon@linkinjeon-Precision-Tower-3620:~/git/smbd_work/ksmbd$ vi Makefile 
linkinjeon@linkinjeon-Precision-Tower-3620:~/git/smbd_work/ksmbd$ sudo make install
rm -f /lib/modules/5.15.0-46-generic/kernel/fs/ksmbd/ksmbd.ko
install -m644 -b -D ksmbd.ko /lib/modules/5.15.0-46-generic/kernel/fs/ksmbd/ksmbd.ko
depmod -a
linkinjeon@linkinjeon-Precision-Tower-3620:~/git/smbd_work/ksmbd$ sudo modprobe ksmbd
linkinjeon@linkinjeon-Precision-Tower-3620:~/git/smbd_work/ksmbd$ lsmod | grep ksmbd
ksmbd                 270336  0
rdma_cm               122880  1 ksmbd
ib_core               393216  4 rdma_cm,iw_cm,ksmbd,ib_cm
libdes                 32768  1 ksmbd

[llamafilm]

That's essentially what I've been doing already. Which driver version do you use? Do you think there is some compatibility issue with Mellanox 5.7 driver version?

[namjaejeon]

No, I have ever installed Mellanox drivers before. ksmbd RDMA work fine to me on my ubuntu although Mellanox driver is not installed. Because there are mellanox drivers in linux kernel mainline. When I see your debug log, Errors are coming from rdma core driver, not mellanox driver. I can reproduce it when I just command "insmod ksmbd" without make install. and work with commanding "make install" and modprobe ksmbd to insmod rdma core drivers.

[namjaejeon]

Can you try to "make clean" and "make" again in ksmbd source ?

[llamafilm]

I tried make clean; git pull; make; make install, that didn't change the outcome.
I tried uninstalling the Mellanox driver but then the NICs disappear completely. Maybe I'll try a slightly older driver version.

[namjaejeon]

@llamafilm Okay:)

[hclee]

It seems the mellanox driver package installed rdma_cm, ib_core, ...
Can you run modinfo ib_core and share the result?

[namjaejeon]

Mellanox 5.7 mean kernel version(i.e. linux 5.7 kernel) ? Maybe, there is linux 5.7 kernel source in mellanox package ?

[hclee]

Mellanox 5.7 mean kernel version(i.e. linux 5.7 kernel) ? Maybe, there is linux 5.7 kernel source in mellanox package ?

I don't know the meaning of the version, but there is all code of drivers/infiniband subdirectories in the package.
And it seems that messages below means the symbol exists but the CRC checksums are different.

[58086.240531] ksmbd: disagrees about version of symbol ib_mr_pool_destroy
[58086.240535] ksmbd: Unknown symbol ib_mr_pool_destroy (err -22)

[hclee]

Maybe have to compile ksmbd against rdma modules provided by the mellanox driver.

[namjaejeon]

In case of Chelsio driver package, The package contains the kernel source corresponding to the driver. So users need to build and install the kernel in package after installing the packages.

[llamafilm]

I don't think 5.7 refers to any kernel, it's just Mellanox own versioning number system. https://network.nvidia.com/products/ethernet-drivers/linux/mlnx_en/

@hclee you're right, the driver does include ib_core and some others. Here's a comparison of ib_core with inbox driver vs the latest. I'm not sure how to compile ksmbd against these other rdma modules.

with inbox driver

# modinfo ib_core
filename:       /lib/modules/5.15.0-46-generic/kernel/drivers/infiniband/core/ib_core.ko
alias:          rdma-netlink-subsys-4
license:        Dual BSD/GPL
description:    core kernel InfiniBand API
author:         Roland Dreier
alias:          net-pf-16-proto-20
alias:          rdma-netlink-subsys-5
srcversion:     90657E3789C1E7B0413615C
depends:        
retpoline:      Y
intree:         Y
name:           ib_core
vermagic:       5.15.0-46-generic SMP mod_unload modversions 
sig_id:         PKCS#7
signer:         Build time autogenerated kernel key
sig_key:        7D:17:41:79:9B:FD:BB:C4:C9:58:54:28:ED:61:EB:C2:E4:6C:2E:06
sig_hashalgo:   sha512
signature:      52:D1:3F:C7:2C:46:88:AB:98:D9:29:41:BB:44:EC:0F:75:53:95:48:
		7A:F5:9A:1B:1D:E9:B8:7D:A7:0C:77:A5:FC:6E:1B:AE:E6:30:B4:0A:
		DB:A6:A7:06:E2:D2:EC:BA:AC:B3:32:60:90:30:A8:59:91:0E:CB:8E:
		76:30:75:A5:19:FA:EF:AE:88:65:85:E7:F2:0E:52:FB:2F:7C:B3:88:
		E8:56:98:4D:5B:9F:81:FE:6D:FB:EF:47:A1:79:BF:16:5A:F8:5E:D4:
		74:A5:54:B4:1F:70:8E:AE:E3:DE:D2:7F:90:74:33:FC:4D:CA:19:8C:
		BD:7F:43:CD:C4:91:0B:0F:4A:E2:40:46:0C:49:C8:BE:EC:BC:AE:C6:
		92:54:35:52:17:46:B8:E8:A2:31:6B:3E:44:9B:FB:BA:DD:9C:46:38:
		8A:D7:EB:48:1B:C5:47:5E:F6:B1:DD:8C:79:01:27:AC:90:43:CD:AF:
		D3:0B:DA:F9:E3:37:90:6B:A7:D4:46:70:6B:3D:C7:D4:6A:2F:E0:CC:
		FA:B4:B7:B0:D5:F7:F9:7E:B2:92:71:99:C6:36:3D:60:39:A3:07:C9:
		1B:04:86:CD:2F:0C:55:38:93:94:C0:EF:FA:DD:51:B3:33:D8:2F:2B:
		E7:16:92:7E:DA:AE:F1:E3:A1:98:CD:AB:3F:84:57:AD:D1:89:83:30:
		42:EA:2F:4E:B4:FB:93:A7:EF:34:7E:3F:87:CF:5C:F0:66:BC:8B:66:
		89:C4:97:9A:E8:F1:7B:BD:F6:30:AE:96:DB:F4:DD:5D:D9:FD:EE:77:
		BA:F0:E0:CC:C8:26:B3:4A:F9:AC:E3:0C:5D:BE:86:2F:6E:35:FC:EE:
		11:A0:4E:C9:67:73:C7:73:79:34:1B:F6:C8:5F:DE:A4:6A:23:DB:98:
		BE:4C:A7:98:10:4D:E4:45:BE:9C:12:C8:0D:5B:94:F2:8F:2B:68:E7:
		6A:01:3B:E3:9F:53:27:A4:CE:AA:53:E8:55:55:25:60:1D:0F:19:F7:
		46:F0:70:30:AA:0F:AF:83:89:E2:95:3B:71:A1:14:8A:5A:EA:A6:98:
		1D:00:9B:7C:9E:E2:F3:90:C7:CE:2E:27:9C:22:FB:3A:E0:9E:EA:83:
		B4:5C:73:30:68:9C:7F:92:83:12:C5:95:FD:C2:3E:F7:12:E8:5D:FE:
		82:C1:0A:BC:F6:7C:93:40:47:4C:4F:F2:9C:5F:4E:F5:26:E8:01:FE:
		6E:55:F2:0F:C1:2F:7A:CB:07:C5:B4:AB:E6:A4:9F:4E:5E:3C:95:51:
		80:44:CF:6F:24:2F:FC:31:9C:C8:38:3A:26:59:EC:FB:86:C9:90:63:
		90:1E:AD:75:92:21:2E:7D:8E:59:20:0E
parm:           send_queue_size:Size of send queue in number of work requests (int)
parm:           recv_queue_size:Size of receive queue in number of work requests (int)
parm:           netns_mode:Share device among net namespaces; default=1 (shared) (bool)
parm:           force_mr:Force usage of MRs for RDMA READ/WRITE operations (bool)

with MLNX-EN --vma

# modinfo ib_core
filename:       /lib/modules/5.15.0-46-generic/updates/dkms/ib_core.ko
alias:          rdma-netlink-subsys-4
license:        Dual BSD/GPL
description:    core kernel InfiniBand API
author:         Roland Dreier
alias:          net-pf-16-proto-20
alias:          rdma-netlink-subsys-5
import_ns:      DMA_BUF
srcversion:     162C74CC0553CF4D9E2C6F0
depends:        mlx_compat
retpoline:      Y
name:           ib_core
vermagic:       5.15.0-46-generic SMP mod_unload modversions 
parm:           send_queue_size:Size of send queue in number of work requests (int)
parm:           recv_queue_size:Size of receive queue in number of work requests (int)
parm:           mad_smp_window:Maximun number of outgoing SMP requests (int)
parm:           netns_mode:Share device among net namespaces; default=1 (shared) (bool)
parm:           force_mr:Force usage of MRs for RDMA READ/WRITE operations (bool)

[hclee]

@llamafilm
I have never compiled the kernel module against others. But modules.txt may help you.

	Use an extra Module.symvers file
		When an external module is built, a Module.symvers file
		is generated containing all exported symbols which are
		not defined in the kernel. To get access to symbols
		from bar.ko, copy the Module.symvers file from the
		compilation of bar.ko to the directory where foo.ko is
		built. During the module build, kbuild will read the
		Module.symvers file in the directory of the external
		module, and when the build is finished, a new
		Module.symvers file is created containing the sum of
		all symbols defined and not part of the kernel.

	Use "make" variable KBUILD_EXTRA_SYMBOLS
		If it is impractical to copy Module.symvers from
		another module, you can assign a space separated list
		of files to KBUILD_EXTRA_SYMBOLS in your build file.
		These files will be loaded by modpost during the
		initialization of its symbol tables.

[consp]

I never got it working with the Mellanox drivers. You miss at minimum several function references which are not (correctly) ported from mainline to the Mellanox drivers and until Mellanox or Nvidia fixes this/adds the missing functions it is almost impossible to get working. I also tried making function aliases to avoid these issues but every fix gave me two more errors and I gave up and used the inbox kernel drivers.

You might want to try sending a bug report to Mellanox since they claim kernel 5.15+ support now.