Custom kernel not loading on UDM about udm-kernel-tools

[fabianishere]

You might find this project useful: https://github.com/tusc/wireguard-kmod
That projects builds WireGuard as kernel module for the stock kernel running on the UDM(P), which might be the easier option for your use-case.

[chriscpritchard]

Hi,

After a bit of a nightmare building an ancient toolchain, cross compiling using:

 * binutils:            binutils-2.29.1
 * gcc:                  gcc-6.4.0
 * headers:           linux-headers-4.19
 * libc:                  glibc-2.26

Has worked using the GPL sources and config released by unifi and manually compiling (I also used /proc/config.gz from my UDM and added forced module loading as the only config option), I use gentoo (so used crossdev and manually created ebuilds for the old versions), but ymmv with other operating systems.

I'm going to do a bit more experimentation to see if I can get the edge config options to work.

gcc has this patch applied:
(allows it to build when glibc-2.26 is used), otherwise everything is stock or has gentoo patches applied (glibc was built with USE=vanilla as there wasn't a gentoo patchset for that version).

To work with gentoo's naming conventions, CROSS_COMPILE in build.sh and prepare.sh was changed to CROSS_COMPILE=aarch64-unknown-linux-gnu-

and dpkg-buildpackage was run using the -d option (doesn't check for dependencies, as gentoo allows dpkg to be installed but doesn't use it as a package manager) by changing line 80 in scripts/package/Makefile to:
+dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) -d -b -nc -uc

If the github runner could use the same versions of binutils, gcc and glibc as the kernel was compiled with I'm hoping it shouldn't cause any incompatibilities with binary kernel modules.

the version of GCC that was used can be found by running cat /proc/version on the UDM

# cat /proc/version
Linux version 4.19.152-al-linux-v10.2.0-v1.11.0.3921-f2e3fac (builder@link-xenial-builder) (gcc version 6.4.0 (Buildroot 0.9.0.0)) #1 SMP Tue Dec 21 08:05:39 UTC 2021

GLIBC by executing /lib64/libc.so.6 on the UDM (the .6 might change to a different number in future versions:

# /lib64/libc.so.6
GNU C Library (Buildroot) stable release version 2.26, by Roland McGrath et al.
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 6.4.0.
Available extensions:
        crypt add-on version 2.1 by Michael Glad and others
        GNU Libidn by Simon Josefsson
        Native POSIX Threads Library by Ulrich Drepper et al
        BIND-8.2.3-T5B
libc ABIs: UNIQUE
For bug reporting instructions, please see:
<http://www.gnu.org/software/libc/bugs.html>.

Binutils was in the buildroot.config file from the udm GPL archive: BR2_BINUTILS_VERSION="2.29.1"

[fabianishere]

I have had someone else having a similar issue with the UDM Base model due to the drivers for the WiFi hardware not working with the custom kernel you have installed. To verify this, could you please run the following command from UbiOS and not UniFi OS (this means SSHing directly into the UDM, you can verify it by checking the unifi-os command exists):

cat /sys/fs/pstore/*

What is your use-case for installing the custom kernel? In this other case, the user only needed IPTV, so I compiled a custom kernel with multicast routing enabled, which did work for him.
However, right now, if you need multicast routing, my advice would be to install the new beta firmware v1.11.0-14 which provides that functionality in the stock kernel.

[sjtuross]

Thank you for the quick response. My use case is to run wireguard in kernel mode. See below for the log.

# cat /sys/fs/pstore/*
[    0.587362] pci-pf-stub 0000:00:04.0: writing to VF config space
[    0.587426] pci-pf-stub 0000:00:05.0: writing to VF config space
[    0.611130] ahci 0001:00:00.0: writing to VF config space
[    0.616638] ahci 0002:00:00.0: writing to VF config space
[    0.622143] ahci 0003:00:00.0: writing to VF config space
[    0.629707] al_eth 0000:00:01.0: writing to VF config space
[    2.604808] al_eth 0000:00:03.0: writing to VF config space
[ 1743.912890] reboot: pre-cmd=01234567
[ 1743.916555] reboot:   called by: [10522]reboot
[ 1743.921114] reboot:   called by: [ 6065]-sh
[ 1743.925469] reboot:   called by: [ 5962]/usr/sbin/dropbear -R -b /etc/issue
[ 1743.935699] reboot:   called by: [ 2011]/usr/sbin/dropbear -R -b /etc/issue
[ 1743.942723] reboot:   called by: [    1]init
[ 1743.947076] reboot:   called by: [    0]
[ 1751.452531] reboot: cmd=01234567
[ 1751.455810] reboot:   called by: [11523]init
[ 1751.460137] reboot:   called by: [    1]init
[ 1751.464434] reboot:   called by: [    0]
[ 1751.499900] reboot: Restarting system

No errors detected
ttingInitialize+0xf4/0x6b0 [mt_wifi],Valid:1,Lock:ffffff800d7cdae8,State:1
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.448135] RTMP_AllTimerListRelease: Cancel timer obj ffffff800d5aa9d0!
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.448178] RTMP_AllTimerListRelease: Timer is allocated by RRM_CfgInit+0x84/0x120 [mt_wifi],Valid:1,Lock:ffffff800d7cdae8,State:0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.448180] RTMP_AllTimerListRelease: Cancel timer obj ffffff800d5aaa28!
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.448220] RTMP_AllTimerListRelease: Timer is allocated by RRM_CfgInit+0x9c/0x120 [mt_wifi],Valid:1,Lock:ffffff800d7cdae8,State:0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.448224] RTMP_AllTimerListRelease: Cancel timer obj ffffff800d5b17c0!
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.448264] RTMP_AllTimerListRelease: Timer is allocated by RRM_CfgInit+0x84/0x120 [mt_wifi],Valid:1,Lock:ffffff800d7cdae8,State:0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.448267] RTMP_AllTimerListRelease: Cancel timer obj ffffff800d5b1818!
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.448306] RTMP_AllTimerListRelease: Timer is allocated by RRM_CfgInit+0x9c/0x120 [mt_wifi],Valid:1,Lock:ffffff800d7cdae8,State:0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.448310] RTMP_AllTimerListRelease: Cancel timer obj ffffff800d5b85b0!
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.448350] RTMP_AllTimerListRelease: Timer is allocated by RRM_CfgInit+0x84/0x120 [mt_wifi],Valid:1,Lock:ffffff800d7cdae8,State:0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.448352] RTMP_AllTimerListRelease: Cancel timer obj ffffff800d5b8608!
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.448392] RTMP_AllTimerListRelease: Timer is allocated by RRM_CfgInit+0x9c/0x120 [mt_wifi],Valid:1,Lock:ffffff800d7cdae8,State:0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.448395] RTMP_AllTimerListRelease: Cancel timer obj ffffff800d5bf3a0!
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.448436] RTMP_AllTimerListRelease: Timer is allocated by RRM_CfgInit+0x84/0x120 [mt_wifi],Valid:1,Lock:ffffff800d7cdae8,State:0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.448438] RTMP_AllTimerListRelease: Cancel timer obj ffffff800d5bf3f8!
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.448478] RTMP_AllTimerListRelease: Timer is allocated by RRM_CfgInit+0x9c/0x120 [mt_wifi],Valid:1,Lock:ffffff800d7cdae8,State:0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.448482] RTMP_AllTimerListRelease: Cancel timer obj ffffff800d5c6190!
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.448523] RTMP_AllTimerListRelease: Timer is allocated by RRM_CfgInit+0x84/0x120 [mt_wifi],Valid:1,Lock:ffffff800d7cdae8,State:0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.448525] RTMP_AllTimerListRelease: Cancel timer obj ffffff800d5c61e8!
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.448566] RTMP_AllTimerListRelease: Timer is allocated by RRM_CfgInit+0x9c/0x120 [mt_wifi],Valid:1,Lock:ffffff800d7cdae8,State:0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.448568] RTMP_AllTimerListRelease: Cancel timer obj ffffff800d5ccf80!
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.448608] RTMP_AllTimerListRelease: Timer is allocated by RRM_CfgInit+0x84/0x120 [mt_wifi],Valid:1,Lock:ffffff800d7cdae8,State:0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.448612] RTMP_AllTimerListRelease: Cancel timer obj ffffff800d5ccfd8!
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.448652] RTMP_AllTimerListRelease: Timer is allocated by RRM_CfgInit+0x9c/0x120 [mt_wifi],Valid:1,Lock:ffffff800d7cdae8,State:0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.448654] RTMP_AllTimerListRelease: Cancel timer obj ffffff800d5d3d70!
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.448694] RTMP_AllTimerListRelease: Timer is allocated by RRM_CfgInit+0x84/0x120 [mt_wifi],Valid:1,Lock:ffffff800d7cdae8,State:0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.448697] RTMP_AllTimerListRelease: Cancel timer obj ffffff800d5d3dc8!
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.448738] RTMP_AllTimerListRelease: Timer is allocated by RRM_CfgInit+0x9c/0x120 [mt_wifi],Valid:1,Lock:ffffff800d7cdae8,State:0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.448740] RTMP_AllTimerListRelease: Cancel timer obj ffffff800d5dab60!
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.448780] RTMP_AllTimerListRelease: Timer is allocated by RRM_CfgInit+0x84/0x120 [mt_wifi],Valid:1,Lock:ffffff800d7cdae8,State:0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.448783] RTMP_AllTimerListRelease: Cancel timer obj ffffff800d5dabb8!
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.448823] RTMP_AllTimerListRelease: Timer is allocated by RRM_CfgInit+0x9c/0x120 [mt_wifi],Valid:1,Lock:ffffff800d7cdae8,State:0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.448825] RTMP_AllTimerListRelease: Cancel timer obj ffffff800d5e1950!
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.448865] RTMP_AllTimerListRelease: Timer is allocated by RRM_CfgInit+0x84/0x120 [mt_wifi],Valid:1,Lock:ffffff800d7cdae8,State:0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.448869] RTMP_AllTimerListRelease: Cancel timer obj ffffff800d5e19a8!
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.448909] RTMP_AllTimerListRelease: Timer is allocated by RRM_CfgInit+0x9c/0x120 [mt_wifi],Valid:1,Lock:ffffff800d7cdae8,State:0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.448911] RTMP_AllTimerListRelease: Cancel timer obj ffffff800d5e8740!
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.448952] RTMP_AllTimerListRelease: Timer is allocated by RRM_CfgInit+0x84/0x120 [mt_wifi],Valid:1,Lock:ffffff800d7cdae8,State:0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.448956] RTMP_AllTimerListRelease: Cancel timer obj ffffff800d5e8798!
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.448997] RTMP_AllTimerListRelease: Timer is allocated by RRM_CfgInit+0x9c/0x120 [mt_wifi],Valid:1,Lock:ffffff800d7cdae8,State:0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.448999] RTMP_AllTimerListRelease: Cancel timer obj ffffff800d5ef530!
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.449039] RTMP_AllTimerListRelease: Timer is allocated by RRM_CfgInit+0x84/0x120 [mt_wifi],Valid:1,Lock:ffffff800d7cdae8,State:0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.449042] RTMP_AllTimerListRelease: Cancel timer obj ffffff800d5ef588!
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.449082] RTMP_AllTimerListRelease: Timer is allocated by RRM_CfgInit+0x9c/0x120 [mt_wifi],Valid:1,Lock:ffffff800d7cdae8,State:0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.449084] RTMP_AllTimerListRelease: Cancel timer obj ffffff800d5f6320!
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.449124] RTMP_AllTimerListRelease: Timer is allocated by RRM_CfgInit+0x84/0x120 [mt_wifi],Valid:1,Lock:ffffff800d7cdae8,State:0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.449128] RTMP_AllTimerListRelease: Cancel timer obj ffffff800d5f6378!
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.449168] RTMP_AllTimerListRelease: Timer is allocated by RRM_CfgInit+0x9c/0x120 [mt_wifi],Valid:1,Lock:ffffff800d7cdae8,State:0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.449170] RTMP_AllTimerListRelease: Cancel timer obj ffffff800d5fd110!
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.449210] RTMP_AllTimerListRelease: Timer is allocated by RRM_CfgInit+0x84/0x120 [mt_wifi],Valid:1,Lock:ffffff800d7cdae8,State:0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.449214] RTMP_AllTimerListRelease: Cancel timer obj ffffff800d5fd168!
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.449254] RTMP_AllTimerListRelease: Timer is allocated by RRM_CfgInit+0x9c/0x120 [mt_wifi],Valid:1,Lock:ffffff800d7cdae8,State:0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.449256] RTMP_AllTimerListRelease: Cancel timer obj ffffff800d603f00!
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.449296] RTMP_AllTimerListRelease: Timer is allocated by RRM_CfgInit+0x84/0x120 [mt_wifi],Valid:1,Lock:ffffff800d7cdae8,State:0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.449299] RTMP_AllTimerListRelease: Cancel timer obj ffffff800d603f58!
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.449339] RTMP_AllTimerListRelease: Timer is allocated by RRM_CfgInit+0x9c/0x120 [mt_wifi],Valid:1,Lock:ffffff800d7cdae8,State:0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.449341] RTMP_AllTimerListRelease: Cancel timer obj ffffff800d60acf0!
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.449381] RTMP_AllTimerListRelease: Timer is allocated by RRM_CfgInit+0x84/0x120 [mt_wifi],Valid:1,Lock:ffffff800d7cdae8,State:0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.449385] RTMP_AllTimerListRelease: Cancel timer obj ffffff800d60ad48!
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.449425] RTMP_AllTimerListRelease: Timer is allocated by RRM_CfgInit+0x9c/0x120 [mt_wifi],Valid:1,Lock:ffffff800d7cdae8,State:0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.449427] RTMP_AllTimerListRelease: Cancel timer obj ffffff800d611ae0!
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.449467] RTMP_AllTimerListRelease: Timer is allocated by RRM_CfgInit+0x84/0x120 [mt_wifi],Valid:1,Lock:ffffff800d7cdae8,State:0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.449470] RTMP_AllTimerListRelease: Cancel timer obj ffffff800d611b38!
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.449511] RTMP_AllTimerListRelease: Timer is allocated by RRM_CfgInit+0x9c/0x120 [mt_wifi],Valid:1,Lock:ffffff800d7cdae8,State:0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.449513] FwOwn()::Return since already in Fw Own...
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.449526] <---HwCtrlThread
Aug 27 10:43:03 UDM user.info kernel: [ 1747.500773] wevent: unregistered
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.500778] ra0: NO2G remove ACL table
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.531688] OS_NumOfMemAlloc = 421662, OS_NumOfMemFree = 421381
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.531691] OS_NumOfPktAlloc = 0, OS_NumOfPktFree = 0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.531693] APCLI init address table. Max count:64
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.555683] OS_NumOfMemAlloc = 421662, OS_NumOfMemFree = 421382
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.555687] OS_NumOfPktAlloc = 0, OS_NumOfPktFree = 0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.579681] OS_NumOfMemAlloc = 421662, OS_NumOfMemFree = 421383
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.579685] OS_NumOfPktAlloc = 0, OS_NumOfPktFree = 0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.603660] OS_NumOfMemAlloc = 421662, OS_NumOfMemFree = 421384
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.603663] OS_NumOfPktAlloc = 0, OS_NumOfPktFree = 0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.631696] OS_NumOfMemAlloc = 421662, OS_NumOfMemFree = 421385
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.631699] OS_NumOfPktAlloc = 0, OS_NumOfPktFree = 0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.675665] OS_NumOfMemAlloc = 421662, OS_NumOfMemFree = 421387
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.675669] OS_NumOfPktAlloc = 0, OS_NumOfPktFree = 0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.727692] OS_NumOfMemAlloc = 421662, OS_NumOfMemFree = 421389
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.727698] OS_NumOfPktAlloc = 0, OS_NumOfPktFree = 0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.771682] OS_NumOfMemAlloc = 421662, OS_NumOfMemFree = 421391
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.771687] OS_NumOfPktAlloc = 0, OS_NumOfPktFree = 0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.811748] OS_NumOfMemAlloc = 421662, OS_NumOfMemFree = 421393
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.811753] OS_NumOfPktAlloc = 0, OS_NumOfPktFree = 0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.867654] OS_NumOfMemAlloc = 421662, OS_NumOfMemFree = 421395
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.867658] OS_NumOfPktAlloc = 0, OS_NumOfPktFree = 0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.919671] OS_NumOfMemAlloc = 421662, OS_NumOfMemFree = 421397
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.919675] OS_NumOfPktAlloc = 0, OS_NumOfPktFree = 0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.951687] OS_NumOfMemAlloc = 421662, OS_NumOfMemFree = 421399
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.951691] OS_NumOfPktAlloc = 0, OS_NumOfPktFree = 0
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.951696] RtmpOSNetDevDetach(): RtmpOSNetDeviceDetach(), dev->name=ra0!
Aug 27 10:43:03 UDM user.warn kernel: [ 1747.991677] 80211> unregister/free wireless device
Aug 27 10:43:03 UDM user.warn kernel: [ 1748.008059] OS_NumOfMemAlloc = 421662, OS_NumOfMemFree = 421663
Aug 27 10:43:03 UDM user.warn kernel: [ 1748.008062] OS_NumOfPktAlloc = 0, OS_NumOfPktFree = 0
Aug 27 10:43:03 UDM user.warn kernel: [ 1748.008097] Unregister PCI device driver
Aug 27 10:43:03 UDM user.info kernel: [ 1748.027523] wevent: unregistered
Aug 27 10:43:03 UDM user.warn kernel: [ 1748.071652] APCLI remove address table
Aug 27 10:43:03 UDM user.warn kernel: [ 1748.071660] WDS_Remove():
Aug 27 10:43:03 UDM user.warn kernel: [ 1748.159682] wlan_operate_exit : Can't find wlan operate for wdev.
Aug 27 10:43:03 UDM user.warn kernel: [ 1748.199671] wlan_operate_exit : Can't find wlan operate for wdev.
Aug 27 10:43:03 UDM user.warn kernel: [ 1748.231683] wlan_operate_exit : Can't find wlan operate for wdev.
Aug 27 10:43:03 UDM user.warn kernel: [ 1748.275678] wlan_operate_exit : Can't find wlan operate for wdev.
Aug 27 10:43:03 UDM user.warn kernel: [ 1748.319652] wlan_operate_exit : Can't find wlan operate for wdev.
Aug 27 10:43:04 UDM user.warn kernel: [ 1748.371675] wlan_operate_exit : Can't find wlan operate for wdev.
Aug 27 10:43:04 UDM user.warn kernel: [ 1748.411739] wlan_operate_exit : Can't find wlan operate for wdev.
Aug 27 10:43:04 UDM user.warn kernel: [ 1748.451685] RtmpOSNetDevDetach(): RtmpOSNetDeviceDetach(), dev->name=rai0!
Aug 27 10:43:04 UDM user.warn kernel: [ 1748.499648] wlan_operate_exit : Can't find wlan operate for wdev.
Aug 27 10:43:04 UDM user.warn kernel: [ 1748.499654] 80211> unregister/free wireless device
Aug 27 10:43:04 UDM user.warn kernel: [ 1748.512251] Unregister PCI device driver
Aug 27 10:43:04 UDM user.warn sysmon: sysmon[2538]: report.ubus_connection_lost(): ubus connection lost, reconnecting...
Aug 27 10:43:04 UDM user.err ubios-init: WARN : Unable to apply default sysctl.conf
Aug 27 10:43:04 UDM user.info kernel: [ 1748.611854] writing to auto_msgmni has no effect
Aug 27 10:43:04 UDM user.info kernel: [ 1748.730404] al_eth 0000:00:01.0 eth0.tmp: renamed from switch0
Aug 27 10:43:04 UDM user.info kernel: [ 1748.762440] al_eth 0000:00:03.0 eth1.tmp: renamed from eth4
Aug 27 10:43:04 UDM user.info kernel: [ 1749.018650] zram: Removed device: zram0
Aug 27 10:43:04 UDM syslog.info syslogd exiting

No errors detected

[chriscpritchard]

Hi,

Sorry, I initially posted this in the wrong bug, it fits better here:

I'm having the same issue. I've even tried compiling the stock kernel (and used GPL sources direct from unifi and the config direct from the UDM) to see if it was a something here that was causing the error, but seems to be some form of bug. I've tried building using GCC 6.4.0 (the same compiler used to build the stock kernel) and I have the same error. Unfortunately, it seems the rlt_wifi driver is not open source, instead it's one created by the manufacturer (mediatek), although an old, GPL version of the module seems to be available on github (https://github.com/openwrt/mtk-wifi-gpl), I've not been able to acquire the driver directly from mediatek although I'm not sure to what extent I'd be able to if I asked, nor am I sure of how the UDM could be told to load it.

[    0.544869] pci-pf-stub 0000:00:04.0: writing to VF config space
[    0.544934] pci-pf-stub 0000:00:05.0: writing to VF config space
[    0.567585] ahci 0001:00:00.0: writing to VF config space
[    0.573081] ahci 0002:00:00.0: writing to VF config space
[    0.578582] ahci 0003:00:00.0: writing to VF config space
[    0.586107] al_eth 0000:00:01.0: writing to VF config space
[    2.525656] al_eth 0000:00:03.0: writing to VF config space
[   14.914001] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000000
[   14.922863] Mem abort info:
[   14.925688]   ESR = 0x96000045
[   14.928774]   Exception class = DABT (current EL), IL = 32 bits
[   14.934749]   SET = 0, FnV = 0
[   14.937834]   EA = 0, S1PTW = 0
[   14.941003] Data abort info:
[   14.943937]   ISV = 0, ISS = 0x00000045
[   14.947800]   CM = 0, WnR = 1
[   14.950799] user pgtable: 4k pages, 39-bit VAs, pgdp = 00000000c993cdec
[   14.957466] [0000000000000000] pgd=0000000000000000, pud=0000000000000000
[   14.964280] Internal error: Oops: 96000045 [#1] SMP
[   14.969209] Modules linked in: rlt_wifi(FO+) sch_fq_codel sch_htb lzo lzo_compress zram gpiodev(PFO) ubnthal(PFO) ubnt_common(PFO)
[   14.980998] Process modprobe (pid: 958, stack limit = 0x00000000b4d6f6e3)
[   14.987809] CPU: 0 PID: 958 Comm: modprobe Tainted: PF          O      4.19.152-custom-al-linux-v10.2.0-v1.11.0.3921-f2e3fac #1
[   14.999330] Hardware name: Annapurna Labs Alpine V2 UBNT (DT)
[   15.005129] pstate: 80000005 (Nzcv daif -PAN -UAO)
[   15.009949] pc : __memcpy+0x94/0x180
[   15.013660] lr : RtmpOSNetDevAttach+0x8c/0x140 [rlt_wifi]
[   15.019082] sp : ffffff800c383870
[   15.022422] x29: ffffff800c383870 x28: 0000000000000001
[   15.027788] x27: ffffff8008121270 x26: 0000000000000000
[   15.033124] x25: ffffff800c800000 x24: ffffffc07c2c7000
[   15.038489] x23: ffffffc07cb9d098 x22: ffffff8000c56000
[   15.043825] x21: ffffff800979d000 x20: ffffffc07c2c7000
[   15.049189] x19: ffffff800c383920 x18: 0000000000020000
[   15.054523] x17: 0000000000000001 x16: 0000000000000007
[   15.059886] x15: ffffff8008cb5b58 x14: ffffff8008c20250
[   15.065221] x13: 0000000000000000 x12: ffffff8008cb5000
[   15.070585] x11: ffffff8008c20000 x10: 0000000000000000
[   15.075921] x9 : 0000000000000002 x8 : 00000000000001e3
[   15.081284] x7 : 0000000000005002 x6 : 0000000000000000
[   15.086620] x5 : 0000000000000000 x4 : ffffff800c383908
[   15.091984] x3 : 0000000000000000 x2 : 0000000000000006
[   15.097319] x1 : ffffff800c383970 x0 : 0000000000000000
[   15.102654] Call trace:
[   15.105129]  __memcpy+0x94/0x180
[   15.108446]  rt_pci_probe+0x43c/0x550 [rlt_wifi]
[   15.113091]  pci_device_probe+0xd8/0x1a0
[   15.117069]  really_probe+0x250/0x390
[   15.120756]  driver_probe_device+0x54/0xf0
[   15.124876]  __driver_attach+0x104/0x120
[   15.128853]  bus_for_each_dev+0x70/0xcc
[   15.132714]  driver_attach+0x20/0x30
[   15.136316]  bus_add_driver+0x164/0x20c
[   15.140205]  driver_register+0x74/0x120
[   15.144066]  __pci_register_driver+0x40/0x4c
[   15.148409]  rt_pci_init_module+0x15c/0x178 [rlt_wifi]
[   15.153647]  wifi_drv_init_module+0x10/0xe88 [rlt_wifi]
[   15.158898]  do_one_initcall+0x50/0x154
[   15.162789]  do_init_module+0x54/0x210
[   15.166564]  load_module+0x1d5c/0x23d0
[   15.170338]  __se_sys_finit_module+0xd4/0xf0
[   15.174660]  __arm64_sys_finit_module+0x18/0x20
[   15.179216]  el0_svc_handler+0xb8/0x1a0
[   15.183078]  el0_svc+0x8/0xc4
[   15.186101] Code: f8408423 f80084c3 36100062 b8404423 (b80044c3)
[   15.192218] ---[ end trace d34057f53bffbf66 ]---
[   15.200715] Kernel panic - not syncing: Fatal exception
[   15.205965] SMP: stopping secondary CPUs
[   15.209933] Kernel Offset: disabled
[   15.213476] CPU features: 0x0,20006008
[   15.217249] Memory Limit: none
[   15.224145] Rebooting in 3 seconds..

Over the next few days I'll try recompiling with kernel debugging enabled to see if that sheds any more light :)

[fabianishere]

One solution I could think of is using the open-source mt76 driver. This one is also available in more recent Linux versions, though I don’t know how well it works.

[chriscpritchard]

One solution I could think of is using the open-source mt76 driver. This one is also available in more recent Linux versions, though I don’t know how well it works.

I've tried with that built into the kernel, but it doesn't seem to work as the rlt_wifi module still tries to load. I could try and blacklist the rlt_wifi module, and I'll try that, but there is clearly something going on and it would be preferable for the rlt_wifi module to work.

[fabianishere]

Yes, that’s the approach I was thinking of.

Alternatively, the only way I expect it to work is to obtain the corresponding kernel sources for the firmware you are running, and use that to build a custom kernel.

What version are you currently running?

[chriscpritchard]

1.11.0 - I've obtained the corresponding kernel sources for that firmware from ubiquiti and built the kernel using them and the stock config - I got the same error (one of the first things I tried). I've not yet tried the stock kernel sources, stock config and GCC 6.4.0 but that's next on my list of things to try (after that I'll try blacklisitng rlt_wifi).

[fabianishere]

The only way I imagine these errors happening, is due to ABI differences between the kernel and the kernel that the module was built for.

It could indeed be the case that a different compiler version may result in ABI changes.

[chriscpritchard]

Having played around I've now built several kernels using the GPL sources provided by unifi for firmware v1.11.0-23 (or here for just the kernel), they sent them to me in a google drive link, but you can obtain them yourself by emailing [email protected], should you not wish to download from my website they were pretty quick at sending them.

root@ubnt:/data/kernel-deb# udm-bootctl list
Version                                         Initrd
4.19.152-al-linux-v1.11.0.3921-edge3-udm        yes
4.19.152-al-linux-v1.11.0.3921-edge3-udm-bpf    yes
4.19.152-al-linux-v1.11.0.3921-stock-udm        yes
4.19.152-al-linux-v1.11.0.3921-stock-udm-bpf    yes

firstly the stock kernel - this is using the config.gz from my UDM, with forced module loading enabled (as the only change from stock): udm-kernel-stock.zip, this boots and runs fine.

secondly the edge3 releases (with and without the bpf changes from fabianishere/udm-kernel/pull/6) - this is using config.gz from my UDM as config.stock.udm and the config.edge.udm from the fabianshare/udm-kernel, this does not boot with the same error

I've also built stock-udm-bpf - config.gz from the UDM, forced module loading and config.local.udm containing the BPF changes from the above pull request: udm-kernel-stock-bpf.zip. This also boots fine.

It therefore looks like the issue has something to do with one of the options enabled in the edge config. I will be moving through them systematically when I get a chance!