Xanmod-5.15-tt kernel freezes some time after booting. about tt-cpu-scheduler

[raykzhao]

Hi @skulldealer @MoisesMH
Could you please try this fix #5 (comment)
It fixes the tasks accounting and stats, in case the issue is related to cpu frequ somehow.

i use intel_cpufreq and schedutil governer. i am not using pstate. i am not sure as i have never applied a patch. Do i need to get xanmod source code, apply the patch and recompile the kernel?

Hi @skulldealer,

The patch at #5 (comment) fixed the frequency scaling issue on my laptop with intel_cpufreq and schedutil when CONFIG_NO_HZ_FULL=n. Make sure you have the compiling toolchain i.e. gcc, binutils, etc. installed on your system. What you may try is:

1. Download the xanmod kernel code from https://github.com/xanmod/linux/tree/5.15-tt via git clone https://github.com/xanmod/linux -b 5.15-tt.
2. Download and unzip the patch to the xanmod kernel code folder.
3. In the xanmod kernel code folder, run patch -p1 -i accounting.patch to apply the patch.
4. In the xanmod kernel code folder, run cp CONFIGS/xanmod/gcc/config .config to apply the xanmod kernel configuration.
5. Run make menuconfig, then go to General setup--->Timers subsystem, change the Timer tick handling to Idle dynticks system (tickless idle).
6. Run make -jx && make install && make modules_install to compile and install the kernel. Replace x with the number of CPU cores e.g. make -j4 for a 4-core CPU.
7. Generate the initramfs for the kernel image and change the bootloader configuration to load the new kenrel. You should lookup how to do it for your Linux distro.

If you are using any out-of-tree kernel modules e.g. nvidia driver, you should also recompile those modules.

[MoisesMH]

I have not checked the non-TT version of Xanmod, but with the TT-version a little freeze happens (1 or 2 secs) when booting, not present in 5.14.16-xanmod1-cacule; then everything from the login runs fine. I'm using amd-pstate as scaling governor, because I have a Zen2 processor (Ryzen 5 3600 to be specific). I should test without it too. Maybe that can be the problem.

Anyway, when playing games, the schedules is as responsive as CacULE as you've said from my perspective. Congratulations!

I suggest this problem should also be mentioned to xanmod developers, just in case they want to test too.

[xanmod]

@skulldealer @MoisesMH

sudo apt install ./linux-image-* ./linux-headers-*

[hamadmarri]

Hi @skulldealer

Does this issue happen with xanmod-5.15 (non TT) too? Could you test please.

Could you please attach the /var/log/syslog

Thank you

syslog.txt

Please check the latest logs first, I uploaded the log after hard reboot from freeze.

Does this issue happen with xanmod-5.15 (non TT) too? Could you test please.

@hamadmarri Yes I have tested other kernels. I am facing this issue since yesterday, immediately after updating to the 5.15 tt kernel and rebooting. I was using cacule kernel 5.14 before. Everything was fine then.
I have also tried changing cpu governor to on-demand and schedutil, but that also doesn't help either

[hamadmarri]

syslog.txt

Please check the latest logs first, I uploaded the log after hard reboot from freeze.

Does this issue happen with xanmod-5.15 (non TT) too? Could you test please.

@hamadmarri Yes I have tested other kernels. I am facing this issue since yesterday, immediately after updating to the 5.15 tt kernel and rebooting. I was using cacule kernel 5.14 before. Everything was fine then. I have also tried changing cpu governor to on-demand and schedutil, but that also doesn't help either

Just to make sure, is it freezing with xanmod-5.15 not the TT?
In case both xanmod-5.15 and xanmod-tt-5.15 have the same issue, it could be either the linux 5.15 version or some xanmod configs that cause the issue.

I am checking the syslog.txt

[hamadmarri]

I can't see any bug report in the syslog but here is some observations:

Does the below error shown in previous working kernel?

Nov  7 16:15:17 localhost io.elementary.wingpanel.desktop[5633]: [5625:5625:1107/161517.913151:ERROR:fallback_task_provider.cc(119)] Every renderer should have at least one task provided by a primary task provider. If a fallback task is shown, it is a bug. Please file a new bug and tag it as a dependency of crbug.com/739782.

I noticed that ananicy is running as daemon, could you disable it and make it not auto started.

Also if you can try without mitigations=off in the kernel parameters.

syslog.txt

Please check the latest logs first, I uploaded the log after hard reboot from freeze.

Does this issue happen with xanmod-5.15 (non TT) too? Could you test please.

@hamadmarri Yes I have tested other kernels. I am facing this issue since yesterday, immediately after updating to the 5.15 tt kernel and rebooting. I was using cacule kernel 5.14 before. Everything was fine then. I have also tried changing cpu governor to on-demand and schedutil, but that also doesn't help either

Just to make sure, is it freezing with xanmod-5.15 not the TT?
In case both xanmod-5.15 and xanmod-tt-5.15 have the same issue, it could be either the linux 5.15 version or some xanmod configs that cause the issue.

I am checking the syslog.txt

Ok, I will do some more testing. 5.15 is landed for cacule and edge version only. I will also try mainstream kernel. It may take few weeks to land as stable lts build. I will check if it's there in the experimental nightly builds. Give me some time, I will get back with some more details.

[hamadmarri]

syslog.txt
Please check the latest logs first, I uploaded the log after hard reboot from freeze.

Does this issue happen with xanmod-5.15 (non TT) too? Could you test please.

@hamadmarri Yes I have tested other kernels. I am facing this issue since yesterday, immediately after updating to the 5.15 tt kernel and rebooting. I was using cacule kernel 5.14 before. Everything was fine then. I have also tried changing cpu governor to on-demand and schedutil, but that also doesn't help either

Just to make sure, is it freezing with xanmod-5.15 not the TT?
In case both xanmod-5.15 and xanmod-tt-5.15 have the same issue, it could be either the linux 5.15 version or some xanmod configs that cause the issue.
I am checking the syslog.txt

Ok, I will do some more testing. 5.15 is landed for cacule and edge version only. I will also try mainstream kernel. It may take few weeks to land as stable lts build. I will check if it's there in the experimental nightly builds. Give me some time, I will get back with some more details.

Yes, please try Linux 5.15 | Latest Mainline (EDGE) | 5.15.1-xanmod1 | 2021-11-06

Thank you

I have not checked the non-TT version of Xanmod, but with the TT-version a little freeze happens (1 or 2 secs) when booting, not present in 5.14.16-xanmod1-cacule; then everything from the login runs fine. I'm using amd-pstate as scaling governor, because I have a Zen2 processor (Ryzen 5 3600 to be specific). I should test without it too. Maybe that can be the problem.

Anyway, when playing games, the schedules is as responsive as CacULE as you've said from my perspective. Congratulations!

I suggest this problem should also be mentioned to xanmod developers, just in case they want to test too.

I made an interesting observation today, I left my computer idle for an hour and then ran stress-ng for another 1 hour. I did not touch my computer then. It was running the xanmod tt kernel. After this I closed all the programs and I was scrolling web as usual and typing a lot of stuff, it suddenly freezes. So it's not a performance issue. But probably misinterpretation of some Xorg input service. It might be possible that this problem of freezing doesn't occur in Wayland. I have no means of testing this, but would be great if somebody could do this.

If you are experiencing the same problem, please try not using the computer after boot for an hour, turn off auto sleep and auto suspend. They just scroll the web or do something basic. Let me know if the computer freezes.

For me I am using auto-cpufreq with schedutil governor. Turbo boost set to auto.

I am not sure but it seems installing the suggested packages also with the xanmod kernel is making a difference.
I ran
sudo apt install clang lld llvm

For additional performance gains I use
irqbalance
nohang
prelockd
memavalid
Ananicy
Auto cpufreq
Preload

here is my /etc/sysctl.conf
sysctl.conf.txt

the modifications in sysctl.conf are inspired by ptr1337 from https://forum.garudalinux.org/t/linux-cacule-feedback-testing/7672

i am gathering more data and observations that might help. let me know if you want me to try something specific. right now i have installed xanmod-edge, will let you know after testing it out.

Also somebody please ask xanmod guys to investigate the problem.

[xanmod]

Please, confirm this issue with CFS version, edge branch: 5.15.1-xanmod1

syslog.txt
Please check the latest logs first, I uploaded the log after hard reboot from freeze.

Does this issue happen with xanmod-5.15 (non TT) too? Could you test please.

@hamadmarri Yes I have tested other kernels. I am facing this issue since yesterday, immediately after updating to the 5.15 tt kernel and rebooting. I was using cacule kernel 5.14 before. Everything was fine then. I have also tried changing cpu governor to on-demand and schedutil, but that also doesn't help either

Just to make sure, is it freezing with xanmod-5.15 not the TT?
In case both xanmod-5.15 and xanmod-tt-5.15 have the same issue, it could be either the linux 5.15 version or some xanmod configs that cause the issue.
I am checking the syslog.txt

Ok, I will do some more testing. 5.15 is landed for cacule and edge version only. I will also try mainstream kernel. It may take few weeks to land as stable lts build. I will check if it's there in the experimental nightly builds. Give me some time, I will get back with some more details.

Yes, please try Linux 5.15 | Latest Mainline (EDGE) | 5.15.1-xanmod1 | 2021-11-06

Thank you

Hey.
Here you go.
I first tested the xanmod tt kernel. It's not a kernel panic. Because the caps lock is not blinking. Although everything is frozen. Caps lock and num lock do not respond with indicator light when pressed during freeze. So all the hardware freezes too. This shows that something went wrong with the kernel and not the desktop environment or Xorg. I waited for 30 minutes after the freeze, no heating issues observed.

For some work related purpose, I used manjaro gnome live usb with their latest iso for 3 hours at a stretch, no freeze occurred. So this is not my hardware issue.

I did not face this problem with the 5.14 cacule kernel. I have been using the cacule kernel for 4 to 5 months.

So something either went wrong specifically with the 5.15tt xanmod kernel or 5.15 xanmod kernel itself. So I tested out the edge 5.15 xanmod.

I can confirm that this issue is specific to xanmod-5.15tt and the xanmod 5.15 edge edition shows no freezing.

So I believe the issue is specific to tt-xanmod kernel only.

I didn't observe this problem under load like mprime and stress-ng.
But I observed this while doing multitasking like scrolling web switching workspaces typing and using touchpad gestures especially soon after boot.

I am still testing the edge kernel. I still recommend someone else experiencing the same issue to try using edge kernel

Please, confirm this issue with CFS version, edge branch: 5.15.1-xanmod1

It's a problem with 5.15.1 version, can't really say if it is specific to cacule/tt version. Need more time to test. These are fairly new kernels.

I have not checked the non-TT version of Xanmod, but with the TT-version a little freeze happens (1 or 2 secs) when booting, not present in 5.14.16-xanmod1-cacule; then everything from the login runs fine. I'm using amd-pstate as scaling governor, because I have a Zen2 processor (Ryzen 5 3600 to be specific). I should test without it too. Maybe that can be the problem.

Anyway, when playing games, the schedules is as responsive as CacULE as you've said from my perspective. Congratulations!

I suggest this problem should also be mentioned to xanmod developers, just in case they want to test too.

Hi @MoisesMH,
Please try

sudo apt install clang lld llvm

It appears to fix the problem for me. I am still testing. Let me know how this works for you.
These packages were recommended to me while installing xanmod

Final Update:

I have observed random system freezes in both xanmod 5.15 edge and cacule/tt kernels several times now.

Should I take my issue to xanmod's GitHub page or continue here?

[xanmod]

I will soon make a new revision build here.

Test before for me if the same problem occurs with this kernel (kernel-ppa):
image
headers
headers

I will soon make a new revision build here, I need you to test it for me

Ok fine. Please provide me the instructions. I use elementary os 6. I don't have prior experience of testing kernels.

Have you figured out the problem?

@xanmod i am using elementary os 6 based on ubuntu 20.04 lts.
which kernel ppa must be installed before installing the deb packages?

~/Desktop ❯❯❯ sudo apt install ./linux-image-* ./linux-headers-*
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Note, selecting 'linux-image-unsigned-5.15.1-051501-generic' instead of './linux-image-unsigned-5.15.1-051501-generic_5.15.1-051501.202111061036_amd64.deb'
Note, selecting 'linux-headers-5.15.1-051501-generic' instead of './linux-headers-5.15.1-051501-generic_5.15.1-051501.202111061036_amd64.deb'
Note, selecting 'linux-headers-5.15.1-051501' instead of './linux-headers-5.15.1-051501_5.15.1-051501.202111061036_all.deb'
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 linux-headers-5.15.1-051501-generic : Depends: libc6 (>= 2.34) but 2.31-0ubuntu9.2 is to be installed
 linux-image-unsigned-5.15.1-051501-generic : Depends: linux-modules-5.15.1-051501-generic but it is not installable
E: Unable to correct problems, you have held broken packages.

[hamadmarri]

@xanmod i am using elementary os 6 based on ubuntu 20.04 lts. which kernel ppa must be installed before installing the deb packages?

You don't need the ppa for the provided packages here (#3 (comment))

Download them and cd to the folder containing them and then run

sudo apt install ./linux-image-* ./linux-headers-*

@xanmod i am using elementary os 6 based on ubuntu 20.04 lts. which kernel ppa must be installed before installing the deb packages?

You don't need the ppa for the provided packages here (#3 (comment))

Download them and cd to the folder containing them and then run

sudo apt install ./linux-image-* ./linux-headers-*

i did the same thing above. please see the error i am getting

[xanmod]

Sorry, also install: https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.15.1/amd64/linux-modules-5.15.1-051501-generic_5.15.1-051501.202111061036_amd64.deb

Disregard the headers packages.

Sorry, also install: https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.15.1/amd64/linux-modules-5.15.1-051501-generic_5.15.1-051501.202111061036_amd64.deb

Disregard the headers packages.

@xanmod which header file should I download. There are two. Should I get both?

[xanmod]

You don't need to install both, they are only needed for dkms proprietary modules, just install image and modules for us to test.

sudo apt install ./linux-image-* ./linux-modules-*

and when rebooting, make sure you are booting the correct kernel.

Sorry hamad for using your issues channel for this.

[hamadmarri]

Sorry hamad for using your issues channel for this.

No need to be sorry, actually I am happy that you are taking care of this issue because I am not sure about the xanmod deb build details.

Thank you so much @xanmod

You don't need to install both, they are only needed for dkms proprietary modules, just install image and modules for us to test.

sudo apt install ./linux-image-* ./linux-modules-*

and when rebooting, make sure you are booting the correct kernel.

Sorry hamad for using your issues channel for this.

Done. I am finally running your kernel build.
I saw this on boot:

@xanmod What should I do now? Do you need any logs to understand the problem? Or should I let you know if the freezing issue is fixed or not with the generic kernel?

Edit: I was using the kernel you provided and it randomly froze up again. How can I help you with the logs?

[hamadmarri]

amd-pstate

Hi @MoisesMH

Please let me know if the amd-pstate is reporting correctly since I have stripped out many tasks accounting and stats.

Glad to here that TT is working well in your machine.

Thank you

[xanmod]

Stopped freezing with the generic?
Please compress and send the kern.log file in /var/log/kern.log

kernlogs.zip

Froze 2 times with the new kernel. Here is the log.

[xanmod]

Well, this issue is not directly related to xanmod, I found conflicts with elementaryos dbus:

Nov  7 07:16:29 localhost kernel: [   72.155890] io.elementary.g[5855]: segfault at 5 ip 00007fe4e145211f sp 00007ffc68d0bc60 error 4 in libgtk-3.so.0.2404.16[7fe4e11e0000+36f000]
Nov  7 07:16:29 localhost kernel: [   72.155930] Code: 56 41 55 41 54 55 53 48 85 ff 0f 84 db 00 00 00 48 89 fd 49 89 f4 41 89 d5 e8 2d fb ff ff 48 89 c6 48 8b 45 00 48 85 c0 74 05 <48> 39 30 74 10 48 89 ef e8 34 cb d9 ff 85 c0 0f 84 ac 00 00 00 4d
Nov  7 07:16:33 localhost kernel: [   75.480055] io.elementary.g[7126]: segfault at 5 ip 00007fed0249c11f sp 00007ffe154f2f90 error 4 in libgtk-3.so.0.2404.16[7fed0222a000+36f000]
Nov  7 07:16:33 localhost kernel: [   75.480092] Code: 56 41 55 41 54 55 53 48 85 ff 0f 84 db 00 00 00 48 89 fd 49 89 f4 41 89 d5 e8 2d fb ff ff 48 89 c6 48 8b 45 00 48 85 c0 74 05 <48> 39 30 74 10 48 89 ef e8 34 cb d9 ff 85 c0 0f 84 ac 00 00 00 4d
Nov  7 07:16:42 localhost kernel: [   84.512718] rfkill: input handler disabled
Nov  7 07:19:14 localhost kernel: [  236.659664] rfkill: input handler enabled
Nov  7 07:19:14 localhost kernel: [  236.680879] dbus-daemon[7426]: segfault at 0 ip 00007fc0eb0f90d7 sp 00007ffd67563148 error 6 in libdbus-1.so.3.19.11[7fc0eb0d5000+2e000]
Nov  7 07:19:14 localhost kernel: [  236.680915] Code: 44 00 00 48 89 d6 e9 58 fe ff ff 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 48 8b 46 08 48 39 f0 74 43 48 8b 16 48 89 42 08 48 8b 16 <48> 89 10 48 39 37 74 11 48 c7 46 08 00 00 00 00 48 c7 06 00 00 00

[xanmod]

Obviously elementary doesn't support unofficial kernels from the distribution, but you can report this issue to element. dev team

Well, this issue is not directly related to xanmod, I found conflicts with elementaryos dbus:

Nov  7 07:16:29 localhost kernel: [   72.155890] io.elementary.g[5855]: segfault at 5 ip 00007fe4e145211f sp 00007ffc68d0bc60 error 4 in libgtk-3.so.0.2404.16[7fe4e11e0000+36f000]
Nov  7 07:16:29 localhost kernel: [   72.155930] Code: 56 41 55 41 54 55 53 48 85 ff 0f 84 db 00 00 00 48 89 fd 49 89 f4 41 89 d5 e8 2d fb ff ff 48 89 c6 48 8b 45 00 48 85 c0 74 05 <48> 39 30 74 10 48 89 ef e8 34 cb d9 ff 85 c0 0f 84 ac 00 00 00 4d
Nov  7 07:16:33 localhost kernel: [   75.480055] io.elementary.g[7126]: segfault at 5 ip 00007fed0249c11f sp 00007ffe154f2f90 error 4 in libgtk-3.so.0.2404.16[7fed0222a000+36f000]
Nov  7 07:16:33 localhost kernel: [   75.480092] Code: 56 41 55 41 54 55 53 48 85 ff 0f 84 db 00 00 00 48 89 fd 49 89 f4 41 89 d5 e8 2d fb ff ff 48 89 c6 48 8b 45 00 48 85 c0 74 05 <48> 39 30 74 10 48 89 ef e8 34 cb d9 ff 85 c0 0f 84 ac 00 00 00 4d
Nov  7 07:16:42 localhost kernel: [   84.512718] rfkill: input handler disabled
Nov  7 07:19:14 localhost kernel: [  236.659664] rfkill: input handler enabled
Nov  7 07:19:14 localhost kernel: [  236.680879] dbus-daemon[7426]: segfault at 0 ip 00007fc0eb0f90d7 sp 00007ffd67563148 error 6 in libdbus-1.so.3.19.11[7fc0eb0d5000+2e000]
Nov  7 07:19:14 localhost kernel: [  236.680915] Code: 44 00 00 48 89 d6 e9 58 fe ff ff 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 48 8b 46 08 48 39 f0 74 43 48 8b 16 48 89 42 08 48 8b 16 <48> 89 10 48 39 37 74 11 48 c7 46 08 00 00 00 00 48 c7 06 00 00 00

Thanks a lot. I guess it is a conflict with the mainline generic 5.15 kernel itself and has nothing to do with xanmod. It should probably get fixed.

@xanmod ,
Please head over to:

xanmod/linux#212

For further discussion on the issue.

[hamadmarri]

xanmod/linux#212 (comment)

Is it solved by disabling ananicy-cpp?

xanmod/linux#212 (comment)

Is it solved by disabling ananicy-cpp?

Hey @hamadmarri & @xanmod , I have upgraded to the newer version of xanmod tt kernel.
Probably this release fixes a few issues after i read the changelog.

805bd018d8f0 char/lrng: IRQ entropy source

Also, Currently I have disabled all my performance tweaks like preload, auto-cpufreq, ananicy, memavalid, prelockd, irqbalance, nohangand moved 99-sysctl-cacule.conf temporarily to my home folder. Everything seems stable now and I am not encountering any freezes.

I am trying to figure out which one of my tweaks cause such random freezes when enabled. Will let you know.

So far not sure if ananicy is the one causing problems.

[hamadmarri]

So far not sure if ananicy is the one causing problems.

I am asking about ananicy because TT doesn't support some groups features in which ananicy does manipulate tasks grouping. I suggest you debug with disabling ananicy first.

So far not sure if ananicy is the one causing problems.

I am asking about ananicy because TT doesn't support some groups features in which ananicy does manipulate tasks grouping. I suggest you debug with disabling ananicy first.

I disabled ananicy, and some other tweaks. My system still freezes. I am trying my best to look for the cause . But I guess @hamadmarri and @xanmod have to inspect my system logs. Please let me know if you need it. Also could you check my /etc/sysctl.conf ?

Things seem to have improved with the new release of xanmod-tt. Freezes are happening less frequently. I have also noticed random restarts happening twice today.

[hamadmarri]

Hi @skulldealer @MoisesMH

Could you please try this fix
#5 (comment)

It fixes the tasks accounting and stats, in case the issue is related to cpu frequ somehow.

Hi @skulldealer @MoisesMH

Could you please try this fix #5 (comment)

It fixes the tasks accounting and stats, in case the issue is related to cpu frequ somehow.

i use intel_cpufreq and schedutil governer. i am not using pstate.
i am not sure as i have never applied a patch. Do i need to get xanmod source code, apply the patch and recompile the kernel?

Can I first try powersave and performance governors with acpi-cpufreq driver? Technically this should solve my freezing problem right?

[hamadmarri]

Can I first try powersave and performance governors with acpi-cpufreq driver? Technically this should solve my freezing problem right?

I think so, or try ondemand too

Can I first try powersave and performance governors with acpi-cpufreq driver? Technically this should solve my freezing problem right?

I think so, or try ondemand too

Everything is stable with powersave governor and acpi-cpufreq but performance is bad due to cpu running at lower frequency

[hamadmarri]

Hi @skulldealer ,

I think you mean powersave is the one has low frequency. Yes this is normal. If performance runs without freezes, then it is highly related to the last patch fix

Hi @skulldealer ,

I think you mean powersave is the one has low frequency. Yes this is normal. If performance runs without freezes, then it is highly related to the last patch fix

I was having freezes in performance, everything is fine with ondemand and powersave.

Maybe max clock speeds are unstable. I guess you can merge the patch so that next xanmod release includes it.