TEST-46-HOMED is flaky in Github Actions about systemd

[DaanDeMeyer]

@yuwata another journal:

gh run download 9126934864 --name ci-mkosi-9126934864-1-arch-rolling-failed-test-journals -D ci/ci-mkosi-9126934864-1-arch-rolling-failed-test-journals && journalctl --file ci/ci-mkosi-9126934864-1-arch-rolling-failed-test-journals/test/journal/TEST-46-HOMED.journal --no-hostname -o short-monotonic -u TEST-46-HOMED.service -p info

[DaanDeMeyer]

Another different failure on Debian in https://github.com/systemd/systemd/actions/runs/9011374536/job/24758890861:

gh run download 9013386982 --name ci-mkosi-9013386982-1-centos-9-failed-test-journals -D ci/ci-mkosi-9013386982-1-centos-9-failed-test-journals && journalctl --file ci/ci-mkosi-9013386982-1-centos-9-failed-test-journals/test/journal/TEST-46-HOMED.journal --no-hostname -o short-monotonic -u testsuite-46.service -p info

This one is a kernel crash:

[   10.834840] kernel: VFS: Busy inodes after unmount of dm-0 (btrfs)
[   10.834861] kernel: ------------[ cut here ]------------
[   10.836255] kernel: kernel BUG at fs/super.c:698!
[   10.837314] kernel: invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
[   10.838764] kernel: CPU: 0 PID: 472 Comm: systemd-homewor Not tainted 6.7.12-cloud-amd64 #1  Debian 6.7.12-1
[   10.840816] kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-2.fc40 04/01/2014
[   10.850226] kernel: RIP: 0010:generic_shutdown_super+0x10e/0x110
[   10.851756] kernel: Code: 91 00 e8 95 67 f3 ff 48 8b bb 00 01 00 00 eb db 48 8b 43 28 48 8d b3 b8 03 00 00 48 c7 c7 68 69 a6 84 48 8b 10 e8 d2 a5 d5 ff <0f> 0b 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa
[   10.856569] kernel: RSP: 0018:ffffbde3009c3db0 EFLAGS: 00010246
[   10.857842] kernel: RAX: 000000000000002e RBX: ffff99c10c7eb000 RCX: 0000000000000027
[   10.859651] kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff99c179c21400
[   10.861386] kernel: RBP: ffffffffc08458c0 R08: 0000000000000000 R09: ffffbde3009c3c58
[   10.862984] kernel: R10: ffffbde3009c3c50 R11: 0000000000000003 R12: ffff99c10d405914
[   10.864559] kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[   10.866223] kernel: FS:  00007f03cb1779c0(0000) GS:ffff99c179c00000(0000) knlGS:0000000000000000
[   10.868342] kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   10.869884] kernel: CR2: 000055e709133000 CR3: 000000000d5d0005 CR4: 0000000000370ef0
[   10.871806] kernel: Call Trace:
[   10.872499] kernel:  <TASK>
[   10.873102] kernel:  ? die+0x36/0x90
[   10.873914] kernel:  ? do_trap+0xda/0x100
[   10.874845] kernel:  ? generic_shutdown_super+0x10e/0x110
[   10.876050] kernel:  ? do_error_trap+0x6a/0x90
[   10.876934] kernel:  ? generic_shutdown_super+0x10e/0x110
[   10.878091] kernel:  ? exc_invalid_op+0x50/0x70
[   10.879095] kernel:  ? generic_shutdown_super+0x10e/0x110
[   10.880230] kernel:  ? asm_exc_invalid_op+0x1a/0x20
[   10.881337] kernel:  ? generic_shutdown_super+0x10e/0x110
[   10.882646] kernel:  kill_anon_super+0x16/0x40
[   10.883649] kernel:  btrfs_kill_super+0x16/0x20 [btrfs]
[   10.884902] kernel:  deactivate_locked_super+0x33/0xb0
[   10.886089] kernel:  cleanup_mnt+0xba/0x150
[   10.886999] kernel:  task_work_run+0x5d/0x90
[   10.887985] kernel:  exit_to_user_mode_prepare+0x1dd/0x1e0
[   10.889279] kernel:  syscall_exit_to_user_mode+0x1b/0x40
[   10.890540] kernel:  do_syscall_64+0x70/0xe0
[   10.891492] kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
[   10.892755] kernel:  ? exit_to_user_mode_prepare+0x40/0x1e0
[   10.894044] kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
[   10.895224] kernel:  ? syscall_exit_to_user_mode+0x22/0x40
[   10.896257] kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
[   10.897402] kernel:  ? do_syscall_64+0x70/0xe0
[   10.898282] kernel:  entry_SYSCALL_64_after_hwframe+0x73/0x7b
[   10.899386] kernel: RIP: 0033:0x7f03cab272f7
[   10.900216] kernel: Code: 0c 00 f7 d8 64 89 02 b8 ff ff ff ff c3 66 0f 1f 44 00 00 31 f6 e9 09 00 00 00 66 0f 1f 84 00 00 00 00 00 b8 a6 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 01 c3 48 8b 15 f9 9a 0c 00 f7 d8 64 89 02 b8
[   10.904541] kernel: RSP: 002b:00007fffec601aa8 EFLAGS: 00000202 ORIG_RAX: 00000000000000a6
[   10.906315] kernel: RAX: 0000000000000000 RBX: 000055e70912f660 RCX: 00007f03cab272f7
[   10.908024] kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: 000055e70912f660
[   10.909691] kernel: RBP: 00007fffec601b10 R08: 0000000000000000 R09: 0000000000000000
[   10.911364] kernel: R10: 0000000000000007 R11: 0000000000000202 R12: 0000000000000010
[   10.912996] kernel: R13: 0000000000000000 R14: 000055e7078991ba R15: 0000000000000000
[   10.914603] kernel:  </TASK>
[   10.915123] kernel: Modules linked in: dm_crypt nls_ascii nls_cp437 vfat fat binfmt_misc btrfs intel_rapl_msr intel_rapl_common blake2b_generic iosf_mbi xor kvm_amd raid6_pq libcrc32c kvm sd_mod sg irqbypass crct10dif_pclmul crc32_p>
[   10.934080] kernel: ---[ end trace 0000000000000000 ]---

[yuwata]

Another different failure on Debian in https://github.com/systemd/systemd/actions/runs/9011374536/job/24758890861:

gh run download 9013386982 --name ci-mkosi-9013386982-1-centos-9-failed-test-journals -D ci/ci-mkosi-9013386982-1-centos-9-failed-test-journals && journalctl --file ci/ci-mkosi-9013386982-1-centos-9-failed-test-journals/test/journal/TEST-46-HOMED.journal --no-hostname -o short-monotonic -u testsuite-46.service -p info

Ugh, the journal is already removed? Please provide another one.

[DaanDeMeyer]

Another different failure on Debian in https://github.com/systemd/systemd/actions/runs/9011374536/job/24758890861:

gh run download 9013386982 --name ci-mkosi-9013386982-1-centos-9-failed-test-journals -D ci/ci-mkosi-9013386982-1-centos-9-failed-test-journals && journalctl --file ci/ci-mkosi-9013386982-1-centos-9-failed-test-journals/test/journal/TEST-46-HOMED.journal --no-hostname -o short-monotonic -u testsuite-46.service -p info

Ugh, the journal is already removed? Please provide another one.

I'll also increase the retention for failed journals to a month or so.

[yuwata]

Another failure:
https://github.com/systemd/systemd/actions/runs/9227184474/job/25388567140?pr=33010

[  102.612678] TEST-46-HOMED.sh[470]: + wait_for_state homedsshtest inactive
[  102.616943] TEST-46-HOMED.sh[470]: + for i in {1..10}
[  102.619418] TEST-46-HOMED.sh[470]: + ((  i > 1  ))
[  102.621764] TEST-46-HOMED.sh[2460]: + homectl inspect homedsshtest
[  102.628040] TEST-46-HOMED.sh[2461]: + grep -qF 'State: inactive'
[  102.634891] TEST-46-HOMED.sh[470]: + for i in {1..10}
[  102.638764] TEST-46-HOMED.sh[470]: + ((  i > 1  ))
[  102.642632] TEST-46-HOMED.sh[470]: + sleep 0.5
[  103.083616] TEST-46-HOMED.sh[2467]: + grep -qF 'State: inactive'
[  103.090356] TEST-46-HOMED.sh[2466]: + homectl inspect homedsshtest
[  103.104474] TEST-46-HOMED.sh[470]: + for i in {1..10}
[  103.122251] TEST-46-HOMED.sh[470]: + ((  i > 1  ))
[  103.123953] TEST-46-HOMED.sh[470]: + sleep 0.5
[  103.613585] TEST-46-HOMED.sh[2470]: + grep -qF 'State: inactive'
[  103.621653] TEST-46-HOMED.sh[2469]: + homectl inspect homedsshtest
[  103.628423] TEST-46-HOMED.sh[470]: + for i in {1..10}
[  103.634196] TEST-46-HOMED.sh[470]: + ((  i > 1  ))
[  103.641496] TEST-46-HOMED.sh[470]: + sleep 0.5
[  104.133878] TEST-46-HOMED.sh[2473]: + grep -qF 'State: inactive'
[  104.139069] TEST-46-HOMED.sh[2472]: + homectl inspect homedsshtest
[  104.168665] TEST-46-HOMED.sh[470]: + for i in {1..10}
[  104.173556] TEST-46-HOMED.sh[470]: + ((  i > 1  ))
[  104.177962] TEST-46-HOMED.sh[470]: + sleep 0.5
[  104.673986] TEST-46-HOMED.sh[2476]: + grep -qF 'State: inactive'
[  104.676147] TEST-46-HOMED.sh[2475]: + homectl inspect homedsshtest
[  104.709866] TEST-46-HOMED.sh[470]: + for i in {1..10}
[  104.711659] TEST-46-HOMED.sh[470]: + ((  i > 1  ))
[  104.714484] TEST-46-HOMED.sh[470]: + sleep 0.5
[  105.220309] TEST-46-HOMED.sh[2479]: + grep -qF 'State: inactive'
[  105.225556] TEST-46-HOMED.sh[2478]: + homectl inspect homedsshtest
[  105.243646] TEST-46-HOMED.sh[470]: + for i in {1..10}
[  105.252630] TEST-46-HOMED.sh[470]: + ((  i > 1  ))
[  105.260127] TEST-46-HOMED.sh[470]: + sleep 0.5
[  105.743572] TEST-46-HOMED.sh[2482]: + grep -qF 'State: inactive'
[  105.750658] TEST-46-HOMED.sh[2481]: + homectl inspect homedsshtest
[  105.766781] TEST-46-HOMED.sh[470]: + for i in {1..10}
[  105.771376] TEST-46-HOMED.sh[470]: + ((  i > 1  ))
[  105.782666] TEST-46-HOMED.sh[470]: + sleep 0.5
[  106.275683] TEST-46-HOMED.sh[2484]: + homectl inspect homedsshtest
[  106.290850] TEST-46-HOMED.sh[2485]: + grep -qF 'State: inactive'
[  106.299811] TEST-46-HOMED.sh[470]: + for i in {1..10}
[  106.301413] TEST-46-HOMED.sh[470]: + ((  i > 1  ))
[  106.305652] TEST-46-HOMED.sh[470]: + sleep 0.5
[  106.793647] TEST-46-HOMED.sh[2488]: + grep -qF 'State: inactive'
[  106.801792] TEST-46-HOMED.sh[2487]: + homectl inspect homedsshtest
[  106.828029] TEST-46-HOMED.sh[470]: + for i in {1..10}
[  106.829714] TEST-46-HOMED.sh[470]: + ((  i > 1  ))
[  106.831127] TEST-46-HOMED.sh[470]: + sleep 0.5
[  107.330315] TEST-46-HOMED.sh[2491]: + grep -qF 'State: inactive'
[  107.337022] TEST-46-HOMED.sh[2490]: + homectl inspect homedsshtest
[  107.360820] TEST-46-HOMED.sh[470]: + at_exit
[  107.364439] TEST-46-HOMED.sh[470]: + set +e
[  107.368074] TEST-46-HOMED.sh[470]: + systemctl is-active -q mysshserver.socket
[  107.382914] TEST-46-HOMED.sh[470]: + systemctl stop mysshserver.socket
[  107.415514] TEST-46-HOMED.sh[470]: + rm -f /tmp/homed.id_ecdsa /run/systemd/system/[email protected] /run/systemd/system/mysshserver.socket
[  107.430518] TEST-46-HOMED.sh[470]: + systemctl daemon-reload
[  108.302878] TEST-46-HOMED.sh[470]: + homectl remove homedsshtest
[  108.342469] TEST-46-HOMED.sh[470]: + for dir in /etc /usr/lib
[  108.357316] TEST-46-HOMED.sh[470]: + [[ -f /etc/pam.d/sshd.bak ]]
[  108.358960] TEST-46-HOMED.sh[470]: + mv /etc/pam.d/sshd.bak /etc/pam.d/sshd
[  108.363485] TEST-46-HOMED.sh[470]: + for dir in /etc /usr/lib
[  108.365106] TEST-46-HOMED.sh[470]: + [[ -f /usr/lib/pam.d/sshd.bak ]]
[  109.849754] reboot: Power down
Finishing after writing 148199 entries
Test failed, relevant logs can be viewed with: 

gh run download 9227184474 --name ci-mkosi-9227184474-1-arch-rolling-failed-test-journals -D ci/ci-mkosi-9227184474-1-arch-rolling-failed-test-journals && journalctl --file ci/ci-mkosi-9227184474-1-arch-rolling-failed-test-journals/test/journal/TEST-46-HOMED.journal --no-hostname -o short-monotonic -u TEST-46-HOMED.service -p info