Boot issue on dedicated server with talos 1.7.0 since 1.7.0-alpha.1 (1.7.0-alpha.0 worked) about talos

[smira]

I wonder if https://cateee.net/lkddb/web-lkddb/EFI_DISABLE_PCI_DMA.html might be the issue here, it was enabled in alpha.1 version.

So I guess the experiment is to add efi=no_disable_early_pci_dma to the kernel command line might fix it.

[smira]

Talos 1.7.2 will have this fix included.

[buroa]

Thanks for this @smira. This pretty much breaks a ton of EFI boot processes. I saw this on the Mac minis as well and patched it inside my builds.

[smira]

This is a tough issue to look into.

There might be two issues here:

• still something with console, as EFI stub is printed via EFI console, and all other messages are printed by the kernel via its own console driver, so I would double-check that, the best is to enter GRUB menu and check that there are no console args at all
• something very incompatible that prevents Linux to boot, but I haven't seen such reports so far. I looked through other changes to the kernel config, and I don't see anything. Does this machine work with other Linux distros which use Linux 6.6+ kernel?

[smira]

P.S. If it's possible, you could try booting in BIOS (non-UEFI) mode to see if that works.

[WinterNis]

Thanks for your answers. This is a tough one indeed.

• We did enter grub and made sure there were no console arguments at all.  We still had the issue.
• We were able to access the bios (non-UEFI). Not sure what we can do in here to help though. Exiting the bios then result in the same boot issue.
• We are able to install ubuntu 24.04 on the server, which is running kernel v6.8.
• We tried on another scale a1 server (same specs, different server). We had the same issue.

I haven't seen such reports so far

As far as I understand, #8657 does report compatibility issues no ?

[smira]

As far as I understand, #8657 does report compatibility issues no ?

This issue seems to be two issues actually, one is console args (there's an issue with Linux kernel that if console=ttyS0 is specified, and there's no serial port, Linux kernel hangs on boot the same way as you report), and another one which is related to the kernel panicking early on boot (but that is on QEMU).

• We were able to access the bios (non-UEFI). Not sure what we can do in here to help though. Exiting the bios then result in the same boot issue.

There should be an option to boot in "legacy mode" (or something like that), which disables UEFI completely. I'm just curious if this is related to UEFI or not. Talos should work both ways, but still.

[frezbo]

I wonder if https://cateee.net/lkddb/web-lkddb/EFI_DISABLE_PCI_DMA.html might be the issue here, it was enabled in alpha.1 version.

So I guess the experiment is to add efi=no_disable_early_pci_dma to the kernel command line might fix it.

that could be it, since it broke booting on arm64

[WinterNis]

Well, that actually solved the problem.

Passing efi=no_disable_early_pci_dma AND removing console arguments did the trick.

With the efi argument, but without removing the console arguments, we had the same issue.

Thanks guys, you truly sniped this 🙏

What’s the best way of "fixing" this ? Should we add the arguments in our custom images and that’s it ? Or do you consider removing the disable because of compatibility issues ?

[smira]

What’s the best way of "fixing" this ? Should we add the arguments in our custom images and that’s it ? Or do you consider removing the disable because of compatibility issues ?

yes, you can do a custom kernel arg for now, and I believe it would still be fine (ignored) if we disable it by default in the kernel config.

[smira]

I will actually remove that kernel option, whoever wants that could do a kernel arg to enforce it, but e.g. Ubuntu doesn't enable it by default.

[smira]

console fix is expected in 1.8

[WinterNis]

That’s good news. Thanks again for your help guys, truly appreciated 🙏

I will let you guys close this issue or keep it open until the fix are landed.