Koder Crashes On Startup about koder

[waddlesplash]

So, we have pinpointed the problem as being somewhere in the new syslibs on 32-bit. Paging @nielx, perhaps he might have some ideas for what we can investigate next; all C++ exception handling being broken sounds like a pretty serious regression...

[waddlesplash]

Could it be that a static libgcc is linked in somewhere?

We do ship a static libgcc.a it appears, yes. But we do the exact same on x86_64. So what's the difference? Should we indeed not ship a static one?

[lonemadmax]

I had and Koder still didn't work. I've tried again but this time I ran the test program. That one worked.
Recompiling both yaml_cpp and koder solves this issue, at least with an up to date nightly.

[nielx]

@jackburton79 You are correct.

And to be even more absolutist, libroot.so depends on libgcc_s.so, so nothing build and run on Haiku can ever include the static libgcc. @waddlesplash asked earlier if we perhaps should stop shipping libgcc_s.a. I am inclined to make that change to prevent this situation from occurring in the future.

[KapiX]

Can you attach the file? Maybe there are some weird characters that are not recognized.

[SamuraiCrow]

On 32-bit x86 it crashes with the same exception when there is no file. On 64-bit nothing unusual goes wrong.

[waddlesplash]

Can you attach the file? Maybe there are some weird characters that are not recognized.

The file doesn't actually exist.

[waddlesplash]

I guess this is probably somehow caused by the GCC 13 upgrade, though I don't know what the mechanism for that would be.

[lonemadmax]

FWIW, a simple program with

try {
    const YAML::Node languages = YAML::LoadFile(std::string(argv[1])); 
} catch (...) {
}

fails the same way, but works in hrev57197 with GCC 11. Moreover, both the GCC 11 and GCC 13 compiled executables work with gcc_x86_syslibs-11.2 and fail with gcc_x86_syslibs-13.2.

[nielx]

Could it be that a static libgcc is linked in somewhere? The issue smells a bit like this issue and the comment about dwarf unwinding pitfalls: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105507#c5

[waddlesplash]

@lonemadmax did you try recompiling the YAML library? I guess we should look into how it links.

[waddlesplash]

OK. Did you check what libgcc is respectively used for each binary?

[jackburton79]

That means that forcing a new yaml_cpp package version would fix the problem?

[lonemadmax]

Did you check what libgcc is respectively used for each binary?

I wouldn't know how, this low-level stuff is way out of my knowledge. lddtree shows the shared library, but then if they also took code from the static one I guess I wouldn't see it? They have different names libgcc_s.so and libgcc.a, so it may be possible?

When I recompiled yesterday I did it to set -v flag in case I could see something in the logs, but the only difference I noticed is the new compiler configure options (jit and enable-host-shared). Both 11.2 and 13.2 call collect2 with -plugin-opt=-pass-through=-lgcc_s and have -lstdc++ -lgcc_s -lroot -lgcc_s at the end when linking libyaml-cpp.so. There's also -shared-libgcc in COLLECT_GCC_OPTIONS, but again that's with both versions.

That means that forcing a new yaml_cpp package version would fix the problem?

It should be enough for the sample program in your issue, but may need more if that was just a reduced case of your real problem. For this one, a new build of koder would also be needed. And then I've been playing a bit more today and seen that scintilla also has some exception handling. It would also need a rebuild if we don't want that to crash (for example: use regex search with a bad regex). So I don't know where this ends.

[jackburton79]

Sorry, I didn't express it very we'll.
I meant if we recompile all the packages against the new GCC syslib the problem would go away.
But that means we have a binary compatibility issue.
If I read that thread correctly, It seems we shouldn't mix GCC shared lib with GCC static lib.

[waddlesplash]

Have we actually proved that static libgcc is linked in and that's the problem here?

[lonemadmax]

Unlikely. I've compiled with an empty libgcc.a (and seen that instead of the system one when passing the -t option to the linker) with the same result.

Another step into reducing the test case. Just this without extra libraries is enough to crash in the exception case when run with one syslibs version and compiled in the other:

void test(int n) {
	if (n > 1)
		throw n;
}

int main(int argc, char** argv) {
	try {
		test(argc);
	} catch (...) {
		return 1;
	}
	return 0;
}

Not only that. Compiling to assembly with one version and then assembling, linking and running in the other gets the same faulty behaviour. Maybe it has nothing to do with linkage (remember I don't know what I'm talking about here). The difference in produced assembly is:

--- minimum.s11 2023-11-27 20:07:59.000000000 +0100
+++ minimum.s13 2023-11-27 20:07:59.000000000 +0100
@@ -7,9 +7,9 @@
        .cfi_startproc
        pushl   %ebp
        .cfi_def_cfa_offset 8
-       .cfi_offset 5, -8
+       .cfi_offset 4, -8
        movl    %esp, %ebp
-       .cfi_def_cfa_register 5
+       .cfi_def_cfa_register 4
        pushl   %ebx
        subl    $4, %esp
        .cfi_offset 3, -12
@@ -34,9 +34,9 @@
        movl    -4(%ebp), %ebx
        movl    %ebp, %esp
        .cfi_restore 3
-       .cfi_def_cfa_register 4
+       .cfi_def_cfa_register 5
        popl    %ebp
-       .cfi_restore 5
+       .cfi_restore 4
        .cfi_def_cfa_offset 4
        ret
        .cfi_endproc
@@ -55,15 +55,15 @@
        pushl   -4(%ecx)
        pushl   %ebp
        movl    %esp, %ebp
-       .cfi_escape 0x10,0x5,0x2,0x75,0
+       .cfi_escape 0x10,0x4,0x2,0x74,0
        pushl   %edi
        pushl   %esi
        pushl   %ebx
        pushl   %ecx
-       .cfi_escape 0xf,0x3,0x75,0x70,0x6
-       .cfi_escape 0x10,0x7,0x2,0x75,0x7c
-       .cfi_escape 0x10,0x6,0x2,0x75,0x78
-       .cfi_escape 0x10,0x3,0x2,0x75,0x74
+       .cfi_escape 0xf,0x3,0x74,0x70,0x6
+       .cfi_escape 0x10,0x7,0x2,0x74,0x7c
+       .cfi_escape 0x10,0x6,0x2,0x74,0x78
+       .cfi_escape 0x10,0x3,0x2,0x74,0x74
        subl    $8, %esp
        call    __x86.get_pc_thunk.si
        addl    $_GLOBAL_OFFSET_TABLE_, %esi
@@ -104,9 +104,9 @@
        popl    %edi
        .cfi_restore 7
        popl    %ebp
-       .cfi_restore 5
+       .cfi_restore 4
        leal    -4(%ecx), %esp
-       .cfi_def_cfa 4, 4
+       .cfi_def_cfa 5, 4
        ret
        .cfi_endproc
 .LFE1:
@@ -168,4 +168,4 @@
        .size   DW.ref.__gxx_personality_v0, 4
 DW.ref.__gxx_personality_v0:
        .long   __gxx_personality_v0
-       .ident  "GCC: (2021_07_28) 11.2.0"
+       .ident  "GCC: (2023_08_10) 13.2.0"

[waddlesplash]

Those are some pretty different call frame informations. It looks like some setting might have changed in the newer GCC? Well, anyway, that looks like the source of the problem.

Since the issue only happens on 32-bit, can you check on 64-bit what the difference (if any) in the assembly is?

[lonemadmax]

can you check on 64-bit what the difference (if any) in the assembly is?

Just the .ident line.

[waddlesplash]

That's what I expected. So, something changed on 32-bit to alter the CFI generation. Did GCC change a default? Did we modify the specs? (Does dumpspecs and other related GCC flags reveal differences?)

[lonemadmax]

--- dumpspecs11     2023-11-28 11:58:44.000000000 +0100
+++ dumpspecs13     2023-11-28 11:58:44.000000000 +0100
@@ -2,10 +2,10 @@
 
 
 *asm_debug:
-%{%:debug-level-gt(0):%{gstabs*:--gstabs;:%{g*:}}} %{fdebug-prefix-map=*:--debug-prefix-map %*}
+%{g*:%{%:debug-level-gt(0):}} %{ffile-prefix-map=*:--debug-prefix-map %*} %{fdebug-prefix-map=*:--debug-prefix-map %*}
 
 *asm_debug_option:
-%{%:debug-level-gt(0):%{!gstabs*:%{g*:%{%:dwarf-version-gt(4):--gdwarf-5 ;%:dwarf-version-gt(3):--gdwarf-4 ;%:dwarf-version-gt(2):--gdwarf-3 ;:--gdwarf2 }}}}
+%{g*:%{%:debug-level-gt(0):%{%:dwarf-version-gt(4):--gdwarf-5 ;%:dwarf-version-gt(3):--gdwarf-4 ;%:dwarf-version-gt(2):--gdwarf-3 ;:--gdwarf2 }}}
 
 *asm_final:
 %{gsplit-dwarf: 
@@ -13,7 +13,7 @@
        objcopy --strip-dwo      %{c:%{o*:%*}%{!o*:%w%b%O}}%{!c:%U%O}     }
 
 *asm_options:
-%{-target-help:%:print-asm-header()} %{v} %{w:-W} %{I*} %(asm_debug_option) %{gz|gz=zlib:--compress-debug-sections=zlib} %{gz=none:--compress-debug-sections=none} %{gz=zlib-gnu:--compress-debug-sections=zlib-gnu} %a %Y %{c:%W{o*}%{!o*:-o %w%b%O}}%{!c:-o %d%w%u%O}
+%{-target-help:%:print-asm-header()} %{v} %{w:-W} %{I*} %(asm_debug_option) %{gz|gz=zlib:--compress-debug-sections=zlib} %{gz=none:--compress-debug-sections=none} %{gz=zlib-gnu:}%a %Y %{c:%W{o*}%{!o*:-o %w%b%O}}%{!c:-o %d%w%u%O}
 
 *invoke_as:
 %{!fwpa*:   %{fcompare-debug=*|fdump-final-insns=*:%:compare-debug-dump-opt()}   %{!S:-o %|.s |
@@ -71,7 +71,7 @@
 0
 
 *version:
-11.2.0
+13.2.0
 
 *multilib:
 . ;
@@ -159,7 +159,7 @@
      %{!nostdlib:%{!nodefaultlibs:%:pass-through-libs(%(link_gcc_c_sequence))}}
      }}%{flto|flto=*:%<fcompare-debug*}
      %{flto} %{fno-lto} %{flto=*} %l %{static|shared|r:;pie:-pie} %{fuse-ld=*:-fuse-ld=%*}
-  %{gz|gz=zlib:--compress-debug-sections=zlib} %{gz=none:--compress-debug-sections=none} %{gz=zlib-gnu:--compress-debug-sections=zlib-gnu} %X %{o*} %{e*} %{N} %{n} %{r}
+  %{gz|gz=zlib:--compress-debug-sections=zlib} %{gz=none:--compress-debug-sections=none} %{gz=zstd:--compress-debug-sections=zstd} %{gz=zlib-gnu:}%X %{o*} %{e*} %{N} %{n} %{r}
     %{s} %{t} %{u*} %{z} %{Z} %{!nostdlib:%{!r:%{!nostartfiles:%S}}}
      %{static|no-pie|static-pie:} %@{L*} %(mfwrap) %(link_libgcc) %{fvtable-verify=none:} %{fvtable-verify=std:
    %e-fvtable-verify=std is not supported in this configuration} %{fvtable-verify=preinit:

The last chunk comes from *link_command:, which is in fact a very long line. I have added newlines before spans of multiple spaces to better see the difference there, but I've not removed spaces in case they are significant.

The only change in gcc -v is the addition of jit and --enable-host-shared in the 'Configured with:' line.

Forcing the specs of the other version doesn't change anything.

[waddlesplash]

Yeah, nothing of relevance in there.

[jackburton79]

Not even the (link_libgcc) which is there only for GCC 13 ?

[jackburton79]

Not even the (link_libgcc) which is there only for GCC 13 ?

Sorry, I misread the diff

[lonemadmax]

Compiling with -fomit-frame-pointer makes an executable that crashes in a gcc 13 system and works in the gcc 11 one, whatever compiler version was used.

[waddlesplash]

Perhaps the problem isn't the GCC 13 upgrade, then, but the change to the omit-frame-pointer defaults that we did in the GCC specs?

[lonemadmax]

If you mean https://git.haiku-os.org/buildtools/commit/?id=ef1ae536062efb3fba8e9389da934689a0c608e3, that's supposed to not change anything in x86, I think? I've rebuilt the gcc packages without that anyway and still get the same result.

[waddlesplash]

Should we escalate this issue into one on Haiku's own bugtracker and make it a beta5 blocker? Broken exception handling in 32-bit builds seems bad.

[jackburton79]

Should we escalate this issue into one on Haiku's own bugtracker and make it a beta5 blocker? Broken exception handling in 32-bit builds seems bad.

I think so.

[waddlesplash]

Done: https://dev.haiku-os.org/ticket/18825

[korli]

I tried to reproduce on x86 with #147 (comment) on r1~beta4_hrev56578_93, without success. I tried again on a clean upgraded r1beta4 iso, same result.
Is there a trick to reproduce?

[waddlesplash]

As the message says, you have to compile with GCC 11 and then run under GCC 13 syslibs or vice versa. The problem is an ABI break somehow.

[korli]

As the message says, you have to compile with GCC 11 and then run under GCC 13 syslibs or vice versa. The problem is an ABI break somehow.

From the comment "both the GCC 11 and GCC 13 compiled executables work with gcc_x86_syslibs-11.2 and fail with gcc_x86_syslibs-13.2.", I would understand the compiling with GCC 13 makes it possible to reproduce.

[waddlesplash]

Hmm, maybe so. Well, I can't reproduce when compiling with GCC 13 either, however Koder still crashes on startup.

[lonemadmax]

"both the GCC 11 and GCC 13 compiled executables work with gcc_x86_syslibs-11.2 and fail with gcc_x86_syslibs-13.2."

That's further in the past, still trying to see if this was Koder or the YAML library.

While I don't remember the details, this is how I read it:
library + caller program compiled with v11 or v13 work with syslibs 11
library + caller program compiled with v11 or v13 crash with syslibs 13
And I think the library back then was compiled with v11.

A later comment seems to add that:
library recompiled with v13 + caller with v11 on syslibs 13 crash
library with v13 + caller with v13 on syslibs 13 work

The small test in #147 (comment) is both the thrower and the catcher, and has to run with a different syslibs version than compiled to crash.

[jackburton79]

The crash happened in Genio, too, so it wasnt limited to Koder

[korli]

fixed in 039cb8c4f98a3b2b71e6576e77a851d937cf7fd3 All GCC13 artefacts are possibly problematic, and would need a rebuild unfortunately.