Unable to use mold with a (pseudo) cross compiler about mold

[nehaljwani]

Thank you! For now, mold works the --sysroot fixes 🙂

[rui314]

0.9.3 has been released, and the fix for this bug is in that release.

[nehaljwani]

Locations tried by mold for libgcc.a:

(gccdev) /t/nwani_1625000438 ❯❯❯ grep stat /tmp/o | grep libgcc.a
4124815 stat("/home/wani/m3/envs/gccdev/bin/../x86_64-conda-linux-gnu/sysroot/tmp/libgcc.a",  <unfinished ...>
4124815 stat("/home/wani/m3/envs/gccdev/bin/../x86_64-conda-linux-gnu/sysroot/home/wani/m3/envs/gccdev/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/libgcc.a",  <unfinished ...>
4124815 stat("/home/wani/m3/envs/gccdev/bin/../x86_64-conda-linux-gnu/sysroot/home/wani/m3/envs/gccdev/bin/../lib/gcc/libgcc.a",  <unfinished ...>
4124815 stat("/home/wani/m3/envs/gccdev/bin/../x86_64-conda-linux-gnu/sysroot/home/wani/m3/envs/gccdev/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/lib/../lib/libgcc.a",  <unfinished ...>
4124815 stat("/home/wani/m3/envs/gccdev/bin/../x86_64-conda-linux-gnu/sysroot/home/wani/m3/envs/gccdev/bin/../x86_64-conda-linux-gnu/sysroot/lib/../lib/libgcc.a",  <unfinished ...>
4124815 stat("/home/wani/m3/envs/gccdev/bin/../x86_64-conda-linux-gnu/sysroot/home/wani/m3/envs/gccdev/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib/../lib/libgcc.a",  <unfinished ...>
4124815 stat("/home/wani/m3/envs/gccdev/bin/../x86_64-conda-linux-gnu/sysroot/home/wani/m3/envs/gccdev/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/lib/libgcc.a",  <unfinished ...>
4124815 stat("/home/wani/m3/envs/gccdev/bin/../x86_64-conda-linux-gnu/sysroot/home/wani/m3/envs/gccdev/bin/../x86_64-conda-linux-gnu/sysroot/lib/libgcc.a",  <unfinished ...>
4124815 stat("/home/wani/m3/envs/gccdev/bin/../x86_64-conda-linux-gnu/sysroot/home/wani/m3/envs/gccdev/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib/libgcc.a",  <unfinished ...>

Actual location for libgcc.a:

/home/wani/m3/envs/gccdev/lib/gcc/x86_64-conda-linux-gnu/9.3.0/libgcc.a

[nehaljwani]

Arguments being passed to mold (/tmp/ld):

[
  "/tmp/ld",
  "-plugin",
  "/home/wani/m3/envs/gccdev/bin/../libexec/gcc/x86_64-conda-linux-gnu/9.3.0/liblto_plugin.so",
  "-plugin-opt=/home/wani/m3/envs/gccdev/bin/../libexec/gcc/x86_64-conda-linux-gnu/9.3.0/lto-wrapper",
  "-plugin-opt=-fresolution=/tmp/ccNSOfal.res",
  "-plugin-opt=-pass-through=-lgcc",
  "-plugin-opt=-pass-through=-lgcc_s",
  "-plugin-opt=-pass-through=-lc",
  "-plugin-opt=-pass-through=-lgcc",
  "-plugin-opt=-pass-through=-lgcc_s",
  "--sysroot=/home/wani/m3/envs/gccdev/bin/../x86_64-conda-linux-gnu/sysroot",
  "--eh-frame-hdr",
  "-m",
  "elf_x86_64",
  "-dynamic-linker",
  "/lib64/ld-linux-x86-64.so.2",
  "-pie",
  "/home/wani/m3/envs/gccdev/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib/../lib/Scrt1.o",
  "/home/wani/m3/envs/gccdev/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib/../lib/crti.o",
  "/home/wani/m3/envs/gccdev/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/crtbeginS.o",
  "-L/tmp",
  "-L/home/wani/m3/envs/gccdev/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0",
  "-L/home/wani/m3/envs/gccdev/bin/../lib/gcc",
  "-L/home/wani/m3/envs/gccdev/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/lib/../lib",
  "-L/home/wani/m3/envs/gccdev/bin/../x86_64-conda-linux-gnu/sysroot/lib/../lib",
  "-L/home/wani/m3/envs/gccdev/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib/../lib",
  "-L/home/wani/m3/envs/gccdev/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/lib",
  "-L/home/wani/m3/envs/gccdev/bin/../x86_64-conda-linux-gnu/sysroot/lib",
  "-L/home/wani/m3/envs/gccdev/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib",
  "-rpath",
  "/home/wani/m3/envs/gccdev/lib",
  "/tmp/cchfMAtK.o",
  "-lgcc",
  "--push-state",
  "--as-needed",
  "-lgcc_s",
  "--pop-state",
  "-lc",
  "-lgcc",
  "--push-state",
  "--as-needed",
  "-lgcc_s",
  "--pop-state",
  "/home/wani/m3/envs/gccdev/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/crtendS.o",
  "/home/wani/m3/envs/gccdev/bin/../x86_64-conda-linux-gnu/sysroot/usr/lib/../lib/crtn.o"
]

The path containing libgcc.a is present in the library directories above. 🤔

[nehaljwani]

lld and gold (the ones not shipped with the cross tool chain) behave as expected:

(gccdev) /t/nwani_1625000438 ❯❯❯ cp /bin/ld.lld /tmp
(gccdev) /t/nwani_1625000438 ❯❯❯ cp /bin/ld.gold /tmp
(gccdev) /t/nwani_1625000438 ❯❯❯ echo 'int main() {return 0;}' | $CC -fuse-ld=lld -xc - -B/tmp
(gccdev) /t/nwani_1625000438 ❯❯❯ echo 'int main() {return 0;}' | $CC -fuse-ld=gold -xc - -B/tmp

It seems that mold is pre-pending the value from --sysroot=<> to all values passed as -L<> arguments

[nehaljwani]

From the man page of ld:

If searchdir begins with "=" or $SYSROOT, then this prefix will be replaced by the sysroot prefix, controlled by the --sysroot option, or specified when the linker is configured.

And test cases of lld:

// Should substitute SysRoot if specified
// RUN: ld.lld -o %t/r %t/m.o --sysroot=%t -L=lib -l:libls.a
// RUN: ld.lld -o %t/r %t/m.o --sysroot=%t -L=/lib -l:libls.a
// Check alias.
// RUN: ld.lld -o %t/r %t/m.o --sysroot %t -L=lib -l:libls.a

// Should not substitute SysRoot if the directory name does not start with '='
// RUN: not ld.lld -o /dev/null %r/m.o --sysroot=%t -Llib -l:libls.a
// RUN: not ld.lld -o /dev/null %r/m.o --sysroot=%t -L/lib -l:libls.a

[rui314]

Thank you for the report! Confirmed that mold's behavior is not correct. Fixing the issue...

[nehaljwani]

Unfortunately, 6432114 doesn't completely fix the --sysroot issue. See below:

$ echo 'int main(){return 0;}' | $CC -xc - -B$PWD 
mold: /path/to/prefix/bin/../x86_64-conda_cos6-linux-gnu/sysroot/usr/lib/../lib/libc.so:5: GROUP ( /lib/libc.so.6 /usr/lib/libc_nonshared.a  AS_NEEDED ( /lib/ld-linux-x86-64.so.2 ) ) 
                                                                                                                            ^ library not found: /lib/libc.so.6
collect2: error: ld returned 1 exit status

$ $CC --print-sysroot
/path/to/prefix/bin/../x86_64-conda_cos6-linux-gnu/sysroot

$ readlink -f $($CC --print-sysroot)/lib/libc.so.6
/path/to/prefix/x86_64-conda_cos6-linux-gnu/sysroot/lib/libc-2.12.2.so

Here is what gold is doing after reading the script:

2469057 openat(AT_FDCWD, "/path/to/prefix/bin/../x86_64-conda_cos6-linux-gnu/sysroot/usr/lib/../lib/libc.so", O_RDONLY|O_CLOEXEC) = 13  
2469057 fstat(13, {st_mode=S_IFREG|0664, st_size=247, ...}) = 0 
2469057 mmap(NULL, 247, PROT_READ, MAP_PRIVATE, 13, 0) = 0x7f62fd692000
2469057 lseek(13, 0, SEEK_SET)          = 0
2469057 read(13, "/* GNU ld script", 16) = 16  
2469057 openat(AT_FDCWD, "/path/to/prefix/bin/../x86_64-conda_cos6-linux-gnu/sysroot/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 14

Here is what lld is doing after reading the script:

2470162 openat(AT_FDCWD, "/path/to/prefix/bin/../x86_64-conda_cos6-linux-gnu/sysroot/usr/lib/../lib/libc.so", O_RDONLY|O_CLOEXEC) = 3 
2470162 fstat(3, {st_mode=S_IFREG|0664, st_size=247, ...}) = 0 
2470162 pread64(3, "/* GNU ld script\n   Use the shared library, but some functions are only in\n   the static library, so try that secondarily.  */\nOUTPUT_FORMAT(elf64-x86-64)\nGROUP ( /lib/libc.so.6 /usr/lib/libc_nonshared.a  AS_NEEDED ( /lib/ld-linux-x86-64.so.2 ) )\n", 247, 0) = 247
2470162 close(3)                        = 0
2470162 stat("/path/to/prefix/bin/../x86_64-conda_cos6-linux-gnu/sysroot", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 
2470162 stat("/path/to/prefix/bin/../x86_64-conda_cos6-linux-gnu/sysroot/usr/lib/../lib/libc.so", {st_mode=S_IFREG|0664, st_size=247, ...}) = 0 
2470162 stat("/path/to/prefix/bin/../x86_64-conda_cos6-linux-gnu/sysroot", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 
2470162 stat("/path/to/prefix/bin/../x86_64-conda_cos6-linux-gnu/sysroot/usr/lib/../lib", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 
2470162 stat("/path/to/prefix/bin/../x86_64-conda_cos6-linux-gnu/sysroot", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 
2470162 stat("/path/to/prefix/bin/../x86_64-conda_cos6-linux-gnu/sysroot/usr/lib/..", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 
2470162 stat("/path/to/prefix/bin/../x86_64-conda_cos6-linux-gnu/sysroot", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 
2470162 stat("/path/to/prefix/bin/../x86_64-conda_cos6-linux-gnu/sysroot/usr/lib", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 
2470162 stat("/path/to/prefix/bin/../x86_64-conda_cos6-linux-gnu/sysroot", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 
2470162 stat("/path/to/prefix/bin/../x86_64-conda_cos6-linux-gnu/sysroot/usr", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 
2470162 stat("/path/to/prefix/bin/../x86_64-conda_cos6-linux-gnu/sysroot", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 
2470162 stat("/path/to/prefix/bin/../x86_64-conda_cos6-linux-gnu/sysroot", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 
2470162 access("/path/to/prefix/bin/../x86_64-conda_cos6-linux-gnu/sysroot/lib/libc.so.6", F_OK) = 0 
2470162 openat(AT_FDCWD, "/path/to/prefix/bin/../x86_64-conda_cos6-linux-gnu/sysroot/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 3 

[nehaljwani]

Easily reproducible with the test case in the repo:

diff --git a/test/sysroot-linker-script.sh b/test/sysroot-linker-script.sh
index 6518f01..beff937 100755
--- a/test/sysroot-linker-script.sh
+++ b/test/sysroot-linker-script.sh
@@ -22,6 +22,6 @@ void foo();
 int main() { foo(); }
 EOF
 
-clang -o $t/exe $t/c.o -Wl,--sysroot=$t/ $t/foo/bar/b.script
+clang -fuse-ld=`pwd`/../mold -o $t/exe $t/c.o -Wl,--sysroot=$t/ $t/foo/bar/b.script
 
 echo OK

$ ./test/sysroot-linker-script.sh 
Testing sysroot-linker-script ... mold: /path/to/mold/test/tmp/sysroot-linker-script/foo/bar/b.script:1: INPUT(/foo/bar/libfoo.a)
                                                                                        ^ library not found: /foo/bar/libfoo.a
clang: error: linker command failed with exit code 1 (use -v to see invocation)

[rui314]

I hope this change really fixes the issue. Feel free to reopen if it doesn't.

[nehaljwani]

The --sysroot fixes didn't get cherry-picked to the 0.9.2 release. 😭 Is it going to land in the next minor?

[rui314]

Ah, sorry, I missed that one if it is not in 0.9.2. Do you need that in a hurry? The next release will be a full release, so it will include everything.

[nehaljwani]

No hurry, I'll gladly wait for the full release.

[nehaljwani]

Seems like 0.9.3 is coming up. Can that one get these changes? 🙂

[rui314]

That change was a little bit risky, so I set it aside and included only changes to tests in 0.9.3-pre1, so that I don't need to test mold again with lots of Gentoo packages. Do you need the sysroot patches for creating binary package or something?

[nehaljwani]

I maintain hundreds of Conda packages in the conda-forge community.

The Conda ecosystem uses a GCC cross compiler on Linux so that the entire toolchain is relocatable to any path prefix and can be used on any distro and can always target a specified kernel + glibc version. So I can't use mold for Conda packages on Linux unless --sysroot is properly supported.

That being said, I was just looking at 0.9.3 as an opportunity. If it is too much work, I'll wait. Being a maintainer is not an easy task and I appreciate all the time and effort you put in.

On a side note, could you release the scripts you use for your Gentoo build out?

[rui314]

Sure, thank you for the explanation, and thank you for maintaining lots of packages! I'll try to cherrypick the changes for sysroot in 0.9.3.

[rui314]

I created a new release candidate which include the sysroot fix: https://github.com/rui314/mold/tree/v0.9.3-rc2