Coder Social home page Coder Social logo

michaeleisel / zld Goto Github PK

View Code? Open in Web Editor NEW
1.2K 35.0 50.0 35.18 MB

A faster version of Apple's linker

License: MIT License

Shell 0.19% Ruby 0.02% HTML 0.14% Roff 0.48% C++ 85.86% C 3.84% Perl 0.11% Makefile 4.93% Objective-C 0.07% Assembly 0.42% DTrace 0.01% Python 0.43% Starlark 1.39% CMake 0.95% NASL 1.17%
ios linker macos

zld's Introduction

NOTE: zld is now archived, consider using lld instead. More info is here.

zld

A faster version of Apple's linker

Introduction

For large projects, the linking phase (explanation) can significantly increase incremental build times. This project is a fork of the Apple linker, ld. It is a drop-in replacement that can substantially speed things up.

Note: it is only intended for debug builds, to make debugging faster.

Performance

Feel free to file an issue if you find that linking is not at least 40% faster for your case (make sure to run it twice in a row to ensure that caches have been generated). Further benchmark details can be found here.

Is it worth it for my project?

It all depends on your risk tolerance and how much you value the speedup in incremental build time. When linking takes more than one second, I'd cut that time in half as the estimated time with this new linker. If that difference is compelling to you, then it'd be worth trying out. Personally, I'd use it in projects with an existing link time of even 500ms (but I am an impatient tinkerer).

Stability

zld is forked from the most recently open-sourced version of ld. It's used by thousands of developers across many of the largest apps in the world. Without a few optimizations around hashing, it would produce byte-for-byte the same executables as the open-source one. Although it's not ideal to mix compiler and linker toolchain versions, the open-source one is fairly recent. zld will continue to be updated with each new version of ld as it is released.

Installation

Below are the installation methods. Note that, if you someday install a newer version of Xcode and zld doesn't work with it, it may be that your version of zld is now too far behind that of the linker shipped in Xcode. In that case, check back here for the latest zld release to fix the problem.

Note: The Xcode app must be installed for zld to work (not just the Xcode command line tools)

Pre-built binary

The pre-built binary for the latest release is here.

Building from source

  • sudo xcode-select -s <path to Xcode>
  • Install cmake
  • Checkout the latest release of zld from master
  • Run make clean && make
  • See in the output where it built zld (probably build/Build/Products/Release/zld).

Usage

If using Xcode:

Get the path of zld from which zld, then add -fuse-ld=<path to zld> -Wl,-zld_original_ld_path,$(DT_TOOLCHAIN_DIR)/usr/bin/ld to "Other Linker Flags" in the build settings (debug configuration). That -zld_original_ld_path provides the path to the linker Xcode would otherwise use, which is important because there are certain known cases (e.g. arm64_32 and Catalyst) where zld knows that it has issues and will silently use that linker instead. Fixing these cases is a work-in-progress (largely blocked by Apple being slow to release source code).

If using Bazel:

Add these to your .bazelrc or pass them to your command line.

build --linkopt=-fuse-ld=<path to zld>
build --linkopt=-Wl,-zld_original_ld_path,__BAZEL_XCODE_DEVELOPER_DIR__/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld

Note that you will need to disable sandbox for this to work now. Additionally, to make the linking actions cacheable, the path to zld must be deterministic (e.g. /tmp/zld-09ea158, where 09ea158 is zld version).

Another option to use zld in Bazel is via rules_apple_linker.

If using Rust:

You can edit ~/.cargo/config to add a linker flag, e.g.:

[target.x86_64-apple-darwin]
# For Apple silicon:
# [target.aarch64-apple-darwin]

rustflags = ["-C", "link-arg=-fuse-ld=<path to zld>"]

Caching

By default, zld stores some metadata in /tmp/zld-... to speed things up. This is the first step towards making zld a truly incremental linker. Currently, the only things that are stored are object file and library names.

Why is it faster?

Apple's approach is a very reasonable one, using C++ with STL data structures. However, there are a number of ways in which zld has sped things up, for instance:

  • Using Swiss Tables instead of STL for hash maps and sets
  • Parallelizing in various places (the parsing of libraries, writing the output file, sorting, etc.)
  • Optimizations around the hashing of strings (caching the hashes, using a better hash function, etc.)

Other things to speed up linking

Whether you use this project or not, there are a number of things that can speed linking up (again, this is only for debug builds):

  • The linker flag -Wl,-random_uuid, which disables content hashing based UUID creation and instead uses a random UUID (using -no_uuid can decrease performance when lldb is attaching to the binary)
  • Turning off dead stripping (referred to as "Dead Code Stripping" in Xcode build settings)
  • For executables and xctest bundles, disable dyld exports trie creation with -Wl,-exported_symbols_list,/dev/null (cannot be used by test host apps that provide symbols to xctests)
  • -Wl,-no_deduplicate, which disables the deduplication pass. In Xcode, this flag is added by default for Debug builds.
  • -Wl,-no_compact_unwind, which disables the creation of compact unwind info. Note that Objective-C and C++ functions that have been linked without their compact unwind info can crash if an exception unwinds into them, rather than continuing the unwind. Swift functions, however, are not affected (even if an Objective-C/C++ exception unwinds into them). So, it's best for pure Swift projects. It can also break crash reporting.
  • If you're not using zld, using -Wl,-force_load for libraries can sometimes speed things up
  • Linking with dynamic libraries instead of static ones

Contributing

The biggest way to contribute to zld is to file issues! If you encountered any problems, feel free to file an issue, and you can expect a prompt response.

Special thanks to @dmaclach's ld64, which helped with building ld.

zld's People

Contributors

crazyfanfan avatar dmaclach avatar jamescoleuk avatar kastiglione avatar keith avatar michaeleisel avatar rmaz avatar rockwotj avatar steipete avatar thii avatar trouv avatar woshiccm avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

zld's Issues

Cannot build from source

I'm trying to build from source the project following the instructions in the README file but I have several issues.

Building from source
Install Xcode 12.2+ and run sudo xcode-select -s <path to Xcode>
Install cmake
Checkout the latest release of zld from master
Run make clean && make

I attached in the following gist the log files. It seems that running the command make clean && make the compilation fails for some missing dependency library. If I run make again without cleaning the compilation makes some steps again but fails without finding corecrypto dependencies (that gets deleted on the clean, is it correct?).

https://gist.github.com/lechuckcaptain/aafdc3877caa2fb8a7f10454af4f4336

I also tried to reset the repo, and run only the make command without any clean. I gets at first the same error of missing some dependency library, but on the second make I get a different error this time. Please find some info in this second gist:

https://gist.github.com/lechuckcaptain/49624151b8974f88576b6e661c873418

Do you have any idea what could be the issue?

Xcode 12 support

according to someone at apple, there's no clear timeline on when the source code for tapi, ld, etc. will ship. i'll wait until end of september for it to happen, and if it hasn't happened, i'll begin reverse-engineering the tapi changes and try to get a stable version for xcode 12 that doesn't have the spurious warnings. @rmaz @keith @milend

Compile error on Big Sur with latest XCode

Last 15 lines from /Users/dvtkrlbs/Library/Logs/Homebrew/zld/02.make:
[100%] Built target flags_parse
find abseil-cpp-20200225/build_arm64/absl -name '*.a' | xargs libtool -static -o abseil-cpp-20200225/build/libabsl_arm64.a
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: warning same member name (escaping.cc.o) in output file used for input files: abseil-cpp-20200225/build_arm64/absl/strings/libabsl_strings.a(escaping.cc.o) and: abseil-cpp-20200225/build_arm64/absl/strings/libabsl_strings_internal.a(escaping.cc.o) due to use of basename, truncation and blank padding
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: warning same member name (flag.cc.o) in output file used for input files: abseil-cpp-20200225/build_arm64/absl/flags/libabsl_flags_internal.a(flag.cc.o) and: abseil-cpp-20200225/build_arm64/absl/flags/libabsl_flags.a(flag.cc.o) due to use of basename, truncation and blank padding
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: warning same member name (usage.cc.o) in output file used for input files: abseil-cpp-20200225/build_arm64/absl/flags/libabsl_flags_usage.a(usage.cc.o) and: abseil-cpp-20200225/build_arm64/absl/flags/libabsl_flags_usage_internal.a(usage.cc.o) due to use of basename, truncation and blank padding
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: file: abseil-cpp-20200225/build_arm64/absl/strings/libabsl_strings.a(string_view.cc.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: file: abseil-cpp-20200225/build_arm64/absl/types/libabsl_bad_optional_access.a(bad_optional_access.cc.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: file: abseil-cpp-20200225/build_arm64/absl/types/libabsl_bad_any_cast_impl.a(bad_any_cast.cc.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: file: abseil-cpp-20200225/build_arm64/absl/types/libabsl_bad_variant_access.a(bad_variant_access.cc.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: file: abseil-cpp-20200225/build_arm64/absl/flags/libabsl_flags.a(flag.cc.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: file: abseil-cpp-20200225/build_arm64/absl/debugging/libabsl_debugging_internal.a(elf_mem_image.cc.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: file: abseil-cpp-20200225/build_arm64/absl/debugging/libabsl_debugging_internal.a(vdso_support.cc.o) has no symbols
lipo -create abseil-cpp-20200225/build/libabsl_x86_64.a abseil-cpp-20200225/build/libabsl_arm64.a -output abseil-cpp-20200225/build/libabsl.a
fatal error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo: abseil-cpp-20200225/build/libabsl_x86_64.a and abseil-cpp-20200225/build/libabsl_arm64.a have the same architectures (x86_64) and can't be in the same fat output file
make: *** [abseil-cpp-20200225] Error 1

Add a CI job

Once #9 is resolved it would be great to set up a CI job to verify changes.

Thorough testing

this is just a ticket to brainstorm testing

fyi the tests are currently:

  • link the "signal", "storage", and "XUL" files used in the benchmarks, then ensure they're byte-for-byte the same as ones linked from a previous version
  • run the tests in unit-tests, and see that aren't any more failures than with the open-source original). either i'm doing something wrong in setting them up, or they aren't maintained? they seem valuable but need care and feeding
  • run the mozilla app, having linked all its libraries with zld, and see that it looks halfway decent

machochecker, a target in the .xcodeproj, seems helpful for doing some checks. it would be very cool if there's some way to test binary "equivalency", i.e. compensating for differences in ordering etc. and compare it against newest (and thus still closed-source) ld64.

an easy thing to do is just test linking more and more open-source libraries for that byte-for-byte compatibility. certainly there are lots of big ones out there, like llvm

Mac Catalyst linker issues

zld works great for us overall. We cannot yet enable it for Mac Catalyst. It seems the linker pulls off some tricks to differentiate between AppKit and iOS-versions of the frameworks.

We see the following error:

Building for Mac Catalyst, but linking in .tbd built for , file '/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/System/Library/Frameworks//CoreImage.framework/CoreImage.tbd' for architecture x86_64

This might be fixed once #40 is resolved.

Thank you for zld! Gave the project a shoutout on my blog.

SIGKILL for Rust projects with build.rs on M1 Macs

I was trying to use zld on an M1 Mac for those sweet link time improvements, but I'm running into this crash while building:

error: failed to run custom build command for `libc v0.2.93`

Caused by:
  process didn't exit successfully: `/Users/ian/Workspace/koi/target/debug/build/libc-dbe33ec8d2132c51/build-script-build` (signal: 9, SIGKILL: kill)

It seems any crate with a build.rs file hits this while building.

@benmkw mentioned this issue as well: #73 (comment)

Unfortunately this makes zld unusable for most projects with Rust / M1 Macs.

ASAN issue

While testing an internal build that used unordered_map instead of absl::flat_map, we got a crash when running under ASAN. Unclear if this is some related to the previous issue of parallel sorting:

=================================================================
==10131==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6130002727fb at pc 0x00010d671173 bp 0x70000ca68e90 sp 0x70000ca68620
READ of size 87 at 0x6130002727fb thread T22
    #0 0x10d671172 in wrap_strlcpy (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x54172)
    #1 0x10a4c3d98 in ld::tool::StringPoolAtom::add(char const*) (zld:x86_64+0x10036ed98)
    #2 0x10a4c3ee5 in ld::tool::StringPoolAtom::add(char const*) (zld:x86_64+0x10036eee5)
    #3 0x10a4c4505 in ld::tool::StringPoolAtom::addUnique(char const*) (zld:x86_64+0x10036f505)
    #4 0x10a58027f in ld::tool::SymbolTableAtom<x86_64>::encode() (zld:x86_64+0x10042b27f)
    #5 0x7fff3cffb7bc in __NSBLOCKOPERATION_IS_CALLING_OUT_TO_A_BLOCK__ (Foundation:x86_64+0x427bc)
    #6 0x7fff3cffb6b4 in -[NSBlockOperation main] (Foundation:x86_64+0x426b4)
    #7 0x7fff3cffb63f in __NSOPERATION_IS_INVOKING_MAIN__ (Foundation:x86_64+0x4263f)
    #8 0x7fff3cffa833 in -[NSOperation start] (Foundation:x86_64+0x41833)
    #9 0x7fff3cffa54d in __NSOPERATIONQUEUE_IS_STARTING_AN_OPERATION__ (Foundation:x86_64+0x4154d)
    #10 0x7fff3cffa417 in __NSOQSchedule_f (Foundation:x86_64+0x41417)
    #11 0x7fff71fc350d in _dispatch_client_callout (libdispatch.dylib:x86_64+0x350d)
    #12 0x7fff71fc5c20 in _dispatch_block_invoke_direct (libdispatch.dylib:x86_64+0x5c20)
    #13 0x10d67d6e5 in __wrap_dispatch_async_block_invoke (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x606e5)
    #14 0x7fff71fc2582 in _dispatch_call_block_and_release (libdispatch.dylib:x86_64+0x2582)
    #15 0x7fff71fc350d in _dispatch_client_callout (libdispatch.dylib:x86_64+0x350d)
    #16 0x7fff71fc56bf in _dispatch_continuation_pop (libdispatch.dylib:x86_64+0x56bf)
    #17 0x7fff71fc4dbd in _dispatch_async_redirect_invoke (libdispatch.dylib:x86_64+0x4dbd)
    #18 0x7fff71fd17e1 in _dispatch_root_queue_drain (libdispatch.dylib:x86_64+0x117e1)
    #19 0x7fff71fd1f21 in _dispatch_worker_thread2 (libdispatch.dylib:x86_64+0x11f21)
    #20 0x7fff7221d6b5 in _pthread_wqthread (libsystem_pthread.dylib:x86_64+0x26b5)
    #21 0x7fff7221c826 in start_wqthread (libsystem_pthread.dylib:x86_64+0x1826)

0x6130002727fb is located 0 bytes to the right of 379-byte region [0x613000272680,0x6130002727fb)
allocated by thread T20 here:
    #0 0x10d67eb17 in wrap_realloc (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x61b17)
    #1 0x7fff720c9a16 in reallocf (libsystem_c.dylib:x86_64+0x5ea16)
    #2 0x7fff720a6791 in __sfvwrite (libsystem_c.dylib:x86_64+0x3b791)
    #3 0x7fff720b0026 in __vfprintf (libsystem_c.dylib:x86_64+0x45026)
    #4 0x7fff720d3e1a in __v2printf (libsystem_c.dylib:x86_64+0x68e1a)
    #5 0x7fff720ac199 in _vasprintf (libsystem_c.dylib:x86_64+0x41199)
    #6 0x10d646ebb in wrap_vasprintf (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x29ebb)
    #7 0x10d64758c in wrap_asprintf (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x2a58c)
    #8 0x10a2e822b in mach_o::relocatable::Parser<x86_64>::parseDebugInfo() (zld:x86_64+0x10019322b)
    #9 0x10a2dd807 in mach_o::relocatable::Parser<x86_64>::parse(mach_o::relocatable::ParserOptions const&) (zld:x86_64+0x100188807)
    #10 0x10a2b3852 in mach_o::relocatable::Parser<x86_64>::parse(unsigned char const*, unsigned long long, char const*, long, ld::File::Ordinal, mach_o::relocatable::ParserOptions const&) (zld:x86_64+0x10015e852)
    #11 0x10a2b2e82 in mach_o::relocatable::parse(unsigned char const*, unsigned long long, char const*, long, ld::File::Ordinal, mach_o::relocatable::ParserOptions const&) (zld:x86_64+0x10015de82)
    #12 0x10a3aa282 in archive::File<x86_64>::makeObjectFileForMember(archive::File<x86_64>::Entry const*) const (zld:x86_64+0x100255282)
    #13 0x10a46389d in tbb::interface9::internal::start_for<tbb::blocked_range<unsigned long>, ld::tool::InputFiles::preParseLibraries() const::$_3, tbb::auto_partitioner const>::execute() (zld:x86_64+0x10030e89d)
    #14 0x10a8bdce1 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::process_bypass_loop(tbb::internal::context_guard_helper<false>&, tbb::task*, long) (zld:x86_64+0x100768ce1)
    #15 0x10a8bd5da in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all(tbb::task&, tbb::task*) (zld:x86_64+0x1007685da)
    #16 0x10a8b8b06 in tbb::internal::arena::process(tbb::internal::generic_scheduler&) (zld:x86_64+0x100763b06)
    #17 0x10a8b850c in tbb::internal::market::process(rml::job&) (zld:x86_64+0x10076350c)
    #18 0x10a8b5263 in tbb::internal::rml::private_worker::run() (zld:x86_64+0x100760263)
    #19 0x10a8b51b2 in tbb::internal::rml::private_worker::thread_routine(void*) (zld:x86_64+0x1007601b2)
    #20 0x7fff72220e64 in _pthread_start (libsystem_pthread.dylib:x86_64+0x5e64)
    #21 0x7fff7221c83a in thread_start (libsystem_pthread.dylib:x86_64+0x183a)

Thread T22 created by T0 here:
    <empty stack>

Thread T20 created by T13 here:
    #0 0x10d67678d in wrap_pthread_create (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x5978d)
    #1 0x10a8b5910 in rml::internal::thread_monitor::launch(void* (*)(void*), void*, unsigned long) (zld:x86_64+0x100760910)
    #2 0x10a8b56da in tbb::internal::rml::private_worker::wake_or_launch() (zld:x86_64+0x1007606da)
    #3 0x10a8b563f in tbb::internal::rml::private_server::wake_some(int) (zld:x86_64+0x10076063f)
    #4 0x10a8b51d8 in tbb::internal::rml::private_worker::run() (zld:x86_64+0x1007601d8)
    #5 0x10a8b51b2 in tbb::internal::rml::private_worker::thread_routine(void*) (zld:x86_64+0x1007601b2)
    #6 0x7fff72220e64 in _pthread_start (libsystem_pthread.dylib:x86_64+0x5e64)
    #7 0x7fff7221c83a in thread_start (libsystem_pthread.dylib:x86_64+0x183a)

Thread T13 created by T12 here:
    #0 0x10d67678d in wrap_pthread_create (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x5978d)
    #1 0x10a8b5910 in rml::internal::thread_monitor::launch(void* (*)(void*), void*, unsigned long) (zld:x86_64+0x100760910)
    #2 0x10a8b56da in tbb::internal::rml::private_worker::wake_or_launch() (zld:x86_64+0x1007606da)
    #3 0x10a8b563f in tbb::internal::rml::private_server::wake_some(int) (zld:x86_64+0x10076063f)
    #4 0x10a8b51d8 in tbb::internal::rml::private_worker::run() (zld:x86_64+0x1007601d8)
    #5 0x10a8b51b2 in tbb::internal::rml::private_worker::thread_routine(void*) (zld:x86_64+0x1007601b2)
    #6 0x7fff72220e64 in _pthread_start (libsystem_pthread.dylib:x86_64+0x5e64)
    #7 0x7fff7221c83a in thread_start (libsystem_pthread.dylib:x86_64+0x183a)

Thread T12 created by T11 here:
    #0 0x10d67678d in wrap_pthread_create (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x5978d)
    #1 0x10a8b5910 in rml::internal::thread_monitor::launch(void* (*)(void*), void*, unsigned long) (zld:x86_64+0x100760910)
    #2 0x10a8b56da in tbb::internal::rml::private_worker::wake_or_launch() (zld:x86_64+0x1007606da)
    #3 0x10a8b563f in tbb::internal::rml::private_server::wake_some(int) (zld:x86_64+0x10076063f)
    #4 0x10a8b51d8 in tbb::internal::rml::private_worker::run() (zld:x86_64+0x1007601d8)
    #5 0x10a8b51b2 in tbb::internal::rml::private_worker::thread_routine(void*) (zld:x86_64+0x1007601b2)
    #6 0x7fff72220e64 in _pthread_start (libsystem_pthread.dylib:x86_64+0x5e64)
    #7 0x7fff7221c83a in thread_start (libsystem_pthread.dylib:x86_64+0x183a)

Thread T11 created by T10 here:
    #0 0x10d67678d in wrap_pthread_create (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x5978d)
    #1 0x10a8b5910 in rml::internal::thread_monitor::launch(void* (*)(void*), void*, unsigned long) (zld:x86_64+0x100760910)
    #2 0x10a8b56da in tbb::internal::rml::private_worker::wake_or_launch() (zld:x86_64+0x1007606da)
    #3 0x10a8b563f in tbb::internal::rml::private_server::wake_some(int) (zld:x86_64+0x10076063f)
    #4 0x10a8b51d8 in tbb::internal::rml::private_worker::run() (zld:x86_64+0x1007601d8)
    #5 0x10a8b51b2 in tbb::internal::rml::private_worker::thread_routine(void*) (zld:x86_64+0x1007601b2)
    #6 0x7fff72220e64 in _pthread_start (libsystem_pthread.dylib:x86_64+0x5e64)
    #7 0x7fff7221c83a in thread_start (libsystem_pthread.dylib:x86_64+0x183a)

Thread T10 created by T0 here:
    #0 0x10d67678d in wrap_pthread_create (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x5978d)
    #1 0x10a8b5910 in rml::internal::thread_monitor::launch(void* (*)(void*), void*, unsigned long) (zld:x86_64+0x100760910)
    #2 0x10a8b56da in tbb::internal::rml::private_worker::wake_or_launch() (zld:x86_64+0x1007606da)
    #3 0x10a8b563f in tbb::internal::rml::private_server::wake_some(int) (zld:x86_64+0x10076063f)
    #4 0x10a8baacf in tbb::internal::generic_scheduler::local_spawn(tbb::task*, tbb::task*&) (zld:x86_64+0x100765acf)
    #5 0x10a4631a6 in tbb::interface9::internal::start_for<tbb::blocked_range<unsigned long>, ld::tool::InputFiles::preParseLibraries() const::$_3, tbb::auto_partitioner const>::execute() (zld:x86_64+0x10030e1a6)
    #6 0x10a8bdce1 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::process_bypass_loop(tbb::internal::context_guard_helper<false>&, tbb::task*, long) (zld:x86_64+0x100768ce1)
    #7 0x10a8bd5da in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all(tbb::task&, tbb::task*) (zld:x86_64+0x1007685da)
    #8 0x10a8bafb9 in tbb::internal::generic_scheduler::local_spawn_root_and_wait(tbb::task*, tbb::task*&) (zld:x86_64+0x100765fb9)
    #9 0x10a43f6b0 in ld::tool::InputFiles::preParseLibraries() const (zld:x86_64+0x1002ea6b0)
    #10 0x10a43c8b7 in ld::tool::InputFiles::forEachInitialAtom(ld::File::AtomHandler&, ld::Internal&) (zld:x86_64+0x1002e78b7)
    #11 0x10a4b4dcd in ld::tool::Resolver::resolve() (zld:x86_64+0x10035fdcd)
    #12 0x10a277e23 in main (zld:x86_64+0x100122e23)
    #13 0x7fff7201c7fc in start (libdyld.dylib:x86_64+0x1a7fc)

SUMMARY: AddressSanitizer: heap-buffer-overflow (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x54172) in wrap_strlcpy
Shadow bytes around the buggy address:
  0x1c260004e4a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1c260004e4b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1c260004e4c0: 00 00 00 00 00 00 00 00 fa fa fa fa fa fa fa fa
  0x1c260004e4d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1c260004e4e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x1c260004e4f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00[03]
  0x1c260004e500: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
  0x1c260004e510: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1c260004e520: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1c260004e530: 00 00 00 00 00 00 fa fa fa fa fa fa fa fa fa fa
  0x1c260004e540: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==10131==ABORTING
clang: error: unable to execute command: Abort trap: 6
clang: error: linker command failed due to signal (use -v to see invocation)

Large Binary Instruments Profile Data

I've recorded profiling data (zld.trace.zip) for a large binary which takes several minutes to link. In terms of numbers:

  • ld64
    • 225.64s user 17.44s system 94% cpu 4:17.81 total
    • 219.63s user 17.33s system 93% cpu 4:14.00 total
    • 221.44s user 21.93s system 88% cpu 4:35.26 total
  • zld
    • 213.75s user 14.14s system 118% cpu 3:12.50 total
    • 218.63s user 16.22s system 115% cpu 3:22.62 total
    • 213.71s user 14.74s system 118% cpu 3:11.99 total

zld is around 1min faster (~23% faster). Machine was rebooted before taking the measurements and the samples taken with hot file caches (i.e., linkers ran before recording the samples above). Machine is a 15" MacBook Pro 2017, i7 3.1GHz, 16GB RAM and 1TB SSD.

Daemonization

This ticket is to discuss at a high-level using daemonization to improve linking time. Currently, we do some file-system caching, but that requires both the complexity and CPU overhead of serialization. Here are some questions:

  • would daemonization cause any hassles with build systems at scale?
  • what can we daemonize?
  • based on what we can daemonize, would zld perform better in a distributed build system if it used file caching rather than daemonization?

The first question should be relatively easy for anyone who manages a large-scale build system to answer for their own case. The others are very exploratory.

Installation Docs improvements

If the build process requires cmake to be installed then the installation docs in the readme should list installing cmake and adding it to your PATH as one of the steps.

Also including an example for the default xcode path might be helpful to newer MacOS developers like myself:
sudo xcode-select -s /Applications/Xcode.app/Contents/Developer

Open call for people's linker profilings

The linker has many ways of using it, and many potential bottlenecks. By sending me a profiling (using the time profiler) of zld, and ideally one of stock ld as well, I can improve the linker for your exact use case!

In addition, if it's possible for your case, it'd be nice to see the invocation (helpful) and/or get the full set of input files to reproduce on my end (extremely helpful)

Provide a build script

To aid with outside contributions, it would be great if there was a simple build script to fetch all the dependencies and build zld. There are also a bunch of absolute paths in the project config that need some fixup.

Integrate ld-530

Apple has released ld-520 and ld-530 (thanks to @rmaz for pointing this out), so we can integrate those changes

Transparent fallback on error to Apple ld

Since zld failing at scale can be very bad for dev productivity, perhaps it would be good if, whenever zld errors out, it tries invoking Apple ld with the same arguments

ARM Mac Support

I get the following when building on m1 arm mac:

      ...
  "absl::lts_2020_02_25::container_internal::UnsampleSlow(absl::lts_2020_02_25::container_internal::HashtablezInfo*)", referenced from:
      absl::lts_2020_02_25::flat_hash_map<ld::Atom const*, unsigned long, absl::lts_2020_02_25::container_internal::HashEq<ld::Atom const*, void>::Hash, absl::lts_2020_02_25::container_internal::HashEq<ld::Atom const*, void>::Eq, std::__1::allocator<std::__1::pair<ld::Atom const* const, unsigned long> > >::~flat_hash_map() in code_dedup.o
      ld::passes::dedup::doPass(Options const&, ld::Internal&) in code_dedup.o
      absl::lts_2020_02_25::container_internal::raw_hash_set<absl::lts_2020_02_25::container_internal::FlatHashMapPolicy<ld::Atom const*, unsigned long>, absl::lts_2020_02_25::container_internal::HashEq<ld::Atom const*, void>::Hash, absl::lts_2020_02_25::container_internal::HashEq<ld::Atom const*, void>::Eq, std::__1::allocator<std::__1::pair<ld::Atom const* const, unsigned long> > >::resize(unsigned long) in code_dedup.o
      absl::lts_2020_02_25::container_internal::raw_hash_set<absl::lts_2020_02_25::container_internal::FlatHashMapPolicy<ld::Atom const*, std::__1::vector<ld::Atom const*, std::__1::allocator<ld::Atom const*> > >, ld::passes::dedup::atom_hashing, ld::passes::dedup::atom_equal, std::__1::allocator<std::__1::pair<ld::Atom const* const, std::__1::vector<ld::Atom const*, std::__1::allocator<ld::Atom const*> > > > >::resize(unsigned long) in code_dedup.o
      absl::lts_2020_02_25::container_internal::raw_hash_set<absl::lts_2020_02_25::container_internal::FlatHashMapPolicy<ld::Atom const*, ld::Atom const*>, absl::lts_2020_02_25::container_internal::HashEq<ld::Atom const*, void>::Hash, absl::lts_2020_02_25::container_internal::HashEq<ld::Atom const*, void>::Eq, std::__1::allocator<std::__1::pair<ld::Atom const* const, ld::Atom const*> > >::resize(unsigned long) in code_dedup.o
      generic::dylib::File<x86_64>::File(char const*, long, ld::File::Ordinal, ld::VersionSet const&, bool, bool, bool, bool, bool) in textstub_dylib_file.o
      generic::dylib::File<x86_64>::~File() in textstub_dylib_file.o
      ...
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

** BUILD FAILED **

It seems the issue is basically just that the target is set to x86 instead of arm so the fix could maybe be simple. Unfortunately I don't really know much about Xcode or how you set up the build yet. If you have a diff/ branch/ something I could/ should try I'm happy to do that and give feedback/ give more information.

Graceful fallbacks for Xcode 12 and arm64_32

It may take some time to have proper tapi support for Xcode 12, and arm64_32 support has long been missing. It would be good to gracefully fall back to normal ld in these cases.

Recommending the -x flag

@bogardon informed me that the -x flag was speeding up linking for him. Indeed, it reduces linking time for me on a sample project by ~5%. In the man page, it says that the flag means, Do not put non-global symbols in the output file's symbol table. However, when I stop at a breakpoint with LLDB, I'm still able to call static functions in the debugger just as easily with the flag as without. 5% would be nice, but can anyone shed some light on the implications of this flag?

zld is missing the force_load fix in Xcode 11

Xcode 10.2 introduced a linker bug that caused linking Swift in static libraries to fail:

https://developer.apple.com/documentation/xcode_release_notes/xcode_10_2_release_notes/swift_5_release_notes_for_xcode_10_2?language=objc

Linking against a static Swift library might create a binary with missing type metadata because the object files that define the metadata inside the static archive are mistakenly considered unused. (47598583)

This can manifest as a Swift runtime error with a message such as: โ€œfailed to demangle superclass of MyClass from mangled name โ€˜โ€™โ€.

Workaround: If you can rebuild the static library, try building it with whole module optimization enabled. Otherwise, add -all_load to the linker flags in the client binary to ensure all object files are linked into it.

This was fixed in Xcode 11:
https://developer.apple.com/documentation/xcode_release_notes/xcode_11_release_notes?language=objc

Static libraries are now always force-loaded in their entirety during linking, fixing most โ€œunable to demangleโ€ runtime errors. (47598583)

We will need to reimplement this fix. My suspicion is the solution was not as simple as described, that they in fact only added force_load for static libraries that contained Swift symbols. A quote from the Swift bug ticket has:

You should be safe if you force load any .a file that contains symbols matching the regex ^_?$s.*Mn$ (that is, beginning with the prefix $s or _$s, and ending with the suffix Mn)

compile failed on Apple M1

Run make but got an error:

error: unknown target CPU 'armv8-a+crypto'
note: valid target CPU values are: nocona, core2, penryn, bonnell, atom, silvermont,
      slm, goldmont, goldmont-plus, tremont, nehalem, corei7, westmere, sandybridge,
      corei7-avx, ivybridge, core-avx-i, haswell, core-avx2, broadwell, skylake,
      skylake-avx512, skx, cascadelake, cooperlake, cannonlake, icelake-client,
      icelake-server, tigerlake, knl, knm, k8, athlon64, athlon-fx, opteron, k8-sse3,
      athlon64-sse3, opteron-sse3, amdfam10, barcelona, btver1, btver2, bdver1, bdver2,
      bdver3, bdver4, znver1, znver2, x86-64

Any plan to support compiling on M1? :)

zld broken with Xcode 12.5 beta 3 because of clang response file support

It seems like the version of clang that shipped with Xcode 12.5b3 (notably not with 12.5b1) contains support for passing arguments via response files to ld64. This appears to result in zld failing because support for response files was added in ld64 with Xcode 12, which isn't on https://opensource.apple.com yet. The failure looks like this:

ld: file not found: @/var/folders/h6/6btvg47n7y1f5xxz4_n9rsd40000gn/T/response-1c3fac.txt

I imagine it's because it's looking for this path literally. When you run clang with -v you can see it's passed via a response file (in this example to ld64 directly, not zld):

 "/Applications/Xcode-12.5.0-beta3.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld" @/var/folders/h6/6btvg47n7y1f5xxz4_n9rsd40000gn/T/response-4652cc.txt
 Arguments passed via response file:
...

Tracking issue for lld's new mach-o backend

Although lld has been historically unmaintained, it seems like there's some new work on it with https://reviews.llvm.org/D75382 . lld, on linux at least, has a good track-record in terms of performance, so it will be interesting to see:

  • will they support all the same things ld64 does?
  • will it have the same API as ld64, making it a drop-in replacement?

At the very least, it should provide some ideas on how to optimize zld further

Potential optimizations

The -no_uuid suggestion led me to relook at an Instruments trace of our project. In addition to -no_uuid (which was ~3.5% in the project I looked at), I found some other flags that may be useful optimizations. I was wondering if you've tried any of these:

-no_deduplicate

The dedup pass is the most expensive pass in our project, around 8%. For functions, this seems fine, but does this result in duplicate data symbols? That would be a problem for mutable data. I presume some of the time saved here is paid back at runtime (larger binary).

-no_compact_unwind

The compact_unwind is the next slowest pass in my profile, 3.2%. Our project is all Swift, no C++ and so I don't think we care about compact unwind info (for developer builds at least).

Other flags I added, that don't seem to have saved a lot of time in our project: -no_order_inits, -no_order_data, -no_function_starts.

duplicate symbol

there is "duplicate symbol" error when use zld, these symbols are in different lib. However, there is only warning when use the origin linker.

zld fails to link (or fallbacks to ld) when linking Watch app with bitcode enabled

/Applications/Xcode-12.0.0-gm-seed.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang -target arm64-apple-watchos7.0-simulator -isysroot /Applications/Xcode-12.0.0-gm-seed.app/Contents/Developer/Platforms/WatchSimulator.platform/Developer/SDKs/WatchSimulator7.0.sdk -L/Users/admin/Library/Developer/Xcode/DerivedData/App2-cncnyrokyfrdldfczjlkcqyvxcfc/Build/Products/Debug-watchsimulator -F/Users/admin/Library/Developer/Xcode/DerivedData/App2-cncnyrokyfrdldfczjlkcqyvxcfc/Build/Products/Debug-watchsimulator -filelist /Users/admin/Library/Developer/Xcode/DerivedData/App2-cncnyrokyfrdldfczjlkcqyvxcfc/Build/Intermediates.noindex/App2.build/Debug-watchsimulator/App2\ WatchKit\ Extension.build/Objects-normal/arm64/App2\ WatchKit\ Extension.LinkFileList -Xlinker -rpath -Xlinker @executable_path/Frameworks -Xlinker -rpath -Xlinker @executable_path/../../Frameworks -dead_strip -Xlinker -object_path_lto -Xlinker /Users/admin/Library/Developer/Xcode/DerivedData/App2-cncnyrokyfrdldfczjlkcqyvxcfc/Build/Intermediates.noindex/App2.build/Debug-watchsimulator/App2\ WatchKit\ Extension.build/Objects-normal/arm64/App2\ WatchKit\ Extension_lto.o -Xlinker -export_dynamic -Xlinker -no_deduplicate -Xlinker -objc_abi_version -Xlinker 2 -fembed-bitcode-marker -fapplication-extension -fobjc-link-runtime -L/Applications/Xcode-12.0.0-gm-seed.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/watchsimulator -L/usr/lib/swift -Xlinker -add_ast_path -Xlinker /Users/admin/Library/Developer/Xcode/DerivedData/App2-cncnyrokyfrdldfczjlkcqyvxcfc/Build/Intermediates.noindex/App2.build/Debug-watchsimulator/App2\ WatchKit\ Extension.build/Objects-normal/arm64/App2_WatchKit_Extension.swiftmodule -fuse-ld=/tmp/zld-09ea158 -Wl,-zld_original_ld_path,/Applications/Xcode-12.0.0-gm-seed.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld -e _WKExtensionMain -framework WatchKit -Xlinker -sectcreate -Xlinker __TEXT -Xlinker __entitlements -Xlinker /Users/admin/Library/Developer/Xcode/DerivedData/App2-cncnyrokyfrdldfczjlkcqyvxcfc/Build/Intermediates.noindex/App2.build/Debug-watchsimulator/App2\ WatchKit\ Extension.build/App2\ WatchKit\ Extension.appex-Simulated.xcent -Xlinker -dependency_info -Xlinker /Users/admin/Library/Developer/Xcode/DerivedData/App2-cncnyrokyfrdldfczjlkcqyvxcfc/Build/Intermediates.noindex/App2.build/Debug-watchsimulator/App2\ WatchKit\ Extension.build/Objects-normal/arm64/App2\ WatchKit\ Extension_dependency_info.dat -o /Users/admin/Library/Developer/Xcode/DerivedData/App2-cncnyrokyfrdldfczjlkcqyvxcfc/Build/Intermediates.noindex/App2.build/Debug-watchsimulator/App2\ WatchKit\ Extension.build/Objects-normal/arm64/Binary/App2\ WatchKit\ Extension

ld: in '/usr/lib/system/libsystem_kernel.dylib', building for watchOS Simulator, but linking in .tbd file (/usr/lib/system/libsystem_kernel.dylib) built for macOS/Mac Catalyst, for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Error Library not loaded.

I am trying to use it for Rust.

rustc 1.49.0-nightly (25c8c53dd 2020-10-03)

I added it in ~/.cargo/config

[target.x86_64-apple-darwin]
rustflags = ["-C", "link-arg=-fuse-ld=/usr/local/bin/zld"]

And as soon as i do cargo run it gives error

โžœ  r5ai git:(main) โœ— cargo run
   Compiling libc v0.2.78
   Compiling proc-macro2 v1.0.24
   Compiling syn v1.0.42
   Compiling log v0.4.11
error: linking with `cc` failed: exit code: 254
  |
  = note: "cc" "-m64" "-arch" "x86_64" "-L" "/Users/shirshak/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib" "/Users/shirshak/Desktop/code/r5ai/target/debug/build/log-24c71d92410457b2/build_script_build-24c71d92410457b2.build_script_build.tdatf7av-cgu.0.rcgu.o" "/Users/shirshak/Desktop/code/r5ai/target/debug/build/log-24c71d92410457b2/build_script_build-24c71d92410457b2.build_script_build.tdatf7av-cgu.1.rcgu.o" "/Users/shirshak/Desktop/code/r5ai/target/debug/build/log-24c71d92410457b2/build_script_build-24c71d92410457b2.build_script_build.tdatf7av-cgu.10.rcgu.o" "/Users/shirshak/Desktop/code/r5ai/target/debug/build/log-24c71d92410457b2/build_script_build-24c71d92410457b2.build_script_build.tdatf7av-cgu.11.rcgu.o" "/Users/shirshak/Desktop/code/r5ai/target/debug/build/log-24c71d92410457b2/build_script_build-24c71d92410457b2.build_script_build.tdatf7av-cgu.12.rcgu.o" "/Users/shirshak/Desktop/code/r5ai/target/debug/build/log-24c71d92410457b2/build_script_build-24c71d92410457b2.build_script_build.tdatf7av-cgu.13.rcgu.o" "/Users/shirshak/Desktop/code/r5ai/target/debug/build/log-24c71d92410457b2/build_script_build-24c71d92410457b2.build_script_build.tdatf7av-cgu.14.rcgu.o" "/Users/shirshak/Desktop/code/r5ai/target/debug/build/log-24c71d92410457b2/build_script_build-24c71d92410457b2.build_script_build.tdatf7av-cgu.15.rcgu.o" "/Users/shirshak/Desktop/code/r5ai/target/debug/build/log-24c71d92410457b2/build_script_build-24c71d92410457b2.build_script_build.tdatf7av-cgu.2.rcgu.o" "/Users/shirshak/Desktop/code/r5ai/target/debug/build/log-24c71d92410457b2/build_script_build-24c71d92410457b2.build_script_build.tdatf7av-cgu.3.rcgu.o" "/Users/shirshak/Desktop/code/r5ai/target/debug/build/log-24c71d92410457b2/build_script_build-24c71d92410457b2.build_script_build.tdatf7av-cgu.4.rcgu.o" "/Users/shirshak/Desktop/code/r5ai/target/debug/build/log-24c71d92410457b2/build_script_build-24c71d92410457b2.build_script_build.tdatf7av-cgu.5.rcgu.o" "/Users/shirshak/Desktop/code/r5ai/target/debug/build/log-24c71d92410457b2/build_script_build-24c71d92410457b2.build_script_build.tdatf7av-cgu.6.rcgu.o" "/Users/shirshak/Desktop/code/r5ai/target/debug/build/log-24c71d92410457b2/build_script_build-24c71d92410457b2.build_script_build.tdatf7av-cgu.7.rcgu.o" "/Users/shirshak/Desktop/code/r5ai/target/debug/build/log-24c71d92410457b2/build_script_build-24c71d92410457b2.build_script_build.tdatf7av-cgu.8.rcgu.o" "/Users/shirshak/Desktop/code/r5ai/target/debug/build/log-24c71d92410457b2/build_script_build-24c71d92410457b2.build_script_build.tdatf7av-cgu.9.rcgu.o" "-o" "/Users/shirshak/Desktop/code/r5ai/target/debug/build/log-24c71d92410457b2/build_script_build-24c71d92410457b2" "/Users/shirshak/Desktop/code/r5ai/target/debug/build/log-24c71d92410457b2/build_script_build-24c71d92410457b2.1afe22bl998ag2f7.rcgu.o" "-Wl,-dead_strip" "-nodefaultlibs" "-L" "/Users/shirshak/Desktop/code/r5ai/target/debug/deps" "-L" "/Users/shirshak/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib" "/Users/shirshak/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libstd-f5a6dfa2dca04db0.rlib" "/Users/shirshak/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libpanic_unwind-a602b1ebad91d2d3.rlib" "/Users/shirshak/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libobject-f7936dd792262a74.rlib" "/Users/shirshak/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libaddr2line-bb56016a39783030.rlib" "/Users/shirshak/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libgimli-1873d4fc46d20288.rlib" "/Users/shirshak/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/librustc_demangle-55bacf2d542c4144.rlib" "/Users/shirshak/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libhashbrown-ff8a738b6f2b559a.rlib" "/Users/shirshak/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/librustc_std_workspace_alloc-3864ce8d4cb7a483.rlib" "/Users/shirshak/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libunwind-de47433f56869db6.rlib" "/Users/shirshak/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libcfg_if-1571a545da17c3ef.rlib" "/Users/shirshak/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/liblibc-48dc7fd9359a0304.rlib" "/Users/shirshak/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/liballoc-0a2b336e241cf742.rlib" "/Users/shirshak/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/librustc_std_workspace_core-465082706ea1ede1.rlib" "/Users/shirshak/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libcore-e3cd79f6ad1173fd.rlib" "/Users/shirshak/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libcompiler_builtins-4071d15d8824a4bb.rlib" "-lSystem" "-lresolv" "-lc" "-lm" "-fuse-ld=/usr/local/bin/zld"
  = note: dyld: Library not loaded: @rpath/libLTO.dylib
            Referenced from: /usr/local/bin/zld
            Reason: image not found
          clang: error: unable to execute command: Abort trap: 6
          clang: error: linker command failed due to signal (use -v to see invocation)


error: aborting due to previous error

error: could not compile `log`

To learn more, run the command again with --verbose.

I am using it after formatting osx is there any other requirement etc it needs?

zld can't autolink libswiftCompatibility50.a

In some cases, zld isn't able to autolink libswiftCompatibility50.a and libswiftCompatibilityDynamicReplacements.a.

Undefined symbols for architecture x86_64:
  "__swift_FORCE_LOAD_$_swiftCompatibility50", referenced from:
      __swift_FORCE_LOAD_$_swiftCompatibility50_$_foo in foo.a(foo.o)
     (maybe you meant: __swift_FORCE_LOAD_$_swiftCompatibility50_$_foo)
  "__swift_FORCE_LOAD_$_swiftCompatibilityDynamicReplacements", referenced from:
      __swift_FORCE_LOAD_$_swiftCompatibilityDynamicReplacements_$_foo in foo.a(foo.o)
     (maybe you meant: __swift_FORCE_LOAD_$_swiftCompatibilityDynamicReplacements_$_foo)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

I added the sample code to reproduce this problem. https://github.com/qyang-nj/zld-autolinking-issue

Q: How to report build issues?

This is a meta issue. I have a rust project that fails to build when using zld but builds fine otherwise. Should I just open an issue with the info I have (almost none, other than the rustc error) or do you have a set of troubleshooting steps I can follow to provide more useful issue reports?

Thanks! :)

Autolinking seems to fail for libswiftAVFoundation

On Xcode 12, when -libswiftAVFoundation is auto-linked from an object file for the simulator, it's successfully able to find the library for ld but not for zld. the system library search paths for both are:

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/iphonesimulator
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator14.0.sdk/usr/lib/swift
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator14.0.sdk/usr/lib

Searching for this library, we see:

$ find . -name '*wiftAVFound*'
Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift-5.0/appletvsimulator/libswiftAVFoundation.dylib
Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift-5.0/appletvos/libswiftAVFoundation.dylib
Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift-5.0/macosx/libswiftAVFoundation.dylib
Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift-5.0/iphoneos/libswiftAVFoundation.dylib
Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift-5.0/iphonesimulator/libswiftAVFoundation.dylib
Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk/usr/lib/swift/libswiftAVFoundation.tbd
Contents/Developer/Platforms/AppleTVOS.platform/Developer/SDKs/AppleTVOS.sdk/usr/lib/swift/libswiftAVFoundation.tbd
Contents/Developer/Platforms/WatchOS.platform/Developer/SDKs/WatchOS.sdk/usr/lib/swift/libswiftAVFoundation.tbd
Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/lib/swift/libswiftAVFoundation.tbd

So, there's a matching .tbd file for each platform besides the simulator, and so iPhoneOS on the other hand works. but it's unclear how newer ld deals with this for libswiftAVFoundation without any issues

Xcode 12 .tbd file warnings

When truing out Xcode 12 to build PSPDFKit for iOS 12 we got some building for iOS Simulator, but linking in .tbd file warnings.

Here are the full details for one target:

Showing All Messages
Ld /Users/matej/Library/Developer/Xcode/DerivedData/PSPDFKit-gyewukrlcuqomubvokoacqvdmluo/Build/Products/Debug-iphonesimulator/PSPDFKitUI.framework/PSPDFKitUI normal (in target 'PSPDFKitUI.framework' from project 'PSPDFKitUI')
    cd /Users/matej/Documents/Work/PSPDFKit/PSPDFKit/iOS/PSPDFKitUI
    /Applications/Xcode-beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++ -target x86_64-apple-ios12.0-simulator -dynamiclib -isysroot /Applications/Xcode-beta.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator14.0.sdk -L/Users/matej/Library/Developer/Xcode/DerivedData/PSPDFKit-gyewukrlcuqomubvokoacqvdmluo/Build/Products/Debug-iphonesimulator -F/Users/matej/Library/Developer/Xcode/DerivedData/PSPDFKit-gyewukrlcuqomubvokoacqvdmluo/Build/Products/Debug-iphonesimulator -filelist /Users/matej/Library/Developer/Xcode/DerivedData/PSPDFKit-gyewukrlcuqomubvokoacqvdmluo/Build/Intermediates.noindex/PSPDFKitUI.build/Debug-iphonesimulator/PSPDFKitUI.framework.build/Objects-normal/x86_64/PSPDFKitUI.LinkFileList -install_name @rpath/PSPDFKitUI.framework/PSPDFKitUI -Xlinker -rpath -Xlinker /usr/lib/swift -Xlinker -rpath -Xlinker @executable_path/Frameworks -Xlinker -rpath -Xlinker @loader_path/Frameworks -Xlinker -object_path_lto -Xlinker /Users/matej/Library/Developer/Xcode/DerivedData/PSPDFKit-gyewukrlcuqomubvokoacqvdmluo/Build/Intermediates.noindex/PSPDFKitUI.build/Debug-iphonesimulator/PSPDFKitUI.framework.build/Objects-normal/x86_64/PSPDFKitUI_lto.o -Xlinker -export_dynamic -Xlinker -no_deduplicate -Xlinker -objc_abi_version -Xlinker 2 -stdlib\=libc++ -fobjc-arc -fobjc-link-runtime -fapplication-extension -fprofile-instr-generate -L/Applications/Xcode-beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/iphonesimulator -L/usr/lib/swift -Xlinker -add_ast_path -Xlinker /Users/matej/Library/Developer/Xcode/DerivedData/PSPDFKit-gyewukrlcuqomubvokoacqvdmluo/Build/Intermediates.noindex/PSPDFKitUI.build/Debug-iphonesimulator/PSPDFKitUI.framework.build/Objects-normal/x86_64/PSPDFKitUI.swiftmodule -ObjC -ObjC -Wl,-no_uuid -fuse-ld\=/Users/matej/Documents/Work/PSPDFKit/PSPDFKit/iOS/PSPDFKitUI/../../iOS/Resources/zld-detect -framework CoreServices -framework MetalKit -framework Metal /Users/matej/Library/Developer/Xcode/DerivedData/PSPDFKit-gyewukrlcuqomubvokoacqvdmluo/Build/Products/Debug-iphonesimulator/PSPDFKit.framework/PSPDFKit -framework CoreSpotlight -lxml2 -lz -lc++ -lsqlite3 -framework WebKit -weak_framework SafariServices -framework Security -framework Accelerate -framework SystemConfiguration -framework CoreImage -framework CoreTelephony -framework AudioToolbox -weak_framework QuickLook -framework AVFoundation -framework CoreGraphics -framework CoreMedia -framework CoreText -framework Foundation -framework ImageIO -framework MediaPlayer -framework MessageUI -framework QuartzCore -framework UIKit -compatibility_version 1 -current_version 1 -Xlinker -dependency_info -Xlinker /Users/matej/Library/Developer/Xcode/DerivedData/PSPDFKit-gyewukrlcuqomubvokoacqvdmluo/Build/Intermediates.noindex/PSPDFKitUI.build/Debug-iphonesimulator/PSPDFKitUI.framework.build/Objects-normal/x86_64/PSPDFKitUI_dependency_info.dat -o /Users/matej/Library/Developer/Xcode/DerivedData/PSPDFKit-gyewukrlcuqomubvokoacqvdmluo/Build/Products/Debug-iphonesimulator/PSPDFKitUI.framework/PSPDFKitUI

ld: warning: building for iOS Simulator, but linking in .tbd file (/Applications/Xcode-beta.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator14.0.sdk/usr/lib/system/libsystem_kernel.tbd) built for macOS
ld: warning: building for iOS Simulator, but linking in .tbd file (/Applications/Xcode-beta.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator14.0.sdk/usr/lib/system/libsystem_platform.tbd) built for macOS
ld: warning: building for iOS Simulator, but linking in .tbd file (/Applications/Xcode-beta.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator14.0.sdk/usr/lib/system/libsystem_pthread.tbd) built for macOS

We get the same for multiple targets:

Screen Shot 2020-06-23 at 15 39 40

We managed to silence those warnings by temporarily disabling zld when using Xcode 12 using this approach: https://twitter.com/steipete/status/1275332693576876034?s=21

I unfortunately don't have any more detailed reproduction steps and I don't know if the issue is related to #43.

No performance improvement

Hi all! :)
I was trying to use this, but I didn't see any improvement in build time. Neither with a clean build nor an incremental one.

Do you have any clue why?
Usual build time is ~70s while incremental build time is ~8s.

Document base ld64 version

First of all, this project looks awesome (although, I haven't tried it yet)!

The readme says "open-source one is a bit old (Xcode 10.2)", but doesn't say what zld uses as its base version of ld64. In the source I see a couple mentions of ld64-409.12, but the most recent release of ld64 is 450.3.

Build Error: dyld: Library not loaded: @rpath/libLTO.dylib

Thanks for building this! Looking forward to faster build times in rust!

I get the following error when I add rustflags = ["-C", "link-arg=-fuse-ld=/usr/local/bin/zld"] to my .cargo/config in a new project.

   Compiling zld-test v0.1.0 (/Users/masonf/tmp/zld-test)
error: linking with `cc` failed: exit code: 254
  |
  = note: "cc" "-m64" "-arch" "x86_64" "-L" "/Users/masonf/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib" "/Users/masonf/tmp/zld-test/target/debug/deps/zld_test.13szzbm9m7o8hf39.rcgu.o" "/Users/masonf/tmp/zld-test/target/debug/deps/zld_test.1lh18tvtd6vcclj1.rcgu.o" "/Users/masonf/tmp/zld-test/target/debug/deps/zld_test.1mmadohbzflleaqa.rcgu.o" "/Users/masonf/tmp/zld-test/target/debug/deps/zld_test.401gmkzqigm812a3.rcgu.o" "/Users/masonf/tmp/zld-test/target/debug/deps/zld_test.49qxyy7lldwzjw1n.rcgu.o" "/Users/masonf/tmp/zld-test/target/debug/deps/zld_test.4l37xpyn5no2cf6k.rcgu.o" "/Users/masonf/tmp/zld-test/target/debug/deps/zld_test.g8z62udr0eka5tu.rcgu.o" "/Users/masonf/tmp/zld-test/target/debug/deps/zld_test.opiiwg2x64dug10.rcgu.o" "-o" "/Users/masonf/tmp/zld-test/target/debug/deps/zld_test" "/Users/masonf/tmp/zld-test/target/debug/deps/zld_test.pqrzvuhhu5pet8g.rcgu.o" "-Wl,-dead_strip" "-nodefaultlibs" "-L" "/Users/masonf/tmp/zld-test/target/debug/deps" "-L" "/Users/masonf/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib" "/Users/masonf/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libstd-2eead91d91f66ffa.rlib" "/Users/masonf/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libpanic_unwind-ca64d94f6b79d639.rlib" "/Users/masonf/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libobject-f7936dd792262a74.rlib" "/Users/masonf/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libaddr2line-bb56016a39783030.rlib" "/Users/masonf/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libgimli-1873d4fc46d20288.rlib" "/Users/masonf/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/librustc_demangle-55bacf2d542c4144.rlib" "/Users/masonf/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libhashbrown-ff8a738b6f2b559a.rlib" "/Users/masonf/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/librustc_std_workspace_alloc-3864ce8d4cb7a483.rlib" "/Users/masonf/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libunwind-b55c7050747b13c5.rlib" "/Users/masonf/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libcfg_if-1571a545da17c3ef.rlib" "/Users/masonf/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/liblibc-ff818040cbe57af0.rlib" "/Users/masonf/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/liballoc-0a2b336e241cf742.rlib" "/Users/masonf/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/librustc_std_workspace_core-465082706ea1ede1.rlib" "/Users/masonf/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libcore-e3cd79f6ad1173fd.rlib" "/Users/masonf/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libcompiler_builtins-4071d15d8824a4bb.rlib" "-lSystem" "-lresolv" "-lc" "-lm" "-fuse-ld=/usr/local/bin/zld"
  = note: dyld: Library not loaded: @rpath/libLTO.dylib
            Referenced from: /usr/local/bin/zld
            Reason: image not found
          clang: error: unable to execute command: Abort trap: 6
          clang: error: linker command failed due to signal (use -v to see invocation

Looks like that file is here: $ ls /Library/Developer/CommandLineTools/usr/lib/libLTO.dylib /Library/Developer/CommandLineTools/usr/lib/libLTO.dylib

I tried running LD_PATH=/Library/Developer/CommandLineTools/usr/lib cargo run with no luck.

Thanks in advance!

-Mason

Missing zld.zip file for 1.2.1 release

Hi @michaeleisel,

I just wanted to create pull request in your https://github.com/michaeleisel/homebrew-zld/ to update brew formula to match most recent 1.2.1 release with fix for Xcode 12.2, but I realised I can't do this since there is not zld.zip asset added for 1.2.1 release.

I wonder if it's something that you don't intent to provide anymore for future releases and I should work on creating formula that builds zld from source. This is first time I do anything with brew formulas so I just wanted to ask before moving on.

Thanks much,
Maciek

zld does not support ARM64_32 architecture

The open source dumps of ld64 do not include support for ARM64_32 used by WatchOS. The following file would fail to link with zld.

hello.c

// xcrun clang -c hello.c -o hello.o -target arm64_32-apple-watchos4.0 -isysroot /var/db/xcode_select_link/Platforms/WatchOS.platform/Developer/SDKs/WatchOS.sdk
// xcrun ld -dylib -arch arm64_32 -watchos_version_min 4.0.0 -syslibroot /var/db/xcode_select_link/Platforms/WatchOS.platform/Developer/SDKs/WatchOS.sdk -o libHello.dylib hello.o -lc

#include <stdio.h>

void hello(void) {
  printf("hello world");
}

"make" fails through release package

I downloaded source files via zip, when entering make, this error pops up:

ld: file not found: /Users/vrixyz/Downloads/zld/ld/../abseil-cpp-20200225/build/libabsl.a

Click to see more
35 warnings generated.

Ld /Users/vrixyz/Downloads/zld/build/Build/Products/Release/zld normal (in target 'zld' from project 'zld')
  cd /Users/vrixyz/Downloads/zld/ld
  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++ -target x86_64-apple-macos10.14 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk -L/Users/vrixyz/Downloads/zld/build/Build/Products/Release -L/Users/vrixyz/Downloads/zld/ld -L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/lib/system -F/Users/vrixyz/Downloads/zld/build/Build/Products/Release -filelist /Users/thierryberger/Downloads/zld/build/Build/Intermediates.noindex/zld.build/Release/zld.build/Objects-normal/x86_64/zld.LinkFileList -Xlinker -rpath -Xlinker @executable_path -Xlinker -rpath -Xlinker /var/db/xcode_select_link/Toolchains/XcodeDefault.xctoolchain/usr/lib -Xlinker -rpath -Xlinker /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib -Xlinker -rpath -Xlinker /usr/local/lib -dead_strip -Xlinker -object_path_lto -Xlinker /Users/vrixyz/Downloads/zld/build/Build/Intermediates.noindex/zld.build/Release/zld.build/Objects-normal/x86_64/zld_lto.o -stdlib\=libc++ -fobjc-link-runtime -Wl,-U,_malloc_default_purgeable_zone -Wl,-t -Wl,-U,_fnd_get_demangled_name -Wl,-client_name,ld -Wl,-alias,_zldVersionString,_ldVersionString -Wl,-all_load -Wl,/Users/vrixyz/Downloads/zld/ld/../abseil-cpp-20200225/build/libabsl.a -stdlib\=libc++ -lxar -Wl,-lazy_library,/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/libLTO.dylib -Wl,-exported_symbol,__mh_execute_header -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib -ltapi -Wl,-stack_size,0x01000000 -lLTO -ltbb -lswiftDemangle -Xlinker -dependency_info -Xlinker /Users/vrixyz/Downloads/zld/build/Build/Intermediates.noindex/zld.build/Release/zld.build/Objects-normal/x86_64/zld_dependency_info.dat -o /Users/vrixyz/Downloads/zld/build/Build/Products/Release/zld
ld: file not found: /Users/vrixyz/Downloads/zld/ld/../abseil-cpp-20200225/build/libabsl.a
clang: error: linker command failed with exit code 1 (use -v to see invocation)

** BUILD FAILED **


The following build commands failed:
  Ld /Users/vrixyz/Downloads/zld/build/Build/Products/Release/zld normal
(1 failure)
make: *** [build] Error 65

Upon inspection, zld/abseil-cpp-20200225/build/libabsl.a is indeed missing.

Then, when attempting to make abseil-cpp-20200225, this error comes up:

fatal: not a git repository (or any of the parent directories): .git
make: `abseil-cpp-20200225' is up to date.

make via a fresh git clone succeeded from first try.

I would have expected the release package zip to be usable as advertised from the Readme.

Support for Apple Silicon (M1)

It seems zld doesn't compile on Apple Silicon

$ brew install michaeleisel/zld/zld
Updating Homebrew...
==> Auto-updated Homebrew!
Updated 1 tap (homebrew/core).
==> Updated Formulae
Updated 1 formula.

==> Tapping michaeleisel/zld
Cloning into '/Users/darth/projects/homebrew/Library/Taps/michaeleisel/homebrew-zld'...
remote: Enumerating objects: 44, done.
remote: Counting objects: 100% (44/44), done.
remote: Compressing objects: 100% (24/24), done.
remote: Total 44 (delta 8), reused 40 (delta 8), pack-reused 0
Unpacking objects: 100% (44/44), done.
Tapped 1 formula (69 files, 35.4KB).
Warning: You are running macOS on a arm64 CPU architecture.
We do not provide support for this (yet).
Reinstall Homebrew under Rosetta 2 until we support it.
You will encounter build failures with some formulae.
Please create pull requests instead of asking for help on Homebrew's GitHub,
Twitter or any other official channels. You are responsible for resolving
any issues you experience while you are running this
unsupported configuration.

==> Installing zld from michaeleisel/zld
==> Downloading https://github.com/michaeleisel/zld/archive/1.2.1.zip
==> Downloading from https://codeload.github.com/michaeleisel/zld/zip/1.2.1
##=O#- #
==> make clean
==> make build
Last 15 lines from /Users/darth/Library/Logs/Homebrew/zld/02.make:
      absl::lts_2020_02_25::container_internal::raw_hash_set<absl::lts_2020_02_25::container_internal::FlatHashMapPolicy<ld::Atom const*, unsigned long>, absl::lts_2020_02_25::container_internal::HashEq<ld::Atom const*, void>::Hash, absl::lts_2020_02_25::container_internal::HashEq<ld::Atom const*, void>::Eq, std::__1::allocator<std::__1::pair<ld::Atom const* const, unsigned long> > >::resize(unsigned long) in code_dedup.o
      absl::lts_2020_02_25::container_internal::raw_hash_set<absl::lts_2020_02_25::container_internal::FlatHashMapPolicy<ld::Atom const*, unsigned long>, absl::lts_2020_02_25::container_internal::HashEq<ld::Atom const*, void>::Hash, absl::lts_2020_02_25::container_internal::HashEq<ld::Atom const*, void>::Eq, std::__1::allocator<std::__1::pair<ld::Atom const* const, unsigned long> > >::drop_deletes_without_resize() in code_dedup.o
      std::__1::pair<unsigned long, bool> absl::lts_2020_02_25::container_internal::raw_hash_set<absl::lts_2020_02_25::container_internal::FlatHashMapPolicy<ld::Atom const*, ld::Atom const*>, absl::lts_2020_02_25::container_internal::HashEq<ld::Atom const*, void>::Hash, absl::lts_2020_02_25::container_internal::HashEq<ld::Atom const*, void>::Eq, std::__1::allocator<std::__1::pair<ld::Atom const* const, ld::Atom const*> > >::find_or_prepare_insert<ld::Atom const*>(ld::Atom const* const&) in code_dedup.o
      absl::lts_2020_02_25::container_internal::raw_hash_set<absl::lts_2020_02_25::container_internal::FlatHashMapPolicy<ld::Atom const*, ld::Atom const*>, absl::lts_2020_02_25::container_internal::HashEq<ld::Atom const*, void>::Hash, absl::lts_2020_02_25::container_internal::HashEq<ld::Atom const*, void>::Eq, std::__1::allocator<std::__1::pair<ld::Atom const* const, ld::Atom const*> > >::resize(unsigned long) in code_dedup.o
      ...
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

** BUILD FAILED **


The following build commands failed:
        Ld /tmp/zld-20201214-36688-g2u5qu/zld-1.2.1/build/Build/Products/Release/zld normal
(1 failure)
make: *** [build] Error 65

Deterministic Output Independent from Repository Path

The Problem

When linking object files compiled with debug support (-g), the symbol table (LC_SYMTAB) will contain N_OSO entries that point to the object files which contain the debug information.

The N_OSO entries will use absolute paths which means that the final linker output (e.g., binaries, dylibs, etc) will be dependent on the absolute repository path.

Example

Assume two different checkout paths:

  • /Users/user/a/repo
  • /Users/ci/b/repo

Assuming that there's an object file at a relative path to the repo at build/out/X.o, then any debug output binaries will have different paths depending on the machine which they were built on. For example:

  • /Users/user/a/repo/build/out/X.o
  • /Users/ci/b/repo/build/out/X.o

Desired Output

To make the output independent of the repository checkout path, N_OSO entries should be relative to the repo root (or another desired root path).

The ramification for debugging is that lldb needs to have the correct working directory to be able to find symbols as it cannot rely on absolute paths. This is usually done with an appropriate lldbinit script.

Proposed Solution

This is a common issue faced by similar tools (e.g., compilers). Both Clang and GCC support the -fdebug-prefix-map option which allows the remapping of paths.

We can adopt the same approach here to make builds independent of checkout path.

Alternatives

There are some alternative approaches to the problem with different tradeoffs in complexity, cost and maintenance.

Post Processing

The symbol and string tables can be rewritten after a binary/dylib is linked. This is the approach taken by Buck which post-processes the output from the linker if it needs to be deterministic.

The major downside is performance. For binaries of significant size (e.g., 500MB+), the symbol + string tables can account for ~50% of the size. Processing and rewriting the binaries becomes an expensive operation, as just a small number of N_OSO entries need to be adjusted.

In-Place Patching

While the string table can be patched in place, the result will only be deterministic if the repository paths are of equal length. Since that's not practical to ensure at scale, it's not a viable optimisation.

Mount Points

Another potential option would be to mount the repository in a predictable location (e.g., using bindfs). Unfortunately, that's not practical at large scale across multiple local and remote development configurations.

Using pre-built zld

It appears that zld doesn't build on OSX 10.15 for ARM, and maybe that's due to missing arm variants in dylibs. During make, I get:

    Run Build Command(s):/usr/bin/make cmTC_2bd92/fast && /Applications/Xcode.app/Contents/Developer/usr/bin/make  -f CMakeFiles/cmTC_2bd92.dir/build.make CMakeFiles/cmTC_2bd92.dir/build
    Building CXX object CMakeFiles/cmTC_2bd92.dir/testCXXCompiler.cxx.o
    /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++   -arch arm64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk -mmacosx-version-min=11.0 -o CMakeFiles/cmTC_2bd92.dir/testCXXCompiler.cxx.o -c /Users/meisel/projects/zld/abseil-cpp-20200225/build_arm64/CMakeFiles/CMakeTmp/testCXXCompiler.cxx
    Linking CXX executable cmTC_2bd92
    /usr/local/Cellar/cmake/3.18.2/bin/cmake -E cmake_link_script CMakeFiles/cmTC_2bd92.dir/link.txt --verbose=1
    /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++  -arch arm64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk -mmacosx-version-min=11.0 -Wl,-search_paths_first -Wl,-headerpad_max_install_names CMakeFiles/cmTC_2bd92.dir/testCXXCompiler.cxx.o -o cmTC_2bd92 
    ld: warning: ignoring file /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/lib/libc++.tbd, missing required architecture arm64 in file /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/lib/libc++.tbd
    ld: warning: ignoring file /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/lib/libSystem.tbd, missing required architecture arm64 in file /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/lib/libSystem.tbd
    ld: dynamic main executables must link with libSystem.dylib for architecture arm64
    clang: error: linker command failed with exit code 1 (use -v to see invocation)

In that case, maybe it's time to give up on recommending people build it from scratch, and just distribute a pre-built fat binary. Can someone provide the built version? @thii ?

ld: unknown option: -platform_version

Today I upgrade Xcode to 11.4 (11E146), and get the following error:

ld: unknown option: -platform_version
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Caches may cause unwanted loading of object files

The cache in /tmp/... provides a list of object files from static libraries that are loaded automatically in parallel at the start. Once the caches invalidate when the invocation changes at all, which is a todo, I think it's unlikely that this would cause issues. Also, any Objective-C files will generally be loaded with -ObjC, and any Swift will be loaded per recent discussions. But this is just to stay aware of it.

error in running Unit Tests with zld linker

Description

When I set -fuse-ld=<zld path> as the linker flag for xcode, the project stops running unit tests.
I use the following command to run tests over my project

xcodebuild test -workspace NinchatSDKSwift.xcworkspace -scheme NinchatSDKSwiftTests "OTHER_LDFLAGS=\$(OTHER_LDFLAGS) -fuse-ld=$(which zld)" -destination "platform=iOS Simulator,name=iPhone 8"

But I get the following error:

Testing failed:
        NinchatSDKSwiftTests:
                xctest (82532) encountered an error (Early unexpected exit, operation never finished bootstrapping - no restart will be attempted. (Underlying error: Crash: xctest (82532): Namespace SIGNAL, Code 0x5))

** TEST FAILED **

Full Log

This Github Action shows a complete log for the steps resulted in the mentioned issue.

zld Version

This is the output of zld -v

@(#)PROGRAM:zld  PROJECT:zld-
BUILD 11:18:20 Mar 20 2020
configured to support archs: i386 x86_64 x86_64h armv6 armv7 armv7s armv7m armv7k arm64
LTO support using: LLVM version 11.0.3, (clang-1103.0.32.62) (static support for 22, runtime is 26)
TAPI support using: Apple TAPI version 11.0.0 (tapi-1100.0.11)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.