staticfloat / llvmbuilder

BinaryBuilder repository for LLVM

LLVMBuilder

This is an example repository showing how to construct a "builder" repository for a binary dependency. Using a combination of BinaryBuilder.jl, Travis, and GitHub releases, we are able to create a fully-automated, github-hosted binary building and serving infrastructure.

Tips and tricks

1. Add BINARYBUILDER_USE_CCACHE=true to your environment to speed up rebuilds
2. Besides the usual BinaryBuilder commandline flags build_tarball.jl also supports
3. --llvm-release Only build a release build
4. --llvm-debug Only build a debug build
5. --llvm-keep-tblgen Keep tblgen around for later builds
6. --llvm-check Run unit tests
7. Use ghr or similar to upload builds to github

llvmbuilder's Issues

Assert build on ppc64le fails with unhelpful ld error

Putting this up so that I don't forget about it

`ld: final link failed: Bad value`

Update [SCEV] Add explicit representations of umin/smin patch

Carry: JuliaLang/julia#28465

Include FreeBSD tarballs

Is there a technical reason why it isn't included or does this just predate FreeBSD support in BinaryBuilder?

Generated tarballs should have a containing directory

Extracting the tarballs with BinaryBuilder messes-up the BB $srcdir. Quoting @staticfloat:

Your tarball doesn't contain a containing directory?

Carry Perf and OProfile patches

• Both

  • https://reviews.llvm.org/D44890

• Perf

  • https://reviews.llvm.org/D47343
  • https://reviews.llvm.org/D44892 (has not landed yet)

• OProfile

  • https://reviews.llvm.org/D47925

Provide tarball for x86_64 musl

When building a package depending on LLVMBuilder we need llvm-config (and maybe even the tblgen's) available. We either need to build a full release or also make the tblgen archive available in a separate build.jl

As an example see https://github.com/vchuravy/HalideBuilder/blob/e8a2c3c07f9923699ab39d1bbe229e0484034760/build_tarballs.jl#L39

Enable LLVM assertions for CI

@vtjnash points out that we should have a secondary build that uses Assertions and that we use for CI purposes.

-DLLVM_ENABLE_ASSERTIONS:BOOL=ON

[OSX] Symbol not found

With a pre-release made from current master it seems that we are hitting a problem how are building libc++ on MacOSX

dyld: Symbol not found: __ZNSt3__15mutexD1Ev
  Referenced from: /Users/travis/build/JuliaLang/julia/usr/tools/../lib/libLLVM.dylib
  Expected in: /Users/travis/build/JuliaLang/julia/usr/tools/../lib/libc++.1.dylib
 in /Users/travis/build/JuliaLang/julia/usr/tools/../lib/libLLVM.dylib

from https://travis-ci.org/JuliaLang/julia/jobs/372492381

this is probably due to

LLVMBuilder/build_tarballs.jl

Lines 169 to 171 in aa52191

	CMAKE_FLAGS="${CMAKE_FLAGS} -DLIBCXX_ENABLE_THREADS=OFF"
	CMAKE_FLAGS="${CMAKE_FLAGS} -DLIBCXX_ENABLE_MONOTONIC_CLOCK=OFF"
	CMAKE_FLAGS="${CMAKE_FLAGS} -DLIBCXXABI_ENABLE_THREADS=OFF"

Which IIRC I set since it caused issues for the Windows build.

Work around macOS bug with llvm-config

ie. BinaryBuilder counterpart of JuliaLang/julia#30033

I imagine it's just adding some lines similar to this hack:

LLVMBuilder/build_tarballs.jl

Lines 277 to 282 in 5708d3d

	# Live is harsh on Windows and dynamic libraries are
	# expected to live alongside the binaries. So we have
	# to copy the *.dll from bin/ to tools/ as well...
	if [[ "${target}" == *mingw* ]]; then
	cp ${prefix}/bin/*.dll ${prefix}/tools/
	fi

instead matching against *darwin*, but the version string doesn't seem available to the recipe and lacking CI I have no idea how to test this change locally, so opening an issue instead.

libwinpthread-1.dll missing on Windows when used independently

I directly downloaded the binary release for win64, and I tried running llc. I got an error complaining about a missing libwinpthread-1.dll. According to @vchuravy, this is provided with Julia.

I've seen this with other CMake/C++ builds, and I ended up just statically compiling on the windows targets. It would be nice if there was another approach.

[Win] Undefined symbols

On both i686 and x86_64 linking Julia against LLVM hits undefined symbols

• https://ci.appveyor.com/project/JuliaLang/julia/build/1.0.26338/job/13byi77vmlhf2ji5
• https://ci.appveyor.com/project/JuliaLang/julia/build/1.0.26338/job/57t4v7rmvipk4qds

/APInt-C.o: In function `ZN4llvm7APFloatC4Ed':
/cygdrive/c/projects/julia/usr/include/llvm/ADT/APFloat.h:843: undefined reference to `llvm::APFloatBase::IEEEdouble()'
/cygdrive/c/projects/julia/usr/include/llvm/ADT/APFloat.h:843: undefined reference to `llvm::detail::IEEEFloat::IEEEFloat(double)'
/cygdrive/c/projects/julia/usr/include/llvm/ADT/APFloat.h:843: undefined reference to `llvm::APFloat::Storage::Storage(llvm::detail::IEEEFloat, llvm::fltSemantics const&)'
/cygdrive/c/projects/julia/usr/include/llvm/ADT/APFloat.h:843: undefined reference to `llvm::detail::IEEEFloat::~IEEEFloat()'
./APInt-C.o: In function `ZN4llvm7APFloat10usesLayoutINS_6detail9IEEEFloatEEEbRKNS_12fltSemanticsE':
/cygdrive/c/projects/julia/usr/include/llvm/ADT/APFloat.h:775: undefined reference to `llvm::APFloatBase::PPCDoubleDouble()'
./APInt-C.o: In function 

This seems to be coming from line:

i686-w64-mingw32-g++ -march=pentium4 -m32 -shared -Wl,--out-implib,/cygdrive/c/projects/julia/usr/lib/libjulia.dll.a -pipe  -fno-rtti -mincoming-stack-boundary=2  -O3 -ggdb2 -falign-functions -momit-leaf-frame-pointer -D_GNU_SOURCE -I. -I/cygdrive/c/projects/julia/src -I/cygdrive/c/projects/julia/src/flisp -I/cygdrive/c/projects/julia/src/support -I/cygdrive/c/projects/julia/usr/include -I/cygdrive/c/projects/julia/usr/include -DLIBRARY_EXPORTS -I/cygdrive/c/projects/julia/deps/valgrind -Wall -Wno-strict-aliasing -fno-omit-frame-pointer -fvisibility=hidden -fno-common -Wpointer-arith -Wundef -DJL_BUILD_ARCH='"i686"' -DJL_BUILD_UNAME='"NT"' -IC:\projects\julia\usr/include -DLLVM_SHLIB "-DJL_SYSTEM_IMAGE_PATH=\"../lib/julia/sys.dll\"" ./jltypes.o ./gf.o ./typemap.o ./ast.o ./builtins.o ./module.o ./interpreter.o ./symbol.o ./dlload.o ./sys.o ./init.o ./task.o ./array.o ./dump.o ./staticdata.o ./toplevel.o ./jl_uv.o ./datatype.o ./simplevector.o ./APInt-C.o ./runtime_intrinsics.o ./runtime_ccall.o ./precompile.o ./threadgroup.o ./threading.o ./stackwalk.o ./gc.o ./gc-debug.o ./gc-pages.o ./method.o ./jlapi.o ./signal-handling.o ./safepoint.o ./jloptions.o ./timing.o ./subtype.o ./rtutils.o ./crc32c.o ./processor.o ./codegen.o ./jitlayers.o ./disasm.o ./debuginfo.o ./llvm-simdloop.o ./llvm-ptls.o ./llvm-muladd.o ./llvm-late-gc-lowering.o ./llvm-lower-handlers.o ./llvm-gc-invariant-verifier.o ./llvm-propagate-addrspaces.o ./llvm-multiversioning.o ./llvm-alloc-opt.o ./cgmemmgr.o ./llvm-api.o  -o /cygdrive/c/projects/julia/usr/bin/libjulia.dll   -Wl,--whole-archive ./flisp/libflisp.a -Wl,--whole-archive ./support/libsupport.a -L/cygdrive/c/projects/julia/usr/bin -L/cygdrive/c/projects/julia/usr/lib /cygdrive/c/projects/julia/usr/lib/libuv.a /cygdrive/c/projects/julia/usr/lib/libutf8proc.a -Wl,--no-whole-archive  -LC:\projects\julia\usr/lib  -lLLVM -Wl,--export-all-symbols -Wl,--version-script=/cygdrive/c/projects/julia/src/julia.expmap -Wl,--no-whole-archive -lpsapi -lkernel32 -lws2_32 -liphlpapi -lwinmm -ldbghelp -luserenv -lssp 

and might be a problem with the buildsystem since: -LC:\projects\julia\usr/lib -lLLVM is not -L/cygdrive/c/projects/julia/usr/lib -lLLVM

LLVM 8.0.0

Would be nice to have :) (e.g. for building Mesa, or I heard ROCm would also love it!)

Build Julia1.0-compatible Clang/LLVM binaries

As per discussed on Slack, there are no versions of Clang/LLVM binaries in the LLVMBuilder that are being compatible with the LLVM shipped with Julia 1.0. For example, loading and using libclang.dylib(6.0.0-6) will crash Julia.
It could work as long as we rebuild Julia with USE_BINARYBUILDER_LLVM=1(use the LLVMBuilder binaries). However, since Julia 1.0 has already been shipped, for compatibility's sake, we may need to tweak LLVMBuilder for building a version that can work along with Julia 1.0.

current status(updated)	Julia 1.0 (LLVM 6.0.0)	Julia 1.x (LLVM 6.0.1)
LLVM original	❌	❌
LLVM w/ WebAssembly	❌	✅
LLVM w/o WebAssembly	✅	✅

cc @staticfloat

julia> versioninfo()
Julia Version 1.0.0
Commit 5d4eaca0c9 (2018-08-08 20:58 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin14.5.0)
  CPU: Intel(R) Core(TM) i5-6267U CPU @ 2.90GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, skylake)
Environment:
  JULIA_EDITOR = atom
  JULIA_NUM_THREADS = 4

julia> using Libdl
julia> clang = Libdl.dlopen_e("./libclang.dylib")  # 6.0.0-6
julia> idx = Libdl.dlsym_e(clang, :clang_createIndex)
julia> ccall(idx, Ptr{Cvoid}, (Int32, Int32), 1, 1)
dyld: lazy symbol binding failed: Symbol not found: _LLVMInitializeWebAssemblyTargetInfo
  Referenced from: ./libclang.dylib
  Expected in: /Applications/Julia-1.0.app/Contents/Resources/julia/lib/julia/libLLVM.dylib

dyld: Symbol not found: _LLVMInitializeWebAssemblyTargetInfo
  Referenced from: ./libclang.dylib
  Expected in: /Applications/Julia-1.0.app/Contents/Resources/julia/lib/julia/libLLVM.dylib


signal (6): Abort trap: 6
in expression starting at nin expression starting at no file:0
in expression starting at no file:0
unknown function (ip: 0x10636b249)
Allocations: 13090829 (Pool: 13088353; Big: 2476); GC: 28
[1]    16239 abort      julia

julia> versioninfo()
Julia Version 1.1.0-DEV.279
Commit 3a15e6bd5b (2018-09-18 01:19 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin17.7.0)
  CPU: Intel(R) Core(TM) i5-6267U CPU @ 2.90GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, skylake)
Environment:
  JULIA_EDITOR = atom
  JULIA_NUM_THREADS = 4

julia> using Libdl

julia> clang = Libdl.dlopen_e("./libclang.dylib")
Ptr{Nothing} @0x00007fd706a308b0

julia> idx = Libdl.dlsym_e(clang, :clang_createIndex)
Ptr{Nothing} @0x0000000122c3ef90

julia> ccall(idx, Ptr{Cvoid}, (Int32, Int32), 1, 1)
Ptr{Nothing} @0x00007fd704417c90

Build with perf/oprofile/vtunes support enabled

/usr/bin/ld: /home/vchuravy/builds/julia/usr/lib/libjulia.so: undefined reference to `llvm::JITEventListener::createPerfJITEventListener()'
collect2: error: ld returned 1 exit status

with USE_PERF_JITEVENTS=1

LLVM_TOOLS_BINARY_DIR points to different location than LLVM_TOOLS_INSTALL_DIR

lib/cmake/llvm/LLVMConfig.cmake contains our build configuration and the variable LLVM_TOOLS_BINARY_DIR points to $prefix/bin and not to LLVM_TOOLS_INSTALL_DIR

LLVMBuilder/build_tarballs.jl

Line 197 in 5708d3d

CMAKE_FLAGS="${CMAKE_FLAGS} -DLLVM_TOOLS_INSTALL_DIR=${prefix}/tools"

Issues encountered

• download URL in build_LLVM.v6.0.1.jl ends with <tag>
• llvm-config is under tools/, but is declared as ExecutableProduct, which implies bin/ subdir, leading to:

ERROR: LoadError: ExecutableProduct("/untmp/jltry/dev/Clang/deps/usr/bin/llvm-config", :llvm_config, "Prefix(/untmp/jltry/dev/Clang/deps/usr)") is not satisfied, cannot generate deps.jl!

(quick patch, changed it to FileProduct(prefix, "tools/llvm-config", :llvm_config))

• cannot be loaded in locally-built julia? (expected, probably..)

┌ Error: Error building `Clang`:
│ : CommandLine Error: Option 'asm-instrumentation' registered more than once!
│ LLVM ERROR: inconsistency in registered CommandLine options

Using build.jl

With Julia 0.7alpha, I'm seeing the following when trying to run the build.jl file from release v6.0.0-3.

julia> include("build.jl")
[ Info: Downloading https://github.com/staticfloat/LLVMBuilder/releases/download/v6.0.0-3/LLVM.x86_64-linux-gnu.tar.gz to /home/tshort/FS/appfiles/Julia/ExportWebAssembly/deps/usr/downloads/LLVM.x86_64-linux-gnu.tar.gz...
[07:02:12] ######################################################################## 100.0%
┌ Warning: `wait(t::Task)` is deprecated, use `fetch(t)` instead.
│   caller = wait(::OutputCollector) at OutputCollector.jl:163
└ @ BinaryProvider OutputCollector.jl:163
ERROR (unhandled task failure): MethodError: no method matching position(::Pipe)
Closest candidates are:
  position(::Base.Filesystem.File) at filesystem.jl:225
  position(::Base.Libc.FILE) at libc.jl:92
  position(::IOStream) at iostream.jl:188
  ...
Stacktrace:
 [1] try_yieldto(::typeof(Base.ensure_rescheduled), ::Base.RefValue{Task}) at ./event.jl:195
 [2] wait() at ./event.jl:254
 [3] wait(::Condition) at ./event.jl:46
 [4] _wait(::Task) at ./task.jl:181
 [5] fetch at ./task.jl:197 [inlined]
 [6] wait(::Task) at ./deprecated.jl:55
 [7] macro expansion at /home/tshort/.julia/dev/BinaryProvider/src/OutputCollector.jl:62 [inlined]
 [8] (::getfield(BinaryProvider, Symbol("##2#4")){Condition,Task})() at ./task.jl:254
ERROR: LoadError: Could not unpack /home/tshort/FS/appfiles/Julia/ExportWebAssembly/deps/usr/downloads/LLVM.x86_64-linux-gnu.tar.gz into /home/tshort/FS/appfiles/Julia/ExportWebAssembly/deps/usr
Stacktrace:
 [1] error at ./error.jl:33 [inlined]
 [2] #unpack#94(::Bool, ::Any, ::String, ::String) at /home/tshort/.julia/dev/BinaryProvider/src/PlatformEngines.jl:599
 [3] #unpack at ./<missing>:0 [inlined]
 [4] #install#111(::Prefix, ::String, ::Bool, ::Bool, ::Bool, ::Any, ::String, ::String) at /home/tshort/.julia/dev/BinaryProvider/src/Prefix.jl:381
 [5] (::getfield(BinaryProvider, Symbol("#kw##install")))(::Any, ::typeof(install), ::String, ::String) at ./<missing>:0
 [6] top-level scope at /home/tshort/FS/appfiles/Julia/ExportWebAssembly/deps/build.jl:35
 [7] include at ./boot.jl:314 [inlined]
 [8] include_relative(::Module, ::String) at ./loading.jl:1071
 [9] include(::Module, ::String) at ./sysimg.jl:29
 [10] include(::String) at ./client.jl:393
 [11] top-level scope
in expression starting at /home/tshort/FS/appfiles/Julia/ExportWebAssembly/deps/build.jl:35

Assertions build missing

julia/llvm.mk has a BINARYBUILDER_LLVM_ASSERTS flag, but that doesn't seem to work:

$ make -C deps install-llvm BINARYBUILDER_LLVM_ASSERTS=1
julia/deps/tools/jldownload julia/deps/srccache/LLVM.asserts.v6.0.1.x86_64-linux-gnu-gcc8.tar.gz https://github.com/JuliaPackaging/Yggdrasil/releases/download/LLVM-v6.0.1-6+nowasm/LLVM.asserts.v6.0.1.x86_64-linux-gnu-gcc8.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   469  100   469    0     0   1146      0 --:--:-- --:--:-- --:--:--  1143
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 404 Not Found
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 404 Not Found
make: *** [julia/deps/llvm.mk:542: julia/deps/srccache/LLVM.asserts.v6.0.1.x86_64-linux-gnu-gcc8.tar.gz] Error 22
make: Leaving directory 'julia/build/release/deps'

Also, why is this a separate flag? Shouldn't it just use LLVM_ASSERTIONS?

Generated CMake files are not relocatable

When using the generated archives in BB, which extracts them to workspace/srcdir, I get the following when actually using the files:

[15:31:54] CMake Error at /workspace/srcdir/LLVM/lib/cmake/llvm/LLVMExports.cmake:751 (message):
[15:31:54]   The imported target "llvm-tblgen" references the file
[15:31:54] 
[15:31:54]      "/workspace/destdir/tools/llvm-tblgen"
[15:31:54] 
[15:31:54]   but this file does not exist.  Possible reasons include:
[15:31:54] 
[15:31:54]   * The file was deleted, renamed, or moved to another location.
[15:31:54] 
[15:31:54]   * An install or uninstall procedure did not complete successfully.
[15:31:54] 
[15:31:54]   * The installation package was faulty and contained
[15:31:54] 
[15:31:54]      "/workspace/srcdir/LLVM/lib/cmake/llvm/LLVMExports.cmake"
[15:31:54] 
[15:31:54]   but not all the files it references.

So now I go through some lengths to extract to the prefix, install my tool to a different prefix, and swap at the end. Maybe, for usability's sake, LLVMBuilder should install into what ends up the directory it will be extracted in by users (/workspace/srcdir/LLVM) so that everything works out of the box?

LLD linker

Would you consider a PR to add the lld linker under tools? I've got it running locally, and it won't be a big change to the build script. It adds about 3 MB to each of the .tar.gz binaries.

My main reason for wanting this is that lld can create WebAssembly binary output (.wasm) from LLVM object files. It may also be useful for other static linking scenarios.

Error on dlopening libLLVM on FreeBSD

Steps to reproduce:

• curl and unpack the FreeBSD tarball for the most recent release. I used the GCC 8 binaries on my FreeBSD 11.2 machine.
• Start Julia (I used 1.1) in the directory and try to dlopen libLLVM.

julia> using Libdl

julia> ptr = dlopen(joinpath(pwd(), "lib", "libLLVM-6.0.1.so"))
: CommandLine Error: Option 'help-list' registered more than once!
LLVM ERROR: inconsistency in registered CommandLine options

This kills Julia immediately. Did I do something dumb or is this an issue?

build.jl issue

The build.jl in release v6.0.0-3 can't be used as a dependency by other Builders because there's a "" embedded in the following string:

bin_prefix = "https://github.com/staticfloat/LLVMBuilder/releases/download/<tag>"

WebAssembly

I see you included WebAssembly when compiling. Awesome!

I compiled @vchuravy's vc/llvmbb branch and ran usr/tools/llc --version and I don't see anything related to wasm32:

LLVM (http://llvm.org/):
  LLVM version 6.0.0
  Optimized build.
  Default target: x86_64-unknown-linux-gnu
  Host CPU: nehalem

  Registered Targets:
    amdgcn  - AMD GCN GPUs
    nvptx   - NVIDIA PTX 32-bit
    nvptx64 - NVIDIA PTX 64-bit
    r600    - AMD GPUs HD2XXX-HD6XXX
    x86     - 32-bit X86: Pentium-Pro and above
    x86-64  - 64-bit X86: EM64T and AMD64

I also downloaded the binary release for win64, and I tried running llc. I get an error complaining about a missing libwinpthread-1.dll. Maybe this should be a separate issue. I've seen that with other CMake/C++ builds, and I ended up just statically compiling on windows.