juliapackaging / yggdrasil

Collection of builder repositories for BinaryBuilder.jl

Yggdrasil

Build Status:

This repository contains recipes for building binaries for Julia packages using BinaryBuilder.jl.

Contributing

To contribute a new recipe, you can either

• use BinaryBuilder.run_wizard(), which will automatically open a pull request to this repository after a successfull build for all requested platforms
• Copy another build recipe using it as a template, and then open a manual pull request to this repository

Yggdrasil builds the tarballs using master version of BinaryBuilder.jl, which requires Julia 1.3.0 or later versions. Note that this BinaryBuilder.jl version has some differences compared to v0.1.4 and the builders generated are slightly different. You are welcome to contribute builders written for BinaryBuilder.jl v0.1.4, but they will likely need minor adjustements.

Buildkite CI is used to test that the builders can successfully produce the tarballs.

If you prefer to test your manual buildscript before opening the pull request, we suggest installing BinaryBuilder.jl on Julia 1.3 or any following release and running julia --color=yes build_tarballs.jl --verbose --debug locally. On MacOS, you will need to have docker installed for this to work.

Updating the version of an existing builder

To trigger the build of a new version of the upstream package simply open a pull request to update the builder as necessary. This usually boils down to only updating the version number and the source (e.g., URL and hash for an archive, or the revision for a git repository), but in some cases more changes may be needed.

Using the generated tarballs

JLL packages

The last versions of BinaryBuilder make use of the Artifacts system shipping in Julia 1.3. This means that BinaryBuilder no longer generates build.jl files that are placed into your Julia package's deps/ folder, but instead generates whole Julia packages (known colloquially as "jll" packages) that are placed within the JuliaBinaryWrappers organization. Merged pull requests to Yggdrasil result in new versions of these wrapper packages being generated, uploaded and registered, allowing your client Julia code to simply invoke using LibFoo_jll to get ahold of your binaries with no need for a Pkg.build() step. (This will, of course, only be the case for Julia 1.3+).

We encourage Julia developers to use JLL packages for their libraries. Read the documention of BinaryBuilder to learn how to use them.

Here are a few examples of pull requests of Julia packages switching to using JLL package to provide the prebuilt binaries to the users:

• Cairo.jl
• FFTW.jl
• GSL.jl
• Gtk.jl
• Rsvg.jl

You can read more about the Artifacts system and how it is important for reproducibility in this post on Julia's blog "Pkg + BinaryBuilder -- The Next Generation".

BinaryProvider.jl

We hope we convinced you about why it is important to switch to JLL packages. However, if you really need to support Julia v1.2 or previous versions, you should keep using BinaryProvider.jl. As explained in the previous section, a build.jl file is not generated after the build anymore as that is not used for JLL packages, instead you would need to generate it yourself: In the top-directory of Yggdrasil there is a script to do this: generate_buildjl.jl. It takes three positional arguments:

• the path to the build_tarballs.jl script
• the owner/name of the repository where the tarballs have been uploaded. If omitted, this defaults to JuliaBinaryWrappers/BuilderName_jll.jl
• the tag name where the tarballs have been uploaded. If omitted, this defaults to the latest version of the JLL package in the General registry. If there are no versions of the package in the registry, the script will fail.

For example, to get the build.jl file for the latest version of Zlib you can run the following command:

julia --color=yes generate_buildjl.jl Z/Zlib/build_tarballs.jl

If instead you want to get the build_tarballs.jl file for the tag named Zlib-v1.2.11+6 you have to run the command

julia --color=yes generate_buildjl.jl Z/Zlib/build_tarballs.jl JuliaBinaryWrappers/Zlib_jll.jl Zlib-v1.2.11+6

Note: you have to manually add prefix as the first argument to all Product constructors in the generated build.jl files. This is necessary because the syntax between BinaryBuilder v0.2+ and BinaryProvider has diverged.

Remember that you will also need the build.jl files for all direct and indirect dependencies.

Here are a few examples of packages using this system to install their libraries:

• FFMPEG.jl;
• MySQL.jl uses JLL packages for Julia v1.3+ and build.jl scripts with BinaryProvider.jl for previous releases.

yggdrasil's Issues

[AZP] Evaluate possibility of using Azure Artifacts

We could look into using Azure Artifacts to provide the tarballs of a pull requests build for inspection.

I'm not sure how it would work, especially with multiple projects built in the same pull request.

Also, there is a limit of 2 GiB of storage for the free plan, so we probably would need to have a very short retention period (like one day) and maybe limit to the latest build of a PR (if that's possible at all). Even so, it's not that hard for us to hit the storage limit in one day.

Libssh2 on Windows has outdated ciphers

Consider upgrading libssh2 to v1.8.2 to resolve this issue with private packages on windows: JuliaLang/julia#28933

Happy to do it myself with some guidance. Specifically, I am not sure whether the patches are still needed and how to test changes I make in my fork of this repo because it appears to be hardcoded in the make file.

Thanks!

Git_jll doesn't work with HTTP remotes?

git() do git
           readlines(`$git ls-remote http://github.com/JuliaPlots/MakieReferenceImages`)
       end
fatal: unable to find remote helper for 'http'
ERROR: failed process: Process(`/Users/anshul/.julia/artifacts/4c5e6fa65bda6837ba17cf2e58e0016f6229e211/bin/git ls-remote http://github.com/JuliaPlots/MakieReferenceImages`, ProcessExited(128)) [128]

This works on my local, command-line git just fine.

using SymEngine_jll makes Julia crash on 64-bit Windows

Following up from #279, I can confirm that just doing

using SymEngine_jll

makes Julia crash on 64-bit Windows. See for example these AppVeyor and Travis jobs. The content of the test script is

@info "Before loading SymEngine_jll..."
using SymEngine_jll
@info "After loading SymEngine_jll..."

@isuruf reported that with old BinaryBuilder he used expand_gcc_version to make this work, but we don't have that function anymore and he's using the pure C interface. Any clue of what could be the problem and possible solutions? Maybe we're hitting a compiler bug and building with a later version of GCC, without expanding, could help?

[julia] julia binaries for BB packages that need them as dependencies

As far as I understand, Cxx and CxxWrap need access to julia binaries at build time.
There is a build recipe here in Yggdrasil, but that doesn't work, because julia can't be cross-compiled.
There's also a repo to host binaries here: https://github.com/JuliaPackaging/JuliaBuilder, but that hasn't been updated with new binaries in a while, so @Gnimuc forked it and put some newer binaries there https://github.com/Gnimuc/JuliaBuilder/releases.
The new BB can't use the build recipes as binary dependencies any more, so after a discussion on slack, I tried to create an Artifacts.toml by hand, PR is here: JuliaRegistries/General#7030. It was closed without a suggestion for a better place to put the binaries.
All of that is a bit of a mess.
I think it would be good to have one canonical location for julia binaries so that they can be pulled in as binary dependencies by BB scripts. I've tried to collect the references to all of the relevant locations in this issue, hopefully that helps with retiring some of them.
cc @barche @giordano @staticfloat

[SDL2] SDL2 does not load on the Mac

SDL2_jll fails to load on the Mac with the following error. Any idea why? Seems to work in Windows and Linux.

cc: @giordano @NHDaly

ERROR: LoadError: InitError: could not load library "/Users/travis/.julia/artifacts/07269157a430173d2493ced9ed3982219f62b22d/lib/libSDL2_image.dylib"

dlopen(/Users/travis/.julia/artifacts/07269157a430173d2493ced9ed3982219f62b22d/lib/libSDL2_image.dylib, 1): Library not loaded: @rpath/libiconv.2.dylib

  Referenced from: /Users/travis/.julia/artifacts/07269157a430173d2493ced9ed3982219f62b22d/lib/libSDL2_image.dylib

  Reason: Incompatible library version: libSDL2_image.dylib requires version 9.0.0 or later, but libiconv.2.dylib provides version 7.0.0

Good to bring Arpack in here.

https://github.com/JuliaLinearAlgebra/ArpackBuilder

libexpat?

This is may be all that stands between the Julia Makefile and total BB dominance, at https://github.com/JuliaLang/julia/blob/828a2ecf3a8da5ac9ed85ef7646af46226105765/Makefile#L594-L612

(cause of CI failure https://build.julialang.org/#/builders/64/builds/5042)

Provide Elemental binaries

Now that we have MPICH, should we use the Elemental from elemental github org, or the LLNL forked version?

Increase openblas threads defaults

As discussed in JuliaLang/julia#14857, the default number of threads on linux is probably too low as core counts have significantly gone up. My suggestion is to make the default NUM_THREADS at compile time to be 64 on Linux (until a better fix is found), and leave it at 16 on Windows, Mac and ARM.

Relevant line here:

Yggdrasil/O/OpenBLAS/build_tarballs.jl

Line 42 in 7fcd72a

# Use 16 threads unless we're on an i686 arch:

make tar files more polite

The tarball files in the releases contain rather unexpected structures ("tar bomb") leading to annoyance when trying to examine them. A polite tar file should be designed to create only a single directory of the appropriate name. e.g. http://www.linfo.org/tarbomb.html

CMake does not have binary

It would be nice to have CMake binary at https://github.com/JuliaBinaryWrappers. Currently we have script at https://github.com/JuliaPackaging/Yggdrasil/blob/master/C/CMake/build_tarballs.jl .

Version support?

I'm trying to build binaries for a library that requires GSL v1. There's a build_tarballs.jl for v2.6 here, but my library won't build against it. What's the preferred solution? Should I vendor GSL v1 as part of my binaries, or open a pull request for, say, G/GSL1?

GCC native binaries for all platforms

pretty pleaase :)

[AZP] Cache of artifacts is not working

Currently, compiler shards are re-download all the time in all jobs, defeating the purpose of having a cache 😕

The CI plan

I'd like to set up a CI system to make both both building and deploying more painless. In short, users should be able to submit PRs, have a CI bot attempt to build it, and then when it gets merged it, it should be rebuilt and registered automatically. We want this CI to run on our own hardware (so that we have no time limits and so that we can cache artifacts and whatnot) so we'll use GitLab CI.

Here's the general overview:

• GitHub PRs and commits to master trigger a GitLab CI runner
• GitLab CI runner launches inside docker container that has BB already built (this docker image already exists, it gets built automatically every time I push something to BB [0], you can run it with docker run -ti staticfloat/binarybuilder.jl)
• The GitLab CI script looks at environment variabled provided by the GitLab CI runner [1] and git info to figure out the following pieces of information:
  • What files were just changed? Use that to build up a list of recipes I need to build for this changeset.
  • Am I a PR or a push to master? If I'm a PR, I'm going to be building with just --verbose. If I'm a push to master, I'm going to be building with --verbose --deploy --register.
• We will need to be able to exclude certain builders, for instance LLVM should probably never be done automatically, since it takes dozens of hours to do a full rebuild of that. Same with the rootfs.

For now, we're just going to run the builds and we'll inspect the textual output. Long-term, I would like to have a system where things like audit passes and whatnot have their information saved in a machine-parsable way so that we can quickly summarize the auditor output. Right now, we ignore most audit output and just keep it around so that we can look back once something breaks to try and find a hint as to why it broke; we are only ever going to get stricter though, so having this information easily viewable will eventually be important. :)

[0] https://cloud.docker.com/u/staticfloat/repository/docker/staticfloat/binarybuilder.jl
[1] https://docs.gitlab.com/ee/ci/variables/

openspecfun

Would be nice to move it in here from:

https://github.com/JuliaMath/OpenspecfunBuilder

Maybe that will help address the issues such as JuliaMath/SpecialFunctions.jl#155

Organize into folders by first letter? like JuliaRegistries/General

Just a thought.. It might make things more manageable.. easier to make that change at an earlier stage..

GR binaries

Currently, GR binaries are served from the GR repo on github. It seems like building GR in BB maybe difficult, but it would be nice to be able to serve the pre-built binaries through BinaryProvider (to make the experience and infrastructure of serving binaries uniform):

https://github.com/jheinen/GR.jl/blob/6e01760db801e5637a48d240069f1447d05c6077/deps/build.jl#L95

cc @jheinen

Nice to get libsndfile into BB

It would be nice to have libsndfile packaged up for serving through the BB infrastructure.

HDF5 header files

I would like to build the NetCDF library which requires HDF5. Unfortunately, the current HDF5_jll, does not install the header files. The windows msys2 package (http://repo.msys2.org/mingw/i686/mingw-w64-i686-hdf5-1.10.5-1-any.pkg.tar.xz) does actually include the necessary header files, but the python whl do not. But apparently, they correspond to HDF5 1.10.4, and presumably they can be extract from https://bitbucket.hdfgroup.org/projects/HDFFV/repos/hdf5/branches?base=hdf5-1_10_4, unless there is some better idea...

Thanks!

$ strings  libhdf5.103.dylib | grep 'HDF5 library version'
Warning! ***HDF5 library version mismatched error***
HDF5 library version: %d.%d.%d
The HDF5 library version information are not consistent in its source code.
HDF5 library version: 1.10.4
HDF5 library version: 1.10.4

Upgrade sundials to use jll.

What do we need to do to upgrade the sundials install to use the new infrastructure? Also, I notice that it downloads suitesparse to build KLU. I would prefer that we build KLU as part of the regular suitesparse in Julia, and have sundials use that.

Not a huge problem, but sundials ships its own libsuitesparseconfig and is using a different version of suitesparse than upstream. Could lead to subtle issues.

cc @ChrisRackauckas

Git_jll is missing Win64 wrapper

Has been noticed in a slack discussion with @giordano.

Git_jll has 12 wrappers. The one for 64bit Windows, x86_64-w64-ming32, is missing.

The build_tarballs.jl shows w64. Maybe there went something wrong in the build process?

GEOS expanded cxx string ABIs are identical

This already came up in #293 (comment), but making a separate issue to keep GEOS issues out of the GDAL PR discussion.

In #302 we expanded the cxx string ABIs for GEOS. However the files for both builds are identical apart from the log. Both are built with cxx11 string ABI.

Does anyone have an idea why expand_cxxstring_abis is not working here, and how we can make it work?

https://github.com/JuliaBinaryWrappers/GEOS_jll.jl/releases/tag/GEOS-v3.8.0%2B2
https://github.com/libgeos/geos

fftw

https://github.com/JuliaMath/FFTWBuilder

Looking for Leptonica and Tesseract binaries under JuliaBinaryWrappers

Binary builder needed for Tesseract OCR and Leptonica Image Processing Library

https://github.com/tesseract-ocr/tesseract
http://www.leptonica.org/source/leptonica-1.78.0.tar.gz

Tesseract utilizes Leptonica internally. It's best both are built together.

Windows builds pass with build.jl, fail with JLL

We recently registered ASL_JLL.jl. I used the generage_buildjl.jl script to generate this build.jl. All builds pass on Linux, macOS, Windows and FreeBSD:

• https://travis-ci.org/JuliaSmoothOptimizers/AmplNLReader.jl/builds/618635333
• https://ci.appveyor.com/project/dpo/amplnlreader-jl/branch/master
• https://cirrus-ci.com/github/JuliaSmoothOptimizers/AmplNLReader.jl

Now I'm trying to get rid of build.jl and depend on ASL_jll instead. In the process I test a variety of architectures on TravisCI, but some builds that used to pass no longer do:

• https://travis-ci.org/JuliaSmoothOptimizers/AmplNLReader.jl/builds/619403421
• https://ci.appveyor.com/project/dpo/amplnlreader-jl/builds/29240265

In particular:

• the Appveyor/MinGW builds fail with the message

  ERROR: LoadError: LoadError: error compiling AmplModel: error compiling AmplModel#1: could not load library 
  "C:\Users\appveyor\.julia\artifacts\143fa69fa594c92770a91f9712d78d6c92bb7b47\bin\libasl-0.dll"
  The specified module could not be found.
  

• the TravisCI/Linux/PPC build fails with

  /home/travis/.travis/functions: line 109: /home/travis/julia/bin/julia: cannot execute binary file: Exec format error
  

  This error is probably not related to BinaryBuilder.jl / Yggdrasil.jl but I thought I'd report it.
• the TravisCI/Windows/x64 build fails with

  ERROR: LoadError: LoadError: error compiling AmplModel: error compiling AmplModel#1: could not load library 
  "C:\Users\travis\.julia\artifacts\143fa69fa594c92770a91f9712d78d6c92bb7b47\bin\libasl-0.dll"
  

Why are the Windows builds failing? Here is the pull request with my changes.

I'll try and debug the Appveyor builds by ssh-ing into the VM but I thought you might have some ideas. Thanks in advance!

[TCL/TK] .tcl files not copied to artifacts

The TCL & Tk builds contains many .tcl files in the share directory. None of them are copied into the artifacts. This makes the build unusable.

Dependency tree for Pango and Cairo

Ok, I feel like I need to clearly write somewhere a dependency tree of Pango and Cairo;

• Pango (PR #41)
  • Cairo (PR #39)
    • libX11 (necessary only on Linux and FreeBSD)
    • LZO
    • Pixman
    • libpng
    • zlib
  • HarfBuzz (necessary only on Linux and FreeBSD) (PR #50)
    • Glib
    • Graphite (PR #49)
    • ICU (maybe only compile-time?)
    • FreeType2
    • Cairo (PR #39)
  • FriBidi
  • Libffi
  • Gettext
  • Libiconv
  • FreeType2
    • bzip2
  • Glib
    • PCRE
  • FontConfig (necessary only on Linux and FreeBSD)
    • Expat
    • Libuuid

libffi v3.2.1 release contains no build artifacts

While most other releases contain tarballs and a build.jl file, the only tag for libffi has neither, which means it can't be used as a dependency for other builders.

libgfortran missing (path to libgfortran not correct)

I have tried to run two different _jll packages which link to libgfortran (i.e., Fortran codes HelloWorldFortran_jll and Mineos_jll), but neither of them work on any machines I have access to because of a failure to find libgfortran when dynamically loading the library.

Workaround

On my Mac, libgfortran.5.dylib lives at /opt/local/lib/gcc. Here, setting DYLD_FALLBACK_LIBRARY_PATH appropriately allows the program to run, though asking users to set this seems against the principle of an easy-to-install version of a binary. What this

On the Linux machine, no such workaround is possible since libgfortran.so.4 does not exist.

Question

Should we also be building libgfortran and so on and depending on those artifacts rather than relying on users having libgfortran (and libc, I guess) installed? Most users will not have this sort of environment.
[Edit: I now realise libgfortran is distributed with Julia itself; I guess the issue is finding this library to link against.]

Error output

On macOS (x86_64-apple-darwin19.0.0):

(min) pkg> add https://github.com/JuliaBinaryWrappers/HelloWorldFortran_jll.jl
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
   Cloning git-repo `https://github.com/JuliaBinaryWrappers/HelloWorldFortran_jll.jl`
  Updating git-repo `https://github.com/JuliaBinaryWrappers/HelloWorldFortran_jll.jl`
  Updating git-repo `https://github.com/JuliaBinaryWrappers/HelloWorldFortran_jll.jl`
 Resolving package versions...
  Updating `/private/tmp/min/Project.toml`
  [b7e79332] + HelloWorldFortran_jll v1.0.0+0 #master (https://github.com/JuliaBinaryWrappers/HelloWorldFortran_jll.jl)
  Updating `/private/tmp/min/Manifest.toml`
  [b7e79332] + HelloWorldFortran_jll v1.0.0+0 #master (https://github.com/JuliaBinaryWrappers/HelloWorldFortran_jll.jl)

julia> import HelloWorldFortran_jll
[ Info: Precompiling HelloWorldFortran_jll [b7e79332-94ae-5b45-a0a6-65d827bd22db]

julia> HelloWorldFortran_jll.hello_world() do p; @show p; run(`$p`) end
p = "/Users/nowacki/.julia/artifacts/c1347606c6d5842b1bfe37daaf1cb67ea85aeafd/bin/hello_world"
dyld: Library not loaded: @rpath/libgfortran.5.dylib
  Referenced from: /Users/nowacki/.julia/artifacts/c1347606c6d5842b1bfe37daaf1cb67ea85aeafd/bin/hello_world
  Reason: image not found
ERROR: failed process: Process(`/Users/nowacki/.julia/artifacts/c1347606c6d5842b1bfe37daaf1cb67ea85aeafd/bin/hello_world`, ProcessSignaled(6)) [0]

Stacktrace:
 [1] pipeline_error at ./process.jl:525 [inlined]
 [2] #run#565(::Bool, ::typeof(run), ::Cmd) at ./process.jl:440
 [3] run at ./process.jl:438 [inlined]
 [4] (::var"#16#17")(::String) at ./REPL[11]:1
 [5] (::HelloWorldFortran_jll.var"#8#9"{var"#16#17"})() at /Users/nowacki/.julia/packages/HelloWorldFortran_jll/QBxub/src/wrappers/x86_64-apple-darwin14-libgfortran5.jl:34
 [6] withenv(::HelloWorldFortran_jll.var"#8#9"{var"#16#17"}, ::Pair{String,String}, ::Vararg{Pair{String,String},N} where N) at ./env.jl:161
 [7] #hello_world#7(::Bool, ::Bool, ::typeof(HelloWorldFortran_jll.hello_world), ::var"#16#17") at /Users/nowacki/.julia/packages/HelloWorldFortran_jll/QBxub/src/wrappers/x86_64-apple-darwin14-libgfortran5.jl:33
 [8] hello_world(::Function) at /Users/nowacki/.julia/packages/HelloWorldFortran_jll/QBxub/src/wrappers/x86_64-apple-darwin14-libgfortran5.jl:17
 [9] top-level scope at REPL[11]:1

On Linux (x86_64-pc-linux-gnu):

(min) pkg> add https://github.com/JuliaBinaryWrappers/HelloWorldFortran_jll.jl
   Cloning git-repo `https://github.com/JuliaBinaryWrappers/HelloWorldFortran_jll.jl`
  Updating git-repo `https://github.com/JuliaBinaryWrappers/HelloWorldFortran_jll.jl`
  Updating git-repo `https://github.com/JuliaBinaryWrappers/HelloWorldFortran_jll.jl`
 Resolving package versions...
  Updating `/tmp/earanow/min/Project.toml`
  [b7e79332] + HelloWorldFortran_jll v1.0.0+0 #master (https://github.com/JuliaBinaryWrappers/HelloWorldFortran_jll.jl)
  Updating `/tmp/earanow/min/Manifest.toml`
  [b7e79332] + HelloWorldFortran_jll v1.0.0+0 #master (https://github.com/JuliaBinaryWrappers/HelloWorldFortran_jll.jl)

julia> import HelloWorldFortran_jll
[ Info: Precompiling HelloWorldFortran_jll [b7e79332-94ae-5b45-a0a6-65d827bd22db]

julia> HelloWorldFortran_jll.hello_world() do p @show p; run(`$p`) end
p = "/nfs/see-fs-01_users/earanow/.julia.el7/artifacts/ebc4df1a83e63aa57057e6b205cc18b805c5fcec/bin/hello_world"
/nfs/see-fs-01_users/earanow/.julia.el7/artifacts/ebc4df1a83e63aa57057e6b205cc18b805c5fcec/bin/hello_world: error while loading shared libraries: libgfortran.so.4: cannot open shared object file: No such file or directory
ERROR: failed process: Process(`/nfs/see-fs-01_users/earanow/.julia.el7/artifacts/ebc4df1a83e63aa57057e6b205cc18b805c5fcec/bin/hello_world`, ProcessExited(127)) [127]

Stacktrace:
 [1] pipeline_error at ./process.jl:525 [inlined]
 [2] #run#565(::Bool, ::typeof(run), ::Cmd) at ./process.jl:440
 [3] run at ./process.jl:438 [inlined]
 [4] (::var"#5#6")(::String) at ./REPL[7]:1
 [5] (::HelloWorldFortran_jll.var"#8#9"{var"#5#6"})() at /nfs/see-fs-01_users/earanow/.julia.el7/packages/HelloWorldFortran_jll/QBxub/src/wrappers/x86_64-linux-gnu-libgfortran4.jl:34
 [6] withenv(::HelloWorldFortran_jll.var"#8#9"{var"#5#6"}, ::Pair{String,String}, ::Vararg{Pair{String,String},N} where N) at ./env.jl:161
 [7] #hello_world#7(::Bool, ::Bool, ::typeof(HelloWorldFortran_jll.hello_world), ::var"#5#6") at /nfs/see-fs-01_users/earanow/.julia.el7/packages/HelloWorldFortran_jll/QBxub/src/wrappers/x86_64-linux-gnu-libgfortran4.jl:33
 [8] hello_world(::Function) at /nfs/see-fs-01_users/earanow/.julia.el7/packages/HelloWorldFortran_jll/QBxub/src/wrappers/x86_64-linux-gnu-libgfortran4.jl:17
 [9] top-level scope at REPL[7]:1

Full versioninfo() for the Mac

Julia Version 1.3.0
Commit 46ce4d7933 (2019-11-26 06:09 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin19.0.0)
  CPU: Intel(R) Core(TM) i5-5287U CPU @ 2.90GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, broadwell)
Environment:
  JULIA_EDITOR = mate -w

Full versioninfo() for the Linux machine:

Julia Version 1.3.0
Commit 46ce4d7933 (2019-11-26 06:09 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, broadwell)
Environment:
  JULIA_DEPOT_PATH = /nfs/see-fs-01_users/earanow/.julia.el7
  JULIA_HOME = /apps/applications/julia/1.2.0/1/default

Request: Peco as an artifact

Peco:
https://github.com/peco/peco/releases

Requested for:
tkf/InteractiveCodeSearch.jl#6

Rebuild Xorg_xkeyboard_config_jll

By rebuilding this package we can get the path of the artifact (thanks to JuliaPackaging/BinaryBuilder.jl#525), to be used in Gtk.jl to fix JuliaGraphics/Gtk.jl#469.

Add SDL2 plugins ?

We would like to switch SimpleDirectMediaLayer.jl to BinaryProvider, I've tried to setup a builder but @giordano pointed out it was already here. One remaining problem is that we need SDL_ttf, and SDL_mixer as well. I also have a PR ready for SDL_image in SimpleDirectMediaLayer.jl, and people might want other ones in the future:

JuliaMultimedia/SimpleDirectMediaLayer.jl#39

Git_jll: Can't clone over HTTPS

When I try to use Git_jll to clone an HTTPS remote, I get the following error. Here is the stacktrace:

Cloning into 'BAZ'...
warning: templates not found in /workspace/destdir/share/git-core/templates
fatal: unable to find remote helper for 'https'
ERROR: failed process: Process(`/home/runner/.julia/artifacts/092d622dbfcb483fc58269f0da261bf5f8e42002/bin/git clone ***github.com/FOO/BAR BAZ`, ProcessExited(128)) [128]

Stacktrace:
[1] pipeline_error at ./process.jl:525 [inlined]
[2] #run#565(::Bool, ::typeof(run), ::Cmd) at ./process.jl:440
[3] run at ./process.jl:438 [inlined]
[4] (::var"#13#21"{String})(::String) at /home/runner/work/General/General/.ci/remember_to_update_registryci.jl:174
[5] (::Git_jll.var"#8#9"{var"#13#21"{String}})() at /home/runner/.julia/packages/Git_jll/vomXC/src/wrappers/x86_64-linux-gnu.jl:41
[6] withenv(::Git_jll.var"#8#9"{var"#13#21"{String}}, ::Pair{String,String}, ::Vararg{Pair{String,String},N} where N) at ./env.jl:161
[7] #git#7(::Bool, ::Bool, ::typeof(Git_jll.git), ::var"#13#21"{String}) at /home/runner/.julia/packages/Git_jll/vomXC/src/wrappers/x86_64-linux-gnu.jl:40
[8] git at /home/runner/.julia/packages/Git_jll/vomXC/src/wrappers/x86_64-linux-gnu.jl:24 [inlined]
[9] #main#11(::String, ::String, ::String, ::String, ::String, ::Array{String,1}, ::String, ::String, ::typeof(main), ::String) at /home/runner/work/General/General/.ci/remember_to_update_registryci.jl:173
[10] (::var"#kw##main")(::NamedTuple{(:registry,),Tuple{String}}, ::typeof(main), ::String) at ./none:0
[11] top-level scope at none:1
##[error]Process completed with exit code 1.
Cleaning up orphan processes

The Julia code looks something similar to this:

Git_jll.git() do git
    run(`$(git) clone https://x-access-token:$(github_token)@github.com/FOO/BAR BAZ`)
end

`jll_init` job should use the name from `meta.json`

jll_init is using the environment variable NAME to initialise the repo

Yggdrasil/azure-pipelines.yml

Lines 135 to 141 in 417371d

	- script: |
	$(JULIA) -e "using BinaryBuilder, Pkg; \
	name = \"${NAME}\"; \
	BinaryBuilder.init_jll_package(name, \
	joinpath(Pkg.devdir(), string(name, \"_jll\")), \
	string(\"JuliaBinaryWrappers/\", name, \"_jll.jl\") \
	)"

which is derived from the directory where the build_tarballs.jl script is, but I think it should use instead the name of the project as it is saved in the corresponding JSON file.

This issue caused this build to fail after creating a repo called Sundials-5_jll.jl.

CC: @staticfloat

GEOS library size is bloated

Looks like dynamic libraries of GEOS are huge in size.

In .julia\artifacts\899bb0e10d4accc34dff74b1c82ffbfe56dc8b23\bin, libgeos-3-8-0.dll and libgeos_c-1.dll are 57339 KB and 3239 KB respectively, whereas in .julia\conda\3\pkgs\geos-3.8.0-h33f27b4_0\Library\bin, geos.dll and geos_c.dll are 1346 KB and 346 KB respectively.

The files at https://github.com/JuliaBinaryWrappers/GEOS_jll.jl/releases/tag/GEOS-v3.8.0%2B1 contain a file logs/GEOS.log that suggest that -O2 was already passed to the compiler, so PR #232 was a clueless attempt to improve.

openmpi launcher doesn't work

julia> using OpenMPI_jll

julia> run(`$(joinpath(OpenMPI_jll.artifact_dir,"bin/mpiexec")) -n 1 echo "hello world"`)
/Users/simon/.julia/artifacts/f24c8af1d7b16743676909b7f55de55de4ffc8e0/bin/mpiexec: Error: unknown option "-n"
ERROR: failed process: Process(`/Users/simon/.julia/artifacts/f24c8af1d7b16743676909b7f55de55de4ffc8e0/bin/mpiexec -n 1 echo 'hello world'`, ProcessExited(1)) [1]

MPICH_jll seems to work fine.

Also, what is the correct way to launch binaries in artifacts?

mbedtls

https://github.com/JuliaWeb/MbedTLSBuilder

How to use zlib builder / release for zlib needed?

My apologies if this is dumb, but: I noticed Z/Zlib/build_tarballs.jl here, which I could use. I am already using GMP, via https://github.com/JuliaPackaging/Yggdrasil/releases/download/GMP-v6.1.2-1/build_GMP.v6.1.2.jl -- so I thought I should be able to find a zlib build.jl in a similar place, i.e., under a tag Zlib-* or something. But as far as I can tell, there is no such tag or release. Maybe I am just missing it, or am misunderstanding something at a deeper level?

Any help would be appreciated!

Package mintty for windows

It would be great if mintty could be made available through BB:

JuliaLang/julia#7267
JuliaLang/julia#33821

Add a builder for GNU Readline

Looks like there's one at https://github.com/benlorenz/readlineBuilder that uses an old version of BinaryBuilder. It'd be nice to have readline in Yggdrasil.

Win32 SuiteSparse build is subtly borked

It's installing 3 copies of every dll, and loading 2 of each at runtime.

Build opencv with cuda support

Currently cuda cannot be found by opencv. Perhaps we need nvcc etc. Can we remove this line https://github.com/JuliaPackaging/Yggdrasil/blob/master/C/CUDA/build_tarballs.jl#L43 ?

Cannot initialize Ncurses in macOS

Hi!

For some reason, I cannot initialize Ncurses using Ncurses_jll in macOS.

julia> using Ncurses_jll

julia> ccall( (:initscr, libncurses), Cvoid, ())
Error opening terminal: xterm-256color.

After that, julia crashes. Passing the env. option TERM=xterm did not help also.

I can start Ncurses normally if I use the library that ships with macOS or the library in home-brew.

On the road to GTK

• GTK (PR #85) (meson.build, meson_options.txt; ArchLinux, HomeBrew, Linux From Scratch, Debian, Spack)
  • glib2
  • gdk-pixbuf (PR #72) (ArchLinux, HomeBrew, Linux From Scratch, Debian, Spack)
    • glib2
    • [x] shared-mime-info (PR #61) (not needed when disabling gio-sniffing in gdk-pixbuf)
    • libjpeg-turbo
    • libpng
    • libtiff (PR #58)
    • libx11
    • [ ] jasper (optional)
  • Pango (#41)
  • rsvg (PR #90) (ArchLinux, HomeBrew, Linux From Scratch, Debian, Spack)
    • gdk-pixbuf (PR #72)
    • libcroco (PR #57)
    • pango
  • ATK (PR #67)
  • at-spi2-atk (only when X11 is enabled, i.e., Linux and FreeBSD) (PR #148) (meson.build, meson_options.txt; ArchLinux, Linux From Sctratch, Debian, Spack)
    • at-spi2-core (PR #146) (meson.build, meson_options.txt; ArchLinux, Linux From Sctratch, Debian, Spack)
      • dbus (PR #142)
        • [ ] audit
          • [ ] krb5
            • [ ] e2fsprogs
            • [x] keyutils (PR #112)
            • [ ] libldap
          • [x] libcap-ng (PR #120)
        • [x] expat
        • [ ] systemd-libs
          • [x] libcap (PR #137)
          • [x] libgcrypt (PR #111)
          • [x] lz4 (PR #97)
          • [x] xz
      • libxtst (part of X11)
      • Glib
  • libepoxy (PR #66) (meson.build, meson_options.txt; ArchLinux, HomeBrew, Linux From Scratch, Debian, Spack)
    • libglvnd (only Linux and FreeBSD) (PR #99) (ArchLinux)
    • x11 (optional)
  • adwaita-icon-theme (PR #68)
    • hicolor-icon-theme (PR #140)
  • Cairo
  • HarfBuzz
  • graphene (PR #77)
  • iso-codes (PR #76)
  • wayland (optional, only for Linux and FreeBSD) (PR #78)
  • wayland-protocols (optional, only for Linux and FreeBSD) (PR #96)
  • xkbcommon (only when wayland is enabled) (PR #147) (ArchLinux)
    • xkeyboard-config (ArchLinux)
      • [x] libxslt (PR #115) (ArchLinux)
        • [x] libgcrypt (PR #111) (ArchLinux)
          • [x] libgpg-error (PR #108) (ArchLinux)
      • xorg-xkbcomp (PR #141) (ArchLinux)
        • libxkbfile (in PR #92) (ArchLinux)
    • libglvnd
    • libxcb (in X11)
  • who-knows-what-else...

Other packages, depending on GTK, needed for Gtk.jl:

• gnome-themes-extra (PR #113) (configure.ac; ArchLinux, HomeBrew, Linux From Scratch, Debian)
  • GTK3 (PR #85)

Moved from IanButterworth/GTK3Builder#1.

GDAL build not linked to dependencies on Linux

I created a PR to switch GDAL.jl over to GDAL_jll at JuliaGeo/GDAL.jl#82.
The good news is that it works on OS X and Windows. Unfortunately the Linux build is not ok.

The bare GDAL build is fine, hence it passed the audits, but it is built without GEOS support, which is essential. Not sure why this only fails on Linux though. The GEOS link is made by a GEOS_jll dependency and this line:

Yggdrasil/G/GDAL/build_tarballs.jl

Lines 31 to 32 in c8e0997

	./configure --prefix=$prefix --host=$target \
	--with-geos=${bindir}/geos-config \

I'm now talking about only GEOS, but it seems from the tests all externally provided format drivers that depend on libsqlite and libcurl are also not working, on Linux only.

This is using GDAL_jll v3.0.3+0: https://github.com/JuliaBinaryWrappers/GDAL_jll.jl/releases/tag/GDAL-v3.0.3%2B0
pulling in these jll packages:

Zlib_jll ──── v1.2.11+7
SQLite_jll ── v3.30.1+1
MbedTLS_jll ─ v2.16.0+1
GEOS_jll ──── v3.8.0+4
PROJ_jll ──── v6.3.0+0
LibCURL_jll ─ v7.66.0+1
GDAL_jll ──── v3.0.3+0
LibSSH2_jll ─ v1.9.0+1

Adding Intel's Decimal Floating-Point Math Library

DecFP.jl depends on Intel's Decimal Floating-Point Math Library, the binaries of which are now distributed at https://github.com/quinnj/DecFPBuilder. I think the package is widely used and thus it's worth adding its dependencies here. If you like it, I'm happy to work on it.

Build `strace` with mpers (multiple personalities)

It should be the default, if I'm reading the configure.ac correctly, so I guess we just need to do a from-source build.

xgboost

XGBoost.jl is using BinDeps. It would be nice to have BB binaries.

https://github.com/dmlc/XGBoost.jl/blob/master/deps/build.jl

dmlc/XGBoost.jl#51