Binary packages about ruby-build

[jasonkarns]

Super old issue, but I just wanted to pop in and say that node-build (direct fork, minimally modified) has added support for installing precompiled binaries. nodenv/node-build@474e872 (source has been modified slightly since that PR was merged, though)

We grab the system info via:

platform(){
  local arch="$(uname -m)"
  local os="$(uname -s | tr '[:upper:]' '[:lower:]')"

  case "$arch" in
    x86_64 | amd64 | i686-64 ) arch=x64 ;;
    i[36]86* | [ix]86pc )      arch=x86 ;;
  esac

  echo "${os}-${arch}"
}

And a build definition looks like:

binary darwin-x64 "https://nodejs.org/dist/v4.2.1/node-v4.2.1-darwin-x64.tar.gz#658686795fac9669d411ac5c5be2da8159058e386767322d8d8151dcdb4810b9"
binary linux-arm64 "https://nodejs.org/dist/v4.2.1/node-v4.2.1-linux-arm64.tar.gz#05df4aeb8a53798f8b10074600518040fc317f2919f9755aeab57b0aaf7227b0"
binary linux-armv6l "https://nodejs.org/dist/v4.2.1/node-v4.2.1-linux-armv6l.tar.gz#6f29286464efda4a362d9464d82733398b98882051c20c0f190004480b6e506a"
binary linux-armv7l "https://nodejs.org/dist/v4.2.1/node-v4.2.1-linux-armv7l.tar.gz#fb4fbef9306962e800804ab5ba615c06bba28deb4e7a3e945a291dba986ef816"
binary linux-x64 "https://nodejs.org/dist/v4.2.1/node-v4.2.1-linux-x64.tar.gz#e766e387934e17daaad92d0460ed76f756655da62b627a5c9cc07faea4a0b824"
binary linux-x86 "https://nodejs.org/dist/v4.2.1/node-v4.2.1-linux-x86.tar.gz#97b5ccea7044073c87a21bcc4b0762f4a6bd77db9cc958206f684ecdfeb89b1f"
binary sunos-x64 "https://nodejs.org/dist/v4.2.1/node-v4.2.1-sunos-x64.tar.gz#acec44790ff0069620c0fd03945d14b9f97c7ccb0a0450f766a5cbe4a906510b"
binary sunos-x86 "https://nodejs.org/dist/v4.2.1/node-v4.2.1-sunos-x86.tar.gz#d7bf8dcf353115b5e55bf64d25c2c34fa2d237e201e46dab97ae3d8e3f051583"

install_package "node-v4.2.1" "https://nodejs.org/dist/v4.2.1/node-v4.2.1.tar.gz#8861b9f4c3b4db380fcda19a710c0430c3d62d03ee176c64db63eef95a672663"

How it works:

The binary function is invoked multiple times from the build definition; given the platform and package url. If the binary's platform matches the current system, the package url is saved to a global variable.

Then when install_package is invoked, if a global BINARY_URL has been set (and the user hasn't forced a compile with --compile), it does an install_package ... copy.

It's a pretty hacky workflow, and I would love to find a more elegant solution. But it works and, I think, could be adopted for ruby-build.

[mislav]

No progress yet. Anyone is welcome to pick this up!

Some notes:

• how rvm builds "movable" ruby
• how Travis builds them (uses rvm)
• how we would build them, in theory:
  1. In our CI, use build flags to make rubies "movable",
  2. (Non-critical) strip unnecessary files from the resulting build,
  3. Upload packaged binaries up to S3,
  4. On the user end, allow an opt-in flag ruby-build --binary 2.6.5 /path/to/destination that first tries to download and extract pre-build binaries,
  5. Eventually, if all this process yields stable result, enable --binary flag by default so everyone can benefit from fast Ruby installs on supported platforms (probably macOS plus several propular Linux distros).

[sstephenson]

Crazy idea: build Ruby with an unreasonably long --prefix, like

/ruby-build/placeholder----------------------------------------------------------------/1.9.3-rc1

...and search-and-replace that with the actual installation location after extracting the package.

We'd need to do this for the shebangs for commands like irb too.

[md5]

@mconigliaro If you want to do that, you can use rbenv by itself and not use ruby-build since its purpose is to build Ruby for rbenv.

From the rbenv docs:

Alternatively to the install command, you can download and compile Ruby manually as a subdirectory of ~/.rbenv/versions/. An entry in that directory can also be a symlink to a Ruby version installed elsewhere on the filesystem. rbenv doesn't care; it will simply treat any entry in the versions/ directory as a separate Ruby version.

[mattes]

Upload packaged binaries up to S3

Can we re-use http://rvm.io/binaries (cached by fastly)? Unless someone else wants to pay for the S3 traffic :-)

[sstephenson]

Looks like --prefix is used to define the initial load path. It's compiled into the ruby binary itself:

$ strings bin/ruby | grep rbenv
/Users/sam/.rbenv/versions/1.9.3-dev
/Users/sam/.rbenv/versions/1.9.3-dev/lib/ruby/site_ruby/1.9.1
/Users/sam/.rbenv/versions/1.9.3-dev/lib/ruby/site_ruby/1.9.1/x86_64-darwin11.1.0
/Users/sam/.rbenv/versions/1.9.3-dev/lib/ruby/site_ruby
/Users/sam/.rbenv/versions/1.9.3-dev/lib/ruby/vendor_ruby/1.9.1
/Users/sam/.rbenv/versions/1.9.3-dev/lib/ruby/vendor_ruby/1.9.1/x86_64-darwin11.1.0
/Users/sam/.rbenv/versions/1.9.3-dev/lib/ruby/vendor_ruby
/Users/sam/.rbenv/versions/1.9.3-dev/lib/ruby/1.9.1
/Users/sam/.rbenv/versions/1.9.3-dev/lib/ruby/1.9.1/x86_64-darwin11.1.0

Obviously these paths aren't going to be the same for everyone. Can we work around this?

[josh]

Skeptical about that. Compiled constants are going to be offset by their length. So replacing the placeholder with a shorter string is probably going to fuck up the memory addresses off all the other constants after it in the data segment.

[sstephenson]

We'd keep the strings the same length, obviously, and just replace the rest of the space with null bytes.

Proof of concept: https://gist.github.com/a95fd5e818f9237a1c9f

Save as rewrite.sh and run like so:

$ ./rewrite.sh < bin/ruby > bin/ruby2

[clowder]

I was digging round the MacRuby source today and came across install_name_tool and instantly thought of this ticket.

On MacRuby it is being used to replace linking to the system MacRuby with one packaged inside the executable (see MacRuby/MacRuby@master).

EDIT: Also useful headerpad_max_install_names

[fschwahn]

so... what happened to this? isn't it ready to go, I remember you tweeting about this a few month ago? if not, is there anything that needs to be done?

I really like the prospect to install ruby in a few seconds instead of a few minutes :)

[john-griffin]

Is there anything we can do to help out with this and get it finished?

[haberbyte]

Here is an interesting blog post: http://yehudakatz.com/2012/06/

I think Yehuda has solved that problem already!

[petejkim]

is it possible by any chance to use rvm's precompiled packages?

[rosenfeld]

When using precompiled packages ruby-build should probably build them with optimization flags (-O3 or -O2) turned on.

Also, they should be also the default when compiling from source. In that case, it would be a good idea to also add -march=native.

[pranas]

Any progress? It's highly desirable feature.

[ptrf]

👍 Any progress? Post a gist maybe?

[barttenbrinke]

+1 Rubber (Ec2 automation) recently switched to Ruby-build (which is great!) but compiling Ruby 2.0 for each Amazon instance you spin up is something the world can really do without.

[loopj]

👍 would love to see support for this

[mattes]

+1
see here also:
tokaido/tokaido-build#6 (comment)

[sheerun]

Support for precompiled ruby would be awesome. +1

[sheerun]

I made plugin for that https://github.com/sheerun/ruby-download It uses binaries precompiled by Boxen.

[garnieretienne]

I also made a plugin to download precompiled rubies from the RVM repository: https://github.com/garnieretienne/rvm-download. It's pretty basic but I can add functionnalities if other people are interrested (like listing available precompiled rubies or checksum verifications)

[kiranos]

as @habermann24 stated in his link: http://yehudakatz.com/2012/06/

Fortunately, the C Ruby folks know about this problem, and include an (undocumented?) 
flag that you can pass to ./configure, --enable-load-relative. This flag fixes the problems
with hardcoded paths:

Adding precompiled binaries would absolutely bee a great new feature. What technical difficulties are still present to make this happen other than structuring it?

[mislav]

I'd love if we had precompiled binaries support. It seems that a lot of people on this thread did some research, and it would be great if someone submitted a pull with a proof-of-concept:

1. What are the flags that we should use when compiling Ruby to make it reusable for other people
2. How do we match a user's system to a precompiled Ruby version?
3. What about dependencies such as libyaml and openssl? We install those explicitly for some Ruby definitions but also sometimes use system/Homebrew ones. For precompiled Rubies, we must include all dependencies. How do we precompile a Ruby version with consistent zlib/libyaml/openssl support?

[errm]

So I had a look at what rvm are doing, with https://github.com/rvm/rvm-binary, they are setting a bunch of flags with rmv install --movable I have had a trawl through the source to work out what flags they are setting but I can't quite work it out, it looks like they are setting CFLAGS="-fPIC" LDFLAGS="-fPIC". Also what they are doing only seems to work for 1.9.3 and above. They are using smf to install dependencies.

I created a gist with the config from the binaries built in this way on ubuntu 14.04 https://gist.github.com/errm/75474a643b8e5d09a444

The generated binaries work fine once the simlink in /lib/ruby/gems/*/cache is removed and replaced with a dir . . .

[michiels]

I am currently working on this problem and my goal is to be able to provide/host Ruby binaries that are built with ruby-build so they can be used as rbenv binaries.

The first step was testing if the compile options in the previous comments on this issue would produce a sharable Ruby directory.

I have set up a a build infrastructure that can build Ruby versions on Ubuntu 12.04 and Ubuntu 14.04. I am publishing those binaries at this location: https://intercityup.com/binaries/ruby/ with PGP signed SHA1 checksums.

The binaries are compiled with

RUBY_CONFIGURE_OPTS=--enable-load-relative ruby-build <version> <version>

It would be great if someone could test those binaries as "drop-in" placement into their .rbenv/versions directory to see if those work. I have already confirmed them personally by downloading them into my rbenv installation, installing some gems like bundler and rails and set up a Rails app with the native mysql2 extension.

On a Ubuntu 14.04 box or Vagrant you can test a binary version with:

$ cd ~/.rbenv/versions/
$ wget https://intercityup.com/binaries/ruby/ubuntu/14.04/x86_64/ruby-2.1.2.tar.bz2
$ tar jxf ruby-2.1.2.tar.bz2
$ rbenv versions
# Verify 2.1.2 shows up!
$ rbenv shell 2.1.2
$ ruby -v

I would really like to hear if they work and what people's thoughts are on:

• Hosting (I'm happy to sponsor the primary CDN from https://intercityup.com/binaries/) but I can understand if the best place for this is the mirror 37s sponsors via CloudFront.
• Implementing as a plugin (like rbenv download) or
• Maybe implement rbenv install --binary

Thanks for testing in advance and I hope we can make rbenv binaries happen real soon!

Michiel

[mislav]

On Wed, Jul 30, 2014 at 10:55 AM, Michiel Sikkes [email protected]
wrote:

Thanks for testing in advance and I hope we can make rbenv binaries happen
real soon!

Yay, this sounds really exciting!

Yes, the best place to host the official binaries would be our own
CloudFront.

I would like to see a pull request from you adding two things:

1. A script for maintainers to generate and publish binaries. As for the
   technical aspect of how to push to our CloudFront, see how we use s3-put.
2. An option for ruby-build (and rbenv-install) to try downloading a
   pre-built binary before it would actually compile a Ruby version.

The most important thing in this whole process is how to match the current
system to the appropriate binary name. I think we need 4 components for
every unique binary build:

1. OS name
2. OS version
3. CPU architecture
4. Ruby version

So I guess we need ways to detect people's OS information.

[michiels]

Thank you Mislav. I'll start a PR with those two components.

For recognizing OS variables we can use the lsb_release on most distributions. For OS X we might need to use something different but it shouldn't be that hard with a lot of other tools already doing this.

My opinion is that right now it is out of scope to set up a script to launch build nodes on EC2 or DigitalOcean. So I think it's best to only include a "publish" script that can be run similar to gem publish. This will use the S3 push.

I will share our current build automation infrastructure scripts too. They can help maintainers build packages in one go.

Does all this make sense / are there any additional thoughts?

Michiel

[mislav]

On Thu, Jul 31, 2014 at 2:05 PM, Michiel Sikkes [email protected]
wrote:

My opinion is that right now it is out of scope to set up a script to
launch build nodes on EC2 or DigitalOcean.

Sounds awesome as the next step. But first let's publish a few manually and
have people test them out.

lsb_release sounds alright for Ubuntu and similar but I'm not sure if all
Linux have them.

[michiels]

Op 31 jul. 2014 om 23:38 heeft Mislav Marohnić [email protected] het volgende geschreven:

lsb_release sounds alright for Ubuntu and similar but I'm not sure if all
Linux have them.

Good point. I'll take a look at how chef/ohai/puppet do these types of OS recognition.

If anyone has ideas for a common method of extracting OS info: please post it! Thanks

[brocktimus]

lsb_release sounds alright for Ubuntu and similar but I'm not sure if all Linux have them.

Can confirm it isn't available on the RHEL variants, at least it can't be relied upon. It's available in an RPM but not installed by default on minimal at least.

Some details available here http://serverfault.com/questions/89654/what-version-of-rhel-am-i-using.

[michiels]

Can confirm it isn't available on the RHEL variants, at least it can't be relied upon. It's available in an RPM but not installed by default on minimal at least.

Some details available here http://serverfault.com/questions/89654/what-version-of-rhel-am-i-using

Thanks! Is the file /etc/lsb_release available?

We might need to build a lot of logic just for OS recognition. Or we can use an external dependency.

Is there any policy for using dependencies in ruby-build? I would assume the answer is "yes. 0" :)

Otherwise we might begin only recognizing common OSes (lsb_release, red hat, Mac, etc) like @sstephenson suggested in the starting comment of this issue.

[brocktimus]

Thanks! Is the file /etc/lsb_release available? We might need to build a lot of logic just for OS recognition. Or we can use an external dependency.

Nope.

I don't know if a dependency of lsb-release would be fine if you wanted the extra functionality. But I'm unsure how far it gets you compared to the alternatives. The other alternatives don't seem that bad since you can use lsb_release if it exists or otherwise check /etc/debian_version or /etc/redhat-release to grab the entire family. I have no idea about OSX though ;-).

http://danielgibbs.co.uk/2013/04/bash-how-to-detect-os/
http://unix.stackexchange.com/questions/6345/how-can-i-get-distribution-name-and-version-number-in-a-simple-shell-script

[michiels]

I don't know if a dependency of lsb-release would be fine if you wanted the extra functionality. But I'm unsure how far it gets you compared to the alternatives. The other alternatives don't seem that bad since you can use lsb_release if it exists or otherwise check /etc/debian_version or /etc/redhat-release to grab the entire family. I have no idea about OSX though ;-).

http://danielgibbs.co.uk/2013/04/bash-how-to-detect-os/
http://unix.stackexchange.com/questions/6345/how-can-i-get-distribution-name-and-version-number-in-a-simple-shell-script

Thanks. I'll take a look!

[garnieretienne]

For info, this is how RVM detect the current OS to download its precompiled ruby in bash:
https://github.com/wayneeseguin/rvm/blob/98c0a3aadfcade0b9b34701978c4eacfbcc17eba/scripts/functions/utility_system#L3

They seems to get the following data: system info, system type, system name, system version, system arch.

• They use the uname command to get the system type (Linux, Darwin, Windows...)
• If Linux is detected as the system type, they use a number of file name and content to find out which is the system name and version (and sometime system arch) (/etc/lsb-release, /etc/altlinux-release, /etc/os-release, /etc/debian_version, ...)
• If non Linux they use dedicated command for each system to find system name, version and arch

At the end they sanitarize each metadata and build the download URL using them.

[mislav]

Here's how I intend to extract basic OS information in ruby-build for build output purposes:

os_information() {
  if type -p lsb_release >/dev/null; then
    lsb_release -sir | xargs echo
  elif type -p sw_vers >/dev/null; then
    echo "OS X $(sw_vers -productVersion)"
  else
    local os="$(cat /etc/{centos,redhat,fedora,system}-release /etc/debian_version 2>/dev/null | head -1)"
    echo "${os:-$(uname -sr)}"
  fi
}

Of course, such a simplistic approach won't be enough for differentiating between binary builds. For example, it misses the information about system arch.

[martinb3]

@sheerun Did ruby-download go away? It sounds useful!

[sheerun]

You probably mean ruby-download. I removed it because Boxen removed most of their builds.

I could reintroduce it, and change provider to http://phusion.github.io/traveling-ruby/ BUT it supports only Ruby 2.1. So it all depends whether @FooBarWidget wishes to support more ruby versions.

[martinb3]

@sheerun It looks like TravisCI and rvm.io also provide binary builds.

[sheerun]

@FooBarWidget Do you know the difference between Travelling Ruby build & Travis?

I can't find rvm's builds. @martinb3 Do you know the API for them?

[martinb3]

@sheerun I know they're hosted here: https://rvm.io/binaries/

[FooBarWidget]

@sheerun No. Traveling Ruby's scope is limited to distributing apps. It's not meant to distribute Ruby binaries for general-purpose or development use. For that reason, we also have no intention to distribute more than a single version.

[garnieretienne]

@sheerun @martinb3 I've made this plugin to download and install rvm binaries for rbenv:
https://github.com/garnieretienne/rvm-download

It use code from rvm to generate an URL and download the requested version of ruby from https://rvm.io/binaries/.

[schlueter]

Any movement on this?

[md5]

@schlueter 👍

[cybertk]

+1

[mconigliaro]

Wow, this still isn't solved? To be honest, I don't even care about getting your binaries to work on my machines. If there was just a simple way to check for a precompiled version of Ruby before building (e.g. RUBY_BUILD_BINARY_URL?), I'd just build my own and host them myself.

[mconigliaro]

@md5 oh yeah, that's a good point. I could always just have my config management system download and unpack a set of precompiled versions to ~/.rbenv/versions/. Thanks!

[mislav]

@jasonkarns Thanks for sharing! That's an interesting approach to fetching precompiled binaries.

In my opinion, it's not such a problem to make ruby-build fetch precompiled binaries as it is to make a system to package and publish precompiled binaries in the first place.

[koenpunt]

Travis actually has precompiled rubies: http://rubies.travis-ci.org.

Edit; I believe these are build against local libraries, so are not universally usable;

dyld: Library not loaded: /Users/travis/.rvm/rubies/ruby-2.3.3/lib/libruby.2.3.0.dylib
  Referenced from: /Users/xcode7/.rbenv/versions/2.3.3/bin/ruby
  Reason: image not found

[AlexWayfer]

No progress here? Or any new ideas?

[wlnirvana]

any update?

[rafaelsales]

Any progress? RVM makes ruby binaries for ages and it's just fine

[technicalpickles]

Posted this on a duplicate #1054 :

GitHub packages is an option now too. Homebrew recently moved to it from bintray recently.

That doesn't address building the packages though.

Some notes from reading through about how RVM does it:

• rvm has a --movable flag
• it's documented at https://github.com/rvm/rvm/blob/35e894c7070eaabc38d91220f6e26d1585a7098a/help/install.md#movable-rubies
• there is logic for checking which environments and ruby versions can be built movable: https://github.com/rvm/rvm/blob/35e894c7070eaabc38d91220f6e26d1585a7098a/scripts/functions/build_config#L32-L77
• when it comes to building:
  • on macos, there's a patch applied for static building, and a set of configure flags
  • otherwise there's a configure flag (--enable-load-relative)

Edit: correction, the osx patch is empty:
https://github.com/rvm/rvm/blob/35e894c7070eaabc38d91220f6e26d1585a7098a/patchsets/ruby/osx_static

[technicalpickles]

Some more thoughts after the initial work and some time to think...

I'm considering if there should be a separate command for the fetch binary ruby from source or not. I'm also thinking through how we'd do checksums on the tarballs.

For hosting of them, homebrew recently moved from bintray to GitHub Packages. It uses skopeo to work with remote images registries (which GitHub Packages is), like retrieving information, images, signing content. I'm not sure how they are building the tarballs to publish yet though.

The big question there for me is how to organize things, and how to automate building of the various versions. My initial thought is like:

• add a 'package' for each ruby+version combo on the rbenv org, ie ruby/3.0.1
• use a timestamp and/or sha of ruby-build for to allow rebuilding when there's an update
  • I don't think they always need to be rebuilt, but I suspect it will be necessary to push an update if there's a bug in ruby-build. I don't know how to manage that without requiring always building (and thus always having updates) or manually doing it

Something I'd like to enable as well is to allow organizations to be publishing their own builds, like if they have custom patches or configuration flags they use. This would also give the option to experiment with this before having a working solution for the hosting side of things.

[wlnirvana]

Any update on this? Ten years have passed since this issues is raised🤦

[eregon]

It's pretty hard and fragile to create movable CRuby builds, see #1761 (comment)
So basically binary builds of CRuby are a problem, they need to be at least (OS, OS version, C compiler path, system libssl version and path, architecture, destination path and probably more)-specific.
And since ruby-build allows installing to any path it could only use those if it's all matching.

IMHO it's fast enough and more reliable to build from source.
If the user's setup is messed up for compilation it won't work when installing C extensions anyway.

[aleksandrs-ledovskis]

It would definitely be a bonus if "ruby-build" provides pre-built packages for multitude of aforementioned OS runtime combinations. However, the situation 10 years ago vs. today is quite different with widespread proliferation and usage of containers which subsequently reduces a need to have precompiled interpeters for OP's stated use case "[b]inary packages would make it much quicker to get up and running".