progrium / buildstep

Buildstep uses Docker and Buildpacks to build applications like Heroku

Hello, I'm Jeff! 👋

I'm a creator and entrepreneur known for my work and ideas in open source developer tooling.

I try to push the boundaries of computing as a magical tool and compiler for the imagination.

Since you found me on GitHub, you can join my Discord.

buildstep's Issues

Changed VHOST and rebuilt, can't deploy anymore

I got a new DNS entry for our Dokku box I wanted to use it. I updated the VHOST file in /home/git and ran make install in the dokku directory to reinstall dokku. When I first pushed a new app, the new DNS entry was working perfectly.

However when I rebooted the box a couple times (to remove VMware snapshots and such), I started to get 504 Gateway Time-out for all of our apps. And when I tried to redeploy some of them, I got this for Ruby apps:

Counting objects: 26, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (23/23), done.
Writing objects: 100% (26/26), 2.50 KiB | 0 bytes/s, done.
Total 26 (delta 7), reused 0 (delta 0)
-----> Building ruby-example ...
       Ruby/Rack app detected
/usr/lib/ruby/1.9.1/psych.rb:297:in `initialize': No such file or directory - ruby_versions.yml (Errno::ENOENT)
remote: from /usr/lib/ruby/1.9.1/psych.rb:297:in `open'
remote: from /usr/lib/ruby/1.9.1/psych.rb:297:in `load_file'
remote: from /build/buildpacks/heroku-buildpack-ruby/lib/language_pack/ruby.rb:216:in `block (2 levels) in ruby_versions'
remote: from /build/buildpacks/heroku-buildpack-ruby/lib/language_pack/ruby.rb:214:in `chdir'
remote: from /build/buildpacks/heroku-buildpack-ruby/lib/language_pack/ruby.rb:214:in `block in ruby_versions'
remote: from /usr/lib/ruby/1.9.1/tmpdir.rb:83:in `mktmpdir'
remote: from /build/buildpacks/heroku-buildpack-ruby/lib/language_pack/ruby.rb:213:in `ruby_versions'
remote: from /build/buildpacks/heroku-buildpack-ruby/lib/language_pack/ruby.rb:265:in `block in install_ruby'
remote: from /build/buildpacks/heroku-buildpack-ruby/lib/language_pack/instrument.rb:19:in `block (2 levels) in instrument'
remote: from /build/buildpacks/heroku-buildpack-ruby/lib/language_pack/instrument.rb:41:in `yield_with_block_depth'
remote: from /build/buildpacks/heroku-buildpack-ruby/lib/language_pack/instrument.rb:18:in `block in instrument'
remote: from /usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
remote: from /build/buildpacks/heroku-buildpack-ruby/lib/language_pack/instrument.rb:17:in `instrument'
remote: from /build/buildpacks/heroku-buildpack-ruby/lib/language_pack/base.rb:42:in `instrument'
remote: from /build/buildpacks/heroku-buildpack-ruby/lib/language_pack/base.rb:38:in `instrument'
remote: from /build/buildpacks/heroku-buildpack-ruby/lib/language_pack/ruby.rb:260:in `install_ruby'
remote: from /build/buildpacks/heroku-buildpack-ruby/lib/language_pack/ruby.rb:87:in `block in compile'
remote: from /build/buildpacks/heroku-buildpack-ruby/lib/language_pack/instrument.rb:19:in `block (2 levels) in instrument'
remote: from /build/buildpacks/heroku-buildpack-ruby/lib/language_pack/instrument.rb:41:in `yield_with_block_depth'
remote: from /build/buildpacks/heroku-buildpack-ruby/lib/language_pack/instrument.rb:18:in `block in instrument'
remote: from /usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
remote: from /build/buildpacks/heroku-buildpack-ruby/lib/language_pack/instrument.rb:17:in `instrument'
remote: from /build/buildpacks/heroku-buildpack-ruby/lib/language_pack/base.rb:42:in `instrument'
remote: from /build/buildpacks/heroku-buildpack-ruby/lib/language_pack/base.rb:38:in `instrument'
remote: from /build/buildpacks/heroku-buildpack-ruby/lib/language_pack/ruby.rb:84:in `compile'
remote: from /build/buildpacks/heroku-buildpack-ruby/bin/compile:12:in `block (2 levels) in <main>'
remote: from /build/buildpacks/heroku-buildpack-ruby/lib/language_pack/base.rb:108:in `log'
remote: from /build/buildpacks/heroku-buildpack-ruby/bin/compile:11:in `block in <main>'
remote: from /build/buildpacks/heroku-buildpack-ruby/lib/language_pack/instrument.rb:36:in `call'
remote: from /build/buildpacks/heroku-buildpack-ruby/lib/language_pack/instrument.rb:36:in `block in trace'
remote: from /build/buildpacks/heroku-buildpack-ruby/lib/language_pack/instrument.rb:19:in `block (2 levels) in instrument'
remote: from /build/buildpacks/heroku-buildpack-ruby/lib/language_pack/instrument.rb:41:in `yield_with_block_depth'
remote: from /build/buildpacks/heroku-buildpack-ruby/lib/language_pack/instrument.rb:18:in `block in instrument'
remote: from /usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
remote: from /build/buildpacks/heroku-buildpack-ruby/lib/language_pack/instrument.rb:17:in `instrument'
remote: from /build/buildpacks/heroku-buildpack-ruby/lib/language_pack/instrument.rb:36:in `trace'
remote: from /build/buildpacks/heroku-buildpack-ruby/bin/compile:9:in `<main>'

And started to get this for Python apps:

Counting objects: 421, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (258/258), done.
Writing objects: 100% (421/421), 114.88 KiB | 0 bytes/s, done.
Total 421 (delta 266), reused 246 (delta 158)
-----> Building jepsen ...
       Python app detected
-----> No runtime.txt provided; assuming python-2.7.4.
-----> Preparing Python runtime (python-2.7.4)
 !     Requested runtime (python-2.7.4) was not found.
 !     Aborting.  More info: https://devcenter.heroku.com/articles/python-support

Any help would be greatly appreciated. Thank you.

Use $app_root and $build_root instead of hard coding /app and /build

In stack/builder you define app_root, but then use it only sometimes.

For example you have:

cp -r /app /build

which I think should become

cp -r $app_root $build_root

HHVM

Could you add HHVM?

https://github.com/hhvm/heroku-buildpack-hhvm

Wrong permissions - non root

I just tested master branch and I'm getting this error during push:

remote: /build/compile.sh: line 3: dirname: command not found
remote: /build/compile.sh: line 3: /config/paths.sh: No such file or directory

I think that it has something to do with strange permissions on /usr and /var directories.

$ docker run progrium/buildstep ls -la /
total 76
drwxr-xr-x  62 root root 4096 Oct 24 11:11 .
drwxr-xr-x  62 root root 4096 Oct 24 11:11 ..
-rwxr-xr-x   1 root root    0 Oct 24 11:11 .dockerenv
-rwxr-xr-x   1 root root    0 Oct 24 11:11 .dockerinit
drwxr-xr-x   2 root root 4096 Sep 30 06:35 bin
drwxr-xr-x   2 root root 4096 Apr 10  2014 boot
drwxr-xr-x   3 root root 4096 Oct 24 08:18 build
drwxr-xr-x   4 root root  340 Oct 24 11:11 dev
drwxr-xr-x  78 root root 4096 Oct 24 11:11 etc
drwxr-xr-x   2 root root 4096 Apr 10  2014 home
drwxr-xr-x  13 root root 4096 Sep 30 06:35 lib
drwxr-xr-x   2 root root 4096 Sep 26 15:44 lib64
drwxr-xr-x   2 root root 4096 Sep 26 15:35 media
drwxr-xr-x   2 root root 4096 Apr 10  2014 mnt
drwxr-xr-x   2 root root 4096 Sep 26 15:35 opt
dr-xr-xr-x 118 root root    0 Oct 24 11:11 proc
drwx------   2 root root 4096 Oct 24 09:23 root
drwxr-xr-x   6 root root 4096 Sep 26 15:45 run
drwxr-xr-x   2 root root 4096 Sep 30 06:41 sbin
drwxr-xr-x   2 root root 4096 Sep 26 15:35 srv
dr-xr-xr-x  13 root root    0 Oct 24 11:11 sys
drwxrwxrwt   3 root root 4096 Oct 24 08:38 tmp
d--x--x---  16 root root 4096 Oct 24 08:32 usr
d--x--x---  16 root root 4096 Oct 24 08:21 var

/build/compile.sh: line 91: /tmp/build/.profile.d/00_config_vars.sh: No such file or directory

I'm getting this on a push or rebuild after updating to HEAD a few days ago.

root@tetrad:~/code/dokku# dokku rebuild logstash
-----> Cleaning up ...
-----> Building logstash ...
-----> Checking deploymentkeys Plugin sanity ...
-----> Installing shared SSH keys in build environment ...
-----> Fetching custom buildpack
-----> static app detected
-----> Bundling Apache v2.2.19
-----> Discovering process types
       Default process types for static -> web
/build/compile.sh: line 91: /tmp/build/.profile.d/00_config_vars.sh: No such file or directory
root@tetrad:~/code/dokku# 

Allow to switch the buildpack branch

Allow to switch the buildpack branch on custom buildpacks by adding #branch to the URL, just like on Heroku.

For example:

BUILDPACK_URL=git://github.com/CHH/heroku-buildpack-php#development

Ruby Jekyll Buildpack

I was wondering if anyone with more capable skills than I, had thought about adding the Ruby-Jekyll buildpack https://github.com/mattmanning/heroku-buildpack-ruby-jekyll for use in dokku.

You would certainly have at least one very grateful user.

Add Scala as supported buildpack

https://github.com/heroku/heroku-buildpack-scala

Also, you should clarify on the README that the Play buildpack is for play1. Play 2 uses the scala buildpack

Go buildpack doesn't reload NGINX

If I deploy an Go application, the files are all created, but the server does not respond with it, it just shows the first app I deployed..

If I ssh in and restart nginx, it works fine from then on. I'm not sure if this is a general issue, or specific to the Go build pack (I would have thought that the reboot of NGINX was external to the build pack as this would need to be done each time

Steps to reproduce

deploy an app to dokku
visit it, it works
deploy a new go app to dokku
visit it, shows first app deployed
ssh in, restart nginx
visit go app, it works

Expected
Works without nginx reboot.

/var/lib/apt/lists/lock - open (13: Permission denied)

Hi!

I desperately try to get one out of many build packs for dokku running. Everything works great for the standard packages (node.js, etc.) but the custom ones all fail with the error message below. In this example it is rhy-jot/buildpack-nginx but the same for other custom packages which use apt to install packages.

Any ideas? I've already rebooted, re-deployed, ...

remote: HEAD is now at 6380bff... Back to nginx
-----> Fetching custom buildpack
-----> nginx app detected
remote: E: Could not open lock file /var/lib/apt/lists/lock - open (13: Permission denied)
remote: E: Unable to lock directory /var/lib/apt/lists/
remote: E: Could not open lock file /var/lib/dpkg/lock - open (13: Permission denied)
remote: E: Unable to lock the administration directory (/var/lib/dpkg/), are you root?

Timezones

I'm not sure if this question belongs here or in the Dokku issues, so I've done so on both (sorry).

I arrived back into work this morning after losing an hour of my Sunday to find that my Docker containers are still running in GMT-0, whereas we're now in GMT+1.

My host server has updated its time properly, however the containers are stuck an hour behind us. It's extremely important to us that they stay in sync since we use them to run a lot of automated processes each morning. This process is now 1 hour behind.

I've tried restarting them but they're still -1 hour. Any ideas?

/build/compile.sh: line 24: cd: /app: Permission denied

Hi!

I get this error when building a Php app (Joomla CMS).
I've done some investigations and I really don't understand what is going on.
It seems that after switching user with /usr/bin/setuidgid the user doesn't have the permissions to read in /app

Here the result of my tests :

Before switching user (so as root) :

ls -hal /

remote: total 84K
remote: drwxr-xr-x  51 root   root   4.0K Dec 19 16:20 .
remote: drwxr-xr-x  51 root   root   4.0K Dec 19 16:20 ..
remote: -rwxr-xr-x   1 root   root      0 Dec 19 16:20 .dockerenv
remote: -rwxr-xr-x   1 root   root      0 Dec 19 16:20 .dockerinit
remote: drwxrwxrwx  32 u30655 u30655 4.0K Dec 19 16:20 app
remote: drwxr-xr-x   2 root   root   4.0K Dec 17 17:40 bin
remote: drwxr-xr-x   2 root   root   4.0K Apr 10  2014 boot
remote: drwxr-xr-x   3 root   root   4.0K Dec 19 16:19 build
remote: drwxr-xr-x   3 u30655 u30655 4.0K Dec 19 13:55 cache
remote: drwxr-xr-x   5 root   u30655  360 Dec 19 16:20 dev
remote: drwxr-xr-x  88 root   root   4.0K Dec 19 16:20 etc
remote: drwxr-xr-x   2 root   root   4.0K Apr 10  2014 home
remote: drwxr-xr-x  13 root   root   4.0K Dec 17 17:40 lib
remote: drwxr-xr-x   2 root   root   4.0K Dec 17 17:39 lib64
remote: drwxr-xr-x   2 root   root   4.0K Nov 25 18:25 media
remote: drwxr-xr-x   2 root   root   4.0K Apr 10  2014 mnt
remote: drwxr-xr-x   2 root   root   4.0K Nov 25 18:25 opt
remote: dr-xr-xr-x 213 root   root      0 Dec 19 16:20 proc
remote: drwx------   2 root   root   4.0K Nov 25 18:25 root
remote: drwxr-xr-x   7 root   root   4.0K Dec 17 17:40 run
remote: drwxr-xr-x   2 root   root   4.0K Nov 25 18:29 sbin
remote: drwxr-xr-x   2 root   root   4.0K Nov 25 18:25 srv
remote: dr-xr-xr-x  13 root   root      0 Dec 19 15:24 sys
remote: drwxrwxrwt   6 u30655 u30655 4.0K Dec 19 16:20 tmp
remote: drwxr-xr-x  16 root   root   4.0K Nov 25 18:25 usr
remote: drwxr-xr-x  19 root   root   4.0K Dec 19 16:20 var


ls -hal /app

remote: total 144K
remote: drwxrwxrwx 32 u30655 u30655 4.0K Dec 19 16:20 .
remote: drwxr-xr-x 51 root   root   4.0K Dec 19 16:20 ..
remote: -rw-r--r--  1 u30655 u30655   51 Dec 19 16:20 .env
remote: -rw-r--r--  1 u30655 u30655   86 Dec 19 16:20 .gitignore
remote: -rw-r--r--  1 u30655 u30655 3.1K Dec 19 16:20 .htaccess
remote: -rw-r--r--  1 u30655 u30655  18K Dec 19 16:20 LICENSE.txt
remote: -rw-r--r--  1 u30655 u30655   55 Dec 19 16:20 Procfile
remote: -rw-r--r--  1 u30655 u30655 4.3K Dec 19 16:20 README.txt
remote: -rw-r--r--  1 u30655 u30655   23 Dec 19 16:20 SCALE
remote: drwxr-xr-x 16 u30655 u30655 4.0K Dec 19 16:20 administrator
remote: drwxr-xr-x  2 u30655 u30655 4.0K Dec 19 16:20 bin
remote: drwxr-xr-x  2 u30655 u30655 4.0K Dec 19 16:20 cli
remote: drwxr-xr-x 32 u30655 u30655 4.0K Dec 19 16:20 components
remote: -rw-r--r--  1 u30655 u30655  362 Dec 19 16:20 composer.json
remote: drwxr-xr-x  2 u30655 u30655 4.0K Dec 19 16:20 config
remote: -rw-r--r--  1 u30655 u30655 2.5K Dec 19 16:20 configuration.php
remote: drwxr-xr-x  2 u30655 u30655 4.0K Dec 19 16:20 images
remote: drwxr-xr-x  2 u30655 u30655 4.0K Dec 19 16:20 includes
remote: -rw-r--r--  1 u30655 u30655 1013 Dec 19 16:20 index.php
remote: drwxr-xr-x  6 u30655 u30655 4.0K Dec 19 16:20 jbox-sql
remote: -rw-r--r--  1 u30655 u30655 1.9K Dec 19 16:20 joomla.xml
remote: drwxr-xr-x  8 u30655 u30655 4.0K Dec 19 16:20 language
remote: drwxr-xr-x  6 u30655 u30655 4.0K Dec 19 16:20 layouts
remote: drwxr-xr-x 24 u30655 u30655 4.0K Dec 19 16:20 libraries
remote: drwxr-xr-x 34 u30655 u30655 4.0K Dec 19 16:20 media
remote: drwxr-xr-x 54 u30655 u30655 4.0K Dec 19 16:20 modules
remote: drwxr-xr-x 26 u30655 u30655 4.0K Dec 19 16:20 plugins
remote: -rw-r--r--  1 u30655 u30655  842 Dec 19 16:20 robots.txt
remote: -rw-r--r--  1 u30655 u30655  108 Dec 19 16:20 tata
remote: drwxr-xr-x  8 u30655 u30655 4.0K Dec 19 16:20 templates
remote: -rw-r--r--  1 u30655 u30655 1.4K Dec 19 16:20 toto

After user switch :

whoami : 
remote: u30655

id :
remote: uid=30655(u30655) gid=30655(u30655) groups=30655(u30655)

ls -hal /

remote: total 84K
remote: drwxr-xr-x  51 root   root   4.0K Dec 19 16:20 .
remote: drwxr-xr-x  51 root   root   4.0K Dec 19 16:20 ..
remote: -rwxr-xr-x   1 root   root      0 Dec 19 16:20 .dockerenv
remote: -rwxr-xr-x   1 root   root      0 Dec 19 16:20 .dockerinit
remote: drwxrwxrwx  32 u30655 u30655 4.0K Dec 19 16:20 app
remote: drwxr-xr-x   2 root   root   4.0K Dec 17 17:40 bin
remote: drwxr-xr-x   2 root   root   4.0K Apr 10  2014 boot
remote: drwxr-xr-x   3 root   root   4.0K Dec 19 16:19 build
remote: drwxr-xr-x   3 u30655 u30655 4.0K Dec 19 13:55 cache
remote: drwxr-xr-x   5 root   u30655  360 Dec 19 16:20 dev
remote: drwxr-xr-x  88 root   root   4.0K Dec 19 16:20 etc
remote: drwxr-xr-x   2 root   root   4.0K Apr 10  2014 home
remote: drwxr-xr-x  13 root   root   4.0K Dec 17 17:40 lib
remote: drwxr-xr-x   2 root   root   4.0K Dec 17 17:39 lib64
remote: drwxr-xr-x   2 root   root   4.0K Nov 25 18:25 media
remote: drwxr-xr-x   2 root   root   4.0K Apr 10  2014 mnt
remote: drwxr-xr-x   2 root   root   4.0K Nov 25 18:25 opt
remote: dr-xr-xr-x 214 root   root      0 Dec 19 16:20 proc
remote: drwx------   2 root   root   4.0K Nov 25 18:25 root
remote: drwxr-xr-x   7 root   root   4.0K Dec 17 17:40 run
remote: drwxr-xr-x   2 root   root   4.0K Nov 25 18:29 sbin
remote: drwxr-xr-x   2 root   root   4.0K Nov 25 18:25 srv
remote: dr-xr-xr-x  13 root   root      0 Dec 19 15:24 sys
remote: drwxrwxrwt   6 u30655 u30655 4.0K Dec 19 16:20 tmp
remote: drwxr-xr-x  16 root   root   4.0K Nov 25 18:25 usr
remote: drwxr-xr-x  19 root   root   4.0K Dec 19 16:20 var

ls -hal /cache (owned by the generated user)

remote: total 12K
remote: drwxr-xr-x  3 u30655 u30655 4.0K Dec 19 13:55 .
remote: drwxr-xr-x 51 root   root   4.0K Dec 19 16:20 ..
remote: drwxr-xr-x  3 u30655 u30655 4.0K Dec 19 14:01 php

ls -hal /tmp (owned by the generated user too)

remote: total 20K
remote: drwxrwxrwt  6 u30655 u30655 4.0K Dec 19 16:20 .
remote: drwxr-xr-x 51 root   root   4.0K Dec 19 16:20 ..
remote: drwxr-xr-x  3 u30655 u30655 4.0K Dec 19 16:20 build
remote: drwxr-xr-x 26 u30655 u30655 4.0K Dec 19 16:20 buildpacks
remote: drwxr-xr-x  2 u30655 u30655 4.0K Dec 19 16:20 env

ls -hal /app

remote: ls: cannot open directory /app: Permission denied

What the hell is going on?
Thanks for your help!

Deploying simple php app with dokku errors due to missing /root/bin

Errors with:

remote: /build/buildpacks/heroku-buildpack-php/bin/compile: line 244: /root/bin/jq: No such file or directory

Because /root/bin does not exist. Ref code here: https://github.com/CHH/heroku-buildpack-php/blob/bb1667220c43f16321e95cec4c61f8152b633c73/bin/compile#L244

curl "http://${S3_BUCKET}.s3.amazonaws.com/jq/jq" -L -s -o - > "$HOME/bin/jq"

security?

Hello,

If I'm not mistaken inside a user container you are the "developer" user but you can do "sudo".
Isn't there a risk that a development environment could be crafted so that the container could attack the host somewhat?

I see that other Docker stack advise to use a USER statement in the Dockerfile to avoid that
deis/deis#26

Sorry, I'm not a Docker specialist, I could be wrong, but what do you think?

Nginx and php buildpack

This one looks good is relatively up to date and active. Anyone smarter than myself care to set it up as a buildpack for dokku https://github.com/iphoting/heroku-buildpack-php-tyler

Make a new release

This would help close ~11 issues in progrium/dokku.

Pass env variables to the compile step

I have a problem with the Clojure buildpack due to buildstep not passing it the path to a dir with env variables. I suppose it is also a problem for other buildpacks. The problem is that the buildpack makes env vars set via config:set visible only at runtime, not at compile time - unless they are whitelisted for being used also at compile time by enumerating them in the env var BUILD_CONFIG_WHITELIST. However setting this has no effect unless compile is invoked with a third param, namely a directory with the env variables - which is not done.

I suppose that the ENV_DIR should be passed on to the buildpack (unless this is some dokku - heroku mismatch).

The relevant code is:
https://github.com/progrium/buildstep/blob/master/stack/builder#L54:

$selected_buildpack/bin/compile "$build_root" "$cache_root"

and https://github.com/heroku/heroku-buildpack-clojure/blob/master/bin/compile:

10 export ENV_DIR=$3
...
115 # extract environment
116 if [ -d "$ENV_DIR" ]; then
117     # if BUILD_CONFIG_WHITELIST is set, read it to know which configs to export
118     if [ -r $ENV_DIR/BUILD_CONFIG_WHITELIST ]; then
119         for e in $(cat $ENV_DIR/BUILD_CONFIG_WHITELIST); do
120             export "$e=$(cat $ENV_DIR/$e)"
121         done

Or am I missing something? Thank you!

Ruby 1.8 app won't start on image generated with buildstep

My app's Gemfile specifies ruby 1.8.7 .

ruby "1.8.7"

Generating a docker image appears to work just fine when I run

$ cat myapp.tar | ./buildstep myapp

But when I try to run

$ docker run -d myapp /bin/bash -c "/start web"

I get

Your Ruby version is 1.9.3, but your Gemfile specified 1.8.7

The problem, I believe boils down to the fact that /usr/local/bin/bundle begins with #!/usr/bin/ruby1.9.1 in the myapp image.

If I run the following commands on the myapp image, and then commit the results to a new image named myapp_ruby1.8, then I'm able to run it.

rm /usr/bin/ruby
ln -s /usr/bin/ruby1.8 /usr/bin/ruby
rm /usr/bin/irb
ln -s /usr/bin/irb1.8 /usr/bin/irb
rm /usr/bin/gem
ln -s /usr/bin/gem1.8 /usr/bin/gem
source /app/.profile.d/ruby.sh
gem install bundler

However, I'm not sure where in the buildstep toolchain this should happen. Plus those commands seem like a workaround as opposed to a real fix. I suspect it might make more sense for the progrium/buildstep image to use RVM instead of the "alternatives" mechanism, and then set the default ruby version in stack/builder ?

Thanks!

Unable to build buildstep because wrong location of Ubuntu release file

The buildstep creation tries to pull down the following link: http://archive.ubuntu.com/ubuntu/dists/quantal/InRelease but it looks like the release information is in http://archive.ubuntu.com/ubuntu/dists/quantal/Release, i.e Release instead of InRelease.

A side note, it seems like this failing to build broke future pushes to Dokku. Is there a way to safeguard against this? Thanks!

Here's my failed build:

sudo make build
docker build -t progrium/buildstep .
Uploading context 204800 bytes
Step 1 : FROM ubuntu:quantal
Pulling repository ubuntu
Pulling image b750fe79269d2ec9a3c593ef05b4332b1d1a02a62b4accb2c21d589ff2f5f2dc (quantal) from ubuntu
 ---> b750fe79269d
Step 2 : MAINTAINER progrium "[email protected]"
 ---> Using cache
 ---> f483a3cc4777
Step 3 : RUN mkdir /build
 ---> Using cache
 ---> 930b06fd8216
Step 4 : ADD ./stack/ /build
 ---> 18de966245bd
Step 5 : RUN LC_ALL=C DEBIAN_FRONTEND=noninteractive /build/prepare
 ---> Running in 1df7e07f4720
Err http://archive.ubuntu.com quantal InRelease

Err http://archive.ubuntu.com quantal Release.gpg
  Temporary failure resolving 'archive.ubuntu.com'
Reading package lists...
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/quantal/InRelease

W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/quantal/Release.gpg  Temporary failure resolving 'archive.ubuntu.com'

W: Some index files failed to download. They have been ignored, or old ones used instead.
Reading package lists...
Building dependency tree...
Package openjdk-6-jre-headless is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
However the following packages replace it:
  icedtea-netx-common icedtea-netx default-jre-headless

Package mercurial is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

Package openjdk-6-jdk is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Unable to locate package daemontools
E: Unable to locate package libssl0.9.8
E: Couldn't find any package by regex 'libssl0.9.8'
E: Package 'mercurial' has no installation candidate
E: Unable to locate package nodejs
E: Package 'openjdk-6-jdk' has no installation candidate
E: Package 'openjdk-6-jre-headless' has no installation candidate
E: Unable to locate package socat
xargs: git: No such file or directory
update-alternatives: error: no alternatives for ruby
update-alternatives: error: no alternatives for gem
/build/prepare: line 30: gem: command not found
/build/prepare: line 31: cd: /build/buildpacks/heroku-buildpack-ruby: No such file or directory
Error build: The command [/bin/sh -c LC_ALL=C DEBIAN_FRONTEND=noninteractive /build/prepare] returned a non-zero code: 1
The command [/bin/sh -c LC_ALL=C DEBIAN_FRONTEND=noninteractive /build/prepare] returned a non-zero code: 1
deploy@devcloud:~/buildstep$ ping archive.ubunut.com
ping: unknown host archive.ubunut.com
deploy@devcloud:~/buildstep$ ping archive.ubuntu.com
PING archive.ubuntu.com (91.189.91.15) 56(84) bytes of data.
64 bytes from likho.canonical.com (91.189.91.15): icmp_req=1 ttl=52 time=19.2 ms
64 bytes from likho.canonical.com (91.189.91.15): icmp_req=2 ttl=52 time=19.2 ms
^C
--- archive.ubuntu.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 19.240/19.248/19.256/0.008 ms
deploy@devcloud:~/buildstep$ sudo make build
docker build -t progrium/buildstep .
Uploading context 204800 bytes
Step 1 : FROM ubuntu:quantal
 ---> b750fe79269d
Step 2 : MAINTAINER progrium "[email protected]"
 ---> Using cache
 ---> f483a3cc4777
Step 3 : RUN mkdir /build
 ---> Using cache
 ---> 930b06fd8216
Step 4 : ADD ./stack/ /build
 ---> 8d4074297c09
Step 5 : RUN LC_ALL=C DEBIAN_FRONTEND=noninteractive /build/prepare
 ---> Running in 46d34e0e60f5
Err http://archive.ubuntu.com quantal InRelease

Err http://archive.ubuntu.com quantal Release.gpg
  Temporary failure resolving 'archive.ubuntu.com'
Reading package lists...
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/quantal/InRelease

W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/quantal/Release.gpg  Temporary failure resolving 'archive.ubuntu.com'

W: Some index files failed to download. They have been ignored, or old ones used instead.
Reading package lists...
Building dependency tree...
Package openjdk-6-jre-headless is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
However the following packages replace it:
  icedtea-netx-common icedtea-netx default-jre-headless

Package mercurial is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

Package openjdk-6-jdk is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Unable to locate package daemontools
E: Unable to locate package libssl0.9.8
E: Couldn't find any package by regex 'libssl0.9.8'
E: Package 'mercurial' has no installation candidate
E: Unable to locate package nodejs
E: Package 'openjdk-6-jdk' has no installation candidate
E: Package 'openjdk-6-jre-headless' has no installation candidate
E: Unable to locate package socat
xargs: git: No such file or directory
update-alternatives: error: no alternatives for ruby
update-alternatives: error: no alternatives for gem
/build/prepare: line 30: gem: command not found
/build/prepare: line 31: cd: /build/buildpacks/heroku-buildpack-ruby: No such file or directory
Error build: The command [/bin/sh -c LC_ALL=C DEBIAN_FRONTEND=noninteractive /build/prepare] returned a non-zero code: 1
The command [/bin/sh -c LC_ALL=C DEBIAN_FRONTEND=noninteractive /build/prepare] returned a non-zero code: 1

PHP Buildpack unable to find PHP extensions specified in composer.json

Not sure if this is the right place for this issue. I'm fairly sure its a problem with the Heroku PHP Buildpack, though I thought it more appropriate to post the issue here as I don't think the buildpack explicitly supports Dokku/Buildstep.

Since sometime today, our apps are failing to build (across various servers with the same setup). The same setup has been working without issue for several weeks, and we haven't changed anything in our setup (although I did update buildstep in an attempt to solve this issue).

[gulp] Starting 'deploy'...
-----> Cleaning up ...
remote: Cloning into '/tmp/tmp.3tMA90T5I7'...
-----> Building geka ...
remote: done.
remote: HEAD is now at ee99e08... deploy
-----> Fetching custom buildpack
-----> PHP app detected
-----> No runtime requirements in composer.json, defaulting to PHP 5.6.3.
-----> Installing system packages...
       - PHP 5.6.3
       - Apache 2.4.10
       - Nginx 1.6.0
-----> Installing PHP extensions...
       - zend-opcache (automatic; bundled, using 'ext-zend-opcache.ini')
-----> Installing dependencies...
       Composer version 1.0-dev (b23a3cd36870ff0eefc161a4638d9fcf49d998ba) 2014-11-21 17:59:11
       Loading composer repositories with package information
       Installing dependencies
       Your requirements could not be resolved to an installable set of packages.

       Problem 1
       - The requested PHP extension ext-mbstring * is missing from your system.
       Problem 2
       - The requested PHP extension ext-imagick * is missing from your system.

My composer.json looks like:

{
  "require": {
    "ext-mbstring": "*",
    "ext-imagick": "*"
  }
}

I suspected it might be a problem with a new version of PHP, although specifying a PHP version in composer.json seems to be ignored.

Make default packages configurable

build step makes heavy assumptions of default packages based on Heroku's. While this is good for compatibility it would be great to have the ability to say "none" or use another default set.

A couple of reasons:

1. Allows easier integration of binaries that must be compiled. Heroku actually falls on it's knees if you need native binaries.

2. More control over the install set. GraphicsMagick has an ImageMagick compat package which can swap out ImageMagick but couldn't if it's already been installed on $PATH.

Remove hack in stack/prepare for bpwatch

heroku/heroku-buildpack-python#101 has been closed, so we should remove the hack at https://github.com/progrium/buildstep/blob/master/stack/prepare#L25

Changelog here:
heroku/heroku-buildpack-python@99b17fa

Migrate to Ubuntu 14.04 Trusty

In the light of http://fridge.ubuntu.com/2014/05/01/ubuntu-12-10-quantal-quetzal-reaches-end-of-life-on-may-16-2014/ and issues like #85, it would be suitable to base buildstep on a LTS release like 14.04

This is being done in forks already (https://github.com/cleawing/buildstep/commit/ee821c6e99984e4e82a894a59a33a7147a053e39, alecgorge@cf2857e, bellycard@50d6210, etc) but having an official 14.04 branch for buildstep would be great!

Instructions for rebuilding from source?

Hi,
I'm sorry, this isn't really an issue so much as a question.
How does one go about rebuilding buildstep on an existing Dokku install?

I attempted the following:

• git clone buildstep
• make build

However this set about creating a new Quantal image from scratch, which didn't seem right as the existing containers were based on Raring.

Having trouble with plain dockerfile

FROM progrium/buildstep
ADD site.tar /app
RUN /build/builder
EXPOSE 80
CMD [ "/start", "web" ]

Running with docker build repeats the following lines below.

gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now

From a little debugging, it seems to happen during the detect part of the ruby buildpack.

Thoughts? Is there anything special I need to be doing with /app?

Thanks.

scala buildpack is broken due to config_vars and quoting

The heroku scala buildpack produces release yaml with vars that contain spaces, which causes the /app/.profile.d/config_vars file to contain this:

PATH=.jdk/bin:.sbt_home/bin:/usr/local/bin:/usr/bin:/bin
JAVA_OPTS=-Xmx384m -Xss512k -XX:+UseCompressedOops
SBT_OPTS=-Xmx384m -Xss512k -XX:+UseCompressedOops
REPO=/app/.sbt_home/.ivy2/cache

which causes this to appear in the logs:

/app/.profile.d/config_vars: line 2: -Xss512k: command not found
/app/.profile.d/config_vars: line 3: -Xss512k: command not found

Fix would appear to be to add quoting to L68 of /stack/builder

Update Ruby Buildpack

The ruby buildpack (don't know if it is an issue in other buildpacks as I only use the ruby one) needs to be updated as heroku/heroku-buildpack-ruby#335 resolved time outs with the nodejs.org servers during deploy.

Without this update deployment for me is impossible.

docker port exposing

Hi, how do i know in which port the app is running? I mean, i want to bind the app to a port (docker -p ...) but i don't know in which port the app is running inside the container.

Any help is appreciated, thanks, Agustin

making /exec and /start the same script breaks things like shoreman

I am ran into a problem with dokku run. Basically it was just running my procfile instead of running one off commands.

It appears that recently the builder script started linking the /start and /exec scripts.

https://github.com/progrium/buildstep/blob/master/builder/builder#L41

Where as before they were different scripts.

The problem with this is that all the process managers like shoreman et all over-write the start script to manage the Procfille process.

With this new setup that means they are also destroying the ability to run one off commands.

Alternate fork to use?

This don't look to be maintained by @progrium anymore. Which fork is recommended for use?

cc/ @yabawock @arthurnn @stuartpb

Hit by Apparmor bug when building the container

Host OS: Ubuntu 13.10 x64

When trying to build the buildstep container, it crashed during the postgres installation with this error:

Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by
postgresql-common'
dpkg: unrecoverable fatal error, aborting:
   failed to fstat previous diversions file: No such file or directory
  E: Sub-process /usr/bin/dpkg returned an error code (2)

After some investigation, I came over this:
https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/925028

The workaround posted there worked for me:

sudo ln -s /etc/apparmor.d/usr.bin.lxc-start /etc/apparmor.d/disable/
sudo apparmor_parser -R /etc/apparmor.d/usr.bin.lxc-start

This issue can be closed, hopefully someone with the same problem will find it.

Non root user "breaks" dokku-volume

Wasn't sure where this should go, so I'm starting here.

When the non-root user feature came to be, /exec was modified to chown -R <user>:<group> /app but we do not have a way to extend this behavior to other mounted volumes in dokku (and I assume any other extension of buildstep)...

example:

1. install dokku
2. install https://github.com/ohardy/dokku-volume

root@dokku:~/dokku# dokku volume:add app /myappcache
root@dokku:~/dokku# dokku run app bash
u22374@0e95e1326e0c:~$ ls -lad /myappcache
drwxr-xr-x 2 1002 1002 4096 Feb 10 16:50 /myappcache
u22374@0e95e1326e0c:~$ touch /myappcache/foo
touch: cannot touch '/myappcache/foo': Permission denied

Should this be solved once in buildstep vs. several times throughout the plugin community or is this out of scope?

Currently dokku-logging-supervisord implemented a solution like this.

# Change ownership of /var/log/app to non-root container user
APP_USER=$(stat -c %U /app)
APP_GROUP=$(stat -c %G /app)
if [[ -n "$APP_USER" ]] && [[ -n "$APP_GROUP" ]]; then
  chown -R "${APP_USER}:${APP_GROUP}" /var/log/app
  chmod -R g+rw /var/log/app
fi

Just to throw an initial spitball out, what if plugins set/update a config variable that is a list of container paths and /exec chowns those as well?

Can't find python runtime?

-----> Building gpsheatmap ...
Python app detected
-----> No runtime.txt provided; assuming python-2.7.4.
-----> Preparing Python runtime (python-2.7.4)
! Requested runtime (python-2.7.4) was not found.

I'm trying to deploy a simple flask app to Dokku. I see that python-2.7.4 is included in packages.txt for buildstep, but I'm not sure why this buildpack is not finding it.

The buildpack downloads the python version from s3:
curl "http://envy-versions.s3.amazonaws.com/$PYTHON_VERSION.tar.bz2" -s | tar jx &> /dev/null
and https://envy-versions.s3.amazonaws.com/python-2.7.4.tar.bz2 does exist.

Any ideas why it fails? permissions issue?

Trying to 'make build' buildstep generates error

Forked, buildstep, added a buildpack, cloned on the server in a 'buildstep' folder before running 'make build' Gets a long way through before throwing the following error.

Any ideas?

-----> Committing changes...
2013/08/09 10:03:01 Error: stat /var/lib/docker/containers/c298bdf004cdcc3d34cee932dd051e98c6cf9ce223eb78a81e3d436ee2061d1e/rootfs: resource temporarily unavailable
make: *** [build] Error 1

Cannot pull down buildpack for Ruby 2.0 - results in error

-----> Building troux ...
remote: HEAD is now at 3593be8... Added foreman and updated profile
       Ruby app detected
-----> Compiling Ruby/Rails
-----> Using Ruby version: ruby-2.0.0
-----> Installing dependencies using 1.5.2
       New app detected loading default bundler cache
 !
 !     Command: 'set -o pipefail; curl --fail --retry 3 --retry-delay 1 --connect-timeout 3 --max-time 30 https://s3-external-1.amazonaws.com/heroku-buildpack-ruby/ruby-2.0.0-p481-default-cache.tgz -s -o - | tar zxf -' failed unexpectedly:
 !     
 !     gzip: stdin: unexpected end of file
 !     tar: Child returned status 1
 !     tar: Error is not recoverable: exiting now

Running the curl command separately:

curl --fail --retry 3 --retry-delay 1 --connect-timeout 3 --max-time 30 https://s3-external-1.amazonaws.com/heroku-buildpack-ruby/ruby-2.0.0-p481-default-cache.tgz
curl: (22) The requested URL returned error: 403 Forbidden

jruby rails app bundling error "exec: java: not found"

I run into this error when pushing a JRuby Rails app up to dokku. Since my buildstep isn't updated I'm not sure if Heroku has changed their jdk build.

After rebuilding buildstep to include latest heroku-buildpacks I'm still getting the same error.

Gem::Installer::ExtensionBuildError: ERROR: Failed to build gem native extension.
/tmp/build/vendor/ruby-1.9.3-jruby-1.7.16/bin/jruby extconf.rb
/tmp/build/vendor/ruby-1.9.3-jruby-1.7.16/bin/jruby: line 388: exec: java: not found

Gem files will remain installed in /tmp/build/vendor/bundle/jruby/1.9/gems/ffi-1.9.6 for inspection.
Results logged to /tmp/build/vendor/bundle/jruby/1.9/gems/ffi-1.9.6/ext/ffi_c/gem_make.out

Whereas heroku has a warning but builds successfully:

/tmp/build_e79a99a7-9ee7-408e-8d66-dc0ecf2c812a/vendor/ruby-1.9.3-jruby-1.7.16/lib/ruby/shared/rubygems/ext/ext_conf_builder.rb:43 warning: Tempfile#unlink or delete called on open file; ignoring

Thoughts?

"tar: This does not look like a tar archive" when piping from git archive

From a Git repository where I have an app:

[stuart@stushiba laterti.me]$ git archive HEAD | sudo docker run -a stdin progrium/buildstep /bin/bash -c "mkdir -p /app && tar -xC /app"
34e3cf0bf9cbe2f0751f1c12ef473d01d41b6d54a6ca19f35b0dd8e5509ac21f
[stuart@stushiba laterti.me]$ sudo docker logs 34e3c
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors

I have no idea what's going wrong. git archive HEAD | /bin/bash -c "mkdir -p /tmp/app && tar -xC /tmp/app" on the host system works just fine, so it doesn't appear to be an issue with git archive.

make build hangs

make build hangs on "Installing build directory..."

ubuntu@x1-build0-12225:~/buildstep$ make build
./build.sh ./stack progrium/buildstep
=====> Building container...
-----> Pulling base image...
       Pulling repository ubuntu from https://index.docker.io/v1
       Pulling image b750fe79269d2ec9a3c593ef05b4332b1d1a02a62b4accb2c21d589ff2f5f2dc (quantal) from ubuntu
-----> Installing build directory...

Looks like something with hanging on reading from the pipe, because if, from a second console, I do docker attach to the running process, then Ctrl-C the docker attach, make build finishes, but there is nothing in /build inside the container, so the prepare command fails.

Relevant details about the machine:

ubuntu@x1-build0-12225:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 12.04.2 LTS
Release:        12.04
Codename:       precise
ubuntu@x1-build0-12225:~$ uname -a
Linux x1-build0-12225 3.8.0-25-generic #37~precise1-Ubuntu SMP Fri Jun 7 16:27:35 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@x1-build0-12225:~$ docker version
Client version: 0.4.6
Server version: 0.4.6
Go version: go1.1

Update rev for buildpack-multi

Hello, could you update the rev for buildpack-multi (ddollar) to at least 6e79094381a1773b9ae63c7b8002f38ad78d1cf0? This rev of buildpack-multi allows for one buildpack to pass environment along to the next, which is useful for some composed environments.

Add ffmpeg to packages.txt

Can't run multiple commands in Procfile

I am having trouble with the way I write Procfiles and the way they are evaluated in bash.

In commit 16e3f3c I started having issues with running
setuidgid u32178 $(eval echo "${command}") vs simply running eval ${command} in the /start script.

My Procfile looks like this.

web: python bin/production.py collectstatic --noinput --verbosity 3 ; gunicorn wsgi

Debugging and tweaking buildpack options

I'm currently attempting to get some alternate PHP buildpacks* to work with Dokku and am running into "Bad Gateway" issues. The basic Heroku PHP buildpack works with my demo .php file, but these other buildpacks don't.

I'd love to be able to tweak the apache conf options and other settings without having to adjust a forked buildpack and then rebuild buildstep.

Is there a way to do what I'm after? Also I apologize if the answer is obvious, my mental model of how buildstep and dokku work is weak at best.

Thanks!

*Alternate Buildpacks:
https://github.com/heroku/heroku-buildpack-php
https://github.com/andrewsg/heroku-buildpack-php-foundry

Quantal no longer in http://archive.ubuntu.com/ubuntu/dists/ affects buildstep, plugins and buildpaks

I recently ran into issues with apt-get update for ubuntu:quantal (which seems to be used by various other dokku plugins for example @Kloadut https://github.com/Kloadut/dokku-md-dockerfiles/blob/master/Dockerfile).

buildstep container completes (as it does not use apt-get update). But for example the dokku-mariadb plugin or https://github.com/rhy-jot/buildpack-nginx.git make use of apt-get update and fail to build as apt-get update returns a non 0 exit code. (quantal is no longer in archives -> 404)

Can someone recommend a fix? just don't use apt-get update ;)? or repackage buildstep/plugins with an ubuntu LTS image?

PHP after switch to Trusty

Hi, I'm trying to deploy PHP application, but I'm getting this error after switching to 14.04:

vendor/php/bin/php: error while loading shared libraries: libreadline.so.5: cannot open shared object file: No such file or directory

Failed to push refs

Can't push to dokku with latest commit 2f56cec

https://gist.github.com/JohnyMoSwag/267a57337c05f644c45d

update-alternatives fails in prepare script

Are these lines needed anymore? https://github.com/expa/buildstep/blob/master/stack/prepare#L25-L26

update-alternatives --set ruby /usr/bin/ruby1.9.1
update-alternatives --set gem /usr/bin/gem1.9.1

Currently they fail since upgrading to 14.04 and updating packages.txt

update-alternatives: error: no alternatives for ruby
update-alternatives: error: no alternatives for gem

Why is the postgresql server package added?

Why not just e.g. libpq-dev? Shouldn't really be needed to run the full server in each container afaik.

Provide extendable docker image

I would love if it would be possible to turn this project into one docker image that could be used as a base image. It would be great to be able to do something like this:

$ cat Dockerfile
FROM progrium/buildstep
CMD /start web
$ docker build -t myapp .
...output...
$ docker run myapp
...running...

This would be great for me, because it would allow me to use my app just like any other Docker app, without having to worry about installing anything on my computer and compiling tarballs.

It looks like tutumcloud/buildstep does basically this, but I am wondering if there is a reason not to include htis behavior in this repo.

`make build` fails because rubygems are not installed

After pulling new buildstep, make build fails with this output in the end:

Selecting previously unselected package fuse.
Preparing to unpack .../fuse_2.9.2-4ubuntu4_amd64.deb ...
Unpacking fuse (2.9.2-4ubuntu4) ...
Processing triggers for ureadahead (0.100.0-16) ...
Setting up libfuse2:amd64 (2.9.2-4ubuntu4) ...
Setting up fuse (2.9.2-4ubuntu4) ...
Creating fuse group...
Adding group `fuse' (GID 105) ...
Done.
Creating fuse device...

makedev fuse c 10 229 root root 0660: failed

dpkg: error processing package fuse (--configure):
 subprocess installed post-installation script returned error exit status 1
Processing triggers for libc-bin (2.19-0ubuntu6) ...
Errors were encountered while processing:
 fuse




/build/prepare: line 27: gem: command not found
2014/07/24 06:58:39 The command [/bin/sh -c LC_ALL=C DEBIAN_FRONTEND=noninteractive /build/prepare] returned a non-zero code: 127
make: *** [build] Error 1

separate out buildpack environments

I might not understand what is going on with buildstep correctly, but it seems that a base image is created containing all the buildpacks, and the final application image will contain all the buildpacks. This seems unecessary if I only need one of the buildpacks. It is easy enough to modify buildpacks.txt, but there is the problem of packages.txt (which to me seems like it should be roled back into the individual build packs).

In my case I actually want to build a binary, so the ideal situation would be to send the tarball of source into the build container and then get back out a new container for deployment that has the binary without installation cruft.