Coder Social home page Coder Social logo

omnibus-flapjack's Introduction

Flapjack Flapjack

Build Status

Flapjack is a flexible monitoring notification routing system that handles:

  • Alert routing (determining who should receive alerts based on interest, time of day, scheduled maintenance, etc)
  • Alert summarisation (with per-user, per media summary thresholds)
  • Your standard operational tasks (setting scheduled maintenance, acknowledgements, etc)

Flapjack will be immediately useful to you if:

  • You want to identify failures faster by rolling up your alerts across multiple monitoring systems.
  • You monitor infrastructures that have multiple teams responsible for keeping them up.
  • Your monitoring infrastructure is multitenant, and each customer has a bespoke alerting strategy.
  • You want to dip your toe in the water and try alternative check execution engines like Sensu, Icinga, or cron in parallel to Nagios.

Try it out with the Quickstart Guide

The Quickstart guide will help you get Flapjack up and running in a VM locally using Vagrant and VirtualBox.

The technical low-down

Flapjack provides a scalable method for dealing with events representing changes in system state (OK -> WARNING -> CRITICAL transitions) and alerting appropriate people as necessary.

At its core, Flapjack processes events received from external check execution engines, such as Nagios. Nagios provides a 'perfdata' event output channel, which writes to a named pipe. flapjack-nagios-receiver then reads from this named pipe, converts each line to JSON and adds them to the events queue.

Flapjack sits downstream of check execution engines (like Nagios, Sensu, Icinga, or cron), processing events to determine:

  • if a problem has been detected
  • who should know about the problem
  • how they should be told

Additional check engines can be supported by adding additional receiver processes similar to the nagios receiver.

Installing

NB: v2 packages will be ready soon -- for the moment these instructions will not work

Ubuntu Precise 64 (12.04):

Tell apt to trust the Flapjack package signing key:

gpg --keyserver keys.gnupg.net --recv-keys 803709B6
gpg -a --export 803709B6 | sudo apt-key add -

Add the Flapjack Debian repository to your Apt sources:

echo "deb http://packages.flapjack.io/deb/v2 precise main" | sudo tee /etc/apt/sources.list.d/flapjack.list

Install the latest Flapjack package:

sudo apt-get update
sudo apt-get install flapjack

Alternatively, download the deb and install using sudo dpkg -i <filename>

The Flapjack package is an Omnibus package and as such contains most dependencies under /opt/flapjack, including Redis.

Installing the package will start Redis (non standard port) and Flapjack. You should now be able to access the Flapjack Web UI at:

http://localhost:3080/

And consume the REST API at:

http://localhost:3081/

N.B. The Redis installed by Flapjack runs on a non-standard port (6380), so it doesn't conflict with other Redis instances you may already have installed.

Other OSes:

Currently we only make a package for Ubuntu Precise (amd64). If you feel comfortable getting a ruby environment going on your preferred OS, then you can also just install Flapjack from rubygems.org:

gem install flapjack

Using a tool like rbenv or rvm is recommended to keep your Ruby applications from intefering with one another.

Alternatively, you can add support for your OS of choice to omnibus-flapjack and build a native package. Pull requests welcome, and we'll help you make this happen!

You'll also need Redis >= 2.6.12.

Configuring

Have a look at the default config file and modify things as required. The package installer copies this to /etc/flapjack/flapjack_config.toml if it doesn't already exist.

# edit the config
sudo vi /etc/flapjack/flapjack_config.toml

# reload the config
sudo /etc/init.d/flapjack reload

Running

Ubuntu Precise 64:

After installing the Flapjack package, Redis and Flapjack should be automatically started.

First up, start Redis if it's not already started:

# status:
sudo /etc/init.d/redis-flapjack status

# start:
sudo /etc/init.d/redis-flapjack start

Operating Flapjack:

# status:
sudo /etc/init.d/flapjack status

# reload:
sudo /etc/init.d/flapjack reload

# restart:
sudo /etc/init.d/flapjack restart

# stop:
sudo /etc/init.d/flapjack stop

# start:
sudo /etc/init.d/flapjack start

Usage

Please see the documentation.

Developing Flapjack

Information on developing more Flapjack components or contributing to core Flapjack development can be found in the Developing section of the docs.

Note that the master branch corresponds to Flapjack 2; maintenance builds for Flapjack 1 are built from the maint/1.x branch.

Documentation Submodule

We have the documentation for this project on a github wiki and also referenced as a submodule at /doc in this project. Run the following commands to populate the local doc/ directory:

git submodule init
git submodule update

If you make changes to the documentation locally, here's how to publish them:

  • Checkout master within the doc subdir, otherwise you'll be commiting to no branch, a.k.a. no man's land.
  • git add, commit and push from inside the doc subdir
  • Add, commit and push the doc dir from the root (this updates the pointer in the main git repo to the correct ref in the doc repo, we think...)

RTFM

All of the documentation.

omnibus-flapjack's People

Contributors

ali-graham avatar auxesis avatar bernd avatar bryanyork avatar glensc avatar jessereynolds avatar jsoriano avatar krisbuytaert avatar michaelneale avatar orthographic-pedant avatar sarahriley avatar someword avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

omnibus-flapjack's Issues

CentOS package installation takes a long time to complete

This was maybe 5 minutes or more? not quite sure.

This is actually on RHEL 7 on ec2:

# yum install --nogpgcheck flapjack
Loaded plugins: amazon-id, rhui-lb
Resolving Dependencies
--> Running transaction check
---> Package flapjack.x86_64 0:1.2.0_0.20141107131800-1.el6 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

===========================================================================================================================================
 Package                    Arch                     Version                                           Repository                     Size
===========================================================================================================================================
Installing:
 flapjack                   x86_64                   1.2.0_0.20141107131800-1.el6                      flapjack-v1                    62 M

Transaction Summary
===========================================================================================================================================
Install  1 Package

Total size: 62 M
Installed size: 190 M
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : flapjack-1.2.0_0.20141107131800-1.el6.x86_64                                                                            1/1

Creating /etc/flapjack/flapjack_config.yaml
Creating /usr/local/lib/flapjackfeeder.o
Creating /etc/init.d/redis-flapjack
Creating /etc/init.d/flapjack
Creating /etc/init.d/flapjack-nagios-receiver
Creating /etc/init.d/flapper
Setting paths
Starting redis-flapjack
Starting redis-flapjack (via systemctl):  Job for redis-flapjack.service failed. See 'systemctl status redis-flapjack.service' and 'journalctl -xn' for details.
[FAILED]
warning: %post(flapjack-1.2.0_0.20141107131800-1.el6.x86_64) scriptlet failed, exit status 1
Non-fatal POSTIN scriptlet failure in rpm package flapjack-1.2.0_0.20141107131800-1.el6.x86_64
  Verifying  : flapjack-1.2.0_0.20141107131800-1.el6.x86_64                                                                            1/1

Installed:
  flapjack.x86_64 0:1.2.0_0.20141107131800-1.el6

Complete!

build completes without building flapjack

omnibus 3.2.1 just did this to me ... that is, it build the nokogiri dependency and cached that build, then just skipped over the last dependency of the flapjack project, that being the flapjack software itself. so it happily produced a .deb without flapjack, only the dependencies. Here's the last bit of the output:

time docker run --rm -i -t -e "FLAPJACK_BUILD_REF=729e3d4"   -e "FLAPJACK_PACKAGE_VERSION=1.0.0~rc3~20140728T054700-729e3d4-1"   flapjack/omnibus-ubuntu bash -c   "cd omnibus-flapjack ; \
  git pull ; \
  bundle install --binstubs ; \
  bin/omnibus build --log-level=info flapjack ; \
  bash"

...snip...

                [Software: bundler] I | Dirtied the cache!
                [Builder: nokogiri] I | Starting build
                                    I | Environment:
                                    I |   PKG_CONFIG_PATH="/opt/flapjack/embedded/lib/pkgconfig"
                                    I |   NOKOGIRI_USE_SYSTEM_LIBRARIES="true"
                                    I | $ /opt/flapjack/embedded/bin/gem install nokogiri -v 1.6.2.1 -- --use-system-libraries --with-xml2-lib=/opt/flapjack/embedded/lib --with-xml2-include=/opt/flapjack/embedded/include/libxml2 --with-xslt-lib=/opt/flapjack/embedded/lib --with-xslt-include=/opt/flapjack/embedded/include/libxslt --with-iconv-dir=/opt/flapjack/embedded --with-zlib-dir=/opt/flapjack/embedded
                [Builder: nokogiri] I | gem `install nokogiri -v 1.6.2.1 -- --use-system-libraries --with-xml2-lib=/opt/flapjack/embedded/lib --with-xml2-include=/opt/flapjack/embedded/include/libxml2 --with-xslt-lib=/opt/flapjack/embedded/lib --with-xslt-include=/opt/flapjack/embedded/include/libxslt --with-iconv-dir=/opt/flapjack/embedded --with-zlib-dir=/opt/flapjack/embedded': 44.3266s
                [Builder: nokogiri] I | Build nokogiri: 44.3278s
                [Builder: nokogiri] I | Finished build
               [Software: nokogiri] I | Caching build
               [GitCache: nokogiri] I | Performing incremental cache
               [GitCache: nokogiri] I | Cache path `/var/cache/omnibus/cache/git_cache/opt/flapjack' exists, skipping creation
               [GitCache: nokogiri] I | Removing git directories
                                    I | $ git --git-dir=/var/cache/omnibus/cache/git_cache/opt/flapjack --work-tree=/opt/flapjack add -A -f
               [GitCache: nokogiri] I | Calculating tag
                                    I | $ git --git-dir=/var/cache/omnibus/cache/git_cache/opt/flapjack --work-tree=/opt/flapjack commit -q -m "Backup of nokogiri-f01352b6a6fd7e2cceba6031512e13365b1076025a6e65eee663dcaa5f3426df"
                                    I | $ git --git-dir=/var/cache/omnibus/cache/git_cache/opt/flapjack --work-tree=/opt/flapjack tag -f "nokogiri-f01352b6a6fd7e2cceba6031512e13365b1076025a6e65eee663dcaa5f3426df"
               [Software: nokogiri] I | Dirtied the cache!
                                    I | $ find /opt/flapjack/ -type f | xargs ldd
                                    I | $ fpm -s dir -t deb -v 1.0.0~rc3~20140728T054700-729e3d4-1 -n flapjack -p flapjack_1.0.0~rc3~20140728T054700-729e3d4-1-1_amd64.deb --iteration 1 -m 'Lindsay Holmwood, Jesse Reynolds, Ali Graham' --description 'The full stack of flapjack' --url http://flapjack.io --after-install '/omnibus-flapjack/package-scripts/flapjack/postinst' --before-remove '/omnibus-flapjack/package-scripts/flapjack/prerm' --after-remove '/omnibus-flapjack/package-scripts/flapjack/postrm' --exclude '.git*' --exclude 'bundler/git' /opt/flapjack

RPM package has a bit of debianism :(

The post installscripts expect a debian host rather than a CentOS or RHEL based platform.

It assumes wrong parameters for adduser
and the existence of update-rc.d

rpm -vih flapjack-0.8.2+20140116214937-1.el6.x86_64.rpm
Preparing... ########################################### [100%]
1:flapjack ########################################### [100%]
adduser: unrecognized option '--quiet'
Usage: useradd [options] LOGIN

Options:
-b, --base-dir BASE_DIR base directory for the home directory of the
new account
-c, --comment COMMENT GECOS field of the new account
-d, --home-dir HOME_DIR home directory of the new account
-D, --defaults print or change default useradd configuration
-e, --expiredate EXPIRE_DATE expiration date of the new account
-f, --inactive INACTIVE password inactivity period of the new account
-g, --gid GROUP name or ID of the primary group of the new
account
-G, --groups GROUPS list of supplementary groups of the new
account
-h, --help display this help message and exit
-k, --skel SKEL_DIR use this alternative skeleton directory
-K, --key KEY=VALUE override /etc/login.defs defaults
-l, --no-log-init do not add the user to the lastlog and
faillog databases
-m, --create-home create the user's home directory
-M, --no-create-home do not create the user's home directory
-N, --no-user-group do not create a group with the same name as
the user
-o, --non-unique allow to create users with duplicate
(non-unique) UID
-p, --password PASSWORD encrypted password of the new account
-r, --system create a system account
-s, --shell SHELL login shell of the new account
-u, --uid UID user ID of the new account
-U, --user-group create a group with the same name as the user
-Z, --selinux-user SEUSER use a specific SEUSER for the SELinux user mapping

Creating /etc/init.d/redis-flapjack
Creating /etc/init.d/flapjack
Creating /etc/init.d/flapjack-nagios-receiver
Creating /etc/init.d/flapper
/var/tmp/rpm-tmp.dYOoWG: line 57: update-rc.d: command not found
warning: %post(flapjack-0.8.2+20140116214937-1.el6.x86_64) scriptlet failed, exit status 127

build_and_publish rake task broken

I broke this last week when refactoring to support the new promote rake task.

Adding pkg/ to the flapjack-v1-trusty-experimental repo

tail end of one of the build logs:

I | $ fpm -s dir -t deb -v 1.2.0~+20141114053030-v1.2.0-trusty -n flapjack -p flapjack_1.2.0~+20141114053030-v1.2.0-trusty-1_amd64.deb --iteration 1 -m 'Lindsay Holmwood, Jesse Reynolds, Ali Graham, Sarah Kowalik' --description 'The full stack of flapjack' --url http://flapjack.io --after-install '/omnibus-flapjack/package-scripts/flapjack/postinst' --before-remove '/omnibus-flapjack/package-scripts/flapjack/prerm' --after-remove '/omnibus-flapjack/package-scripts/flapjack/postrm' --exclude '.git*' --exclude 'bundler/git' --deb-user root --rpm-user root --solaris-user root --deb-group root --rpm-group root --solaris-group root /opt/flapjack
dpkg-deb: warning: 'repackage/DEBIAN/control' contains user-defined field 'License'
dpkg-deb: warning: 'repackage/DEBIAN/control' contains user-defined field 'Vendor'
dpkg-deb: warning: ignoring 2 warnings about the control file(s)

dpkg-deb: building package flapjack' incandidate_flapjack_1.2.0~+20141114053030-v1.2.0-trusty-1_amd64.deb'.
STDERR:
Unable to find image 'flapjack/omnibus-ubuntu:trusty' locally
Pulling repository flapjack/omnibus-ubuntu
Status: Downloaded newer image for flapjack/omnibus-ubuntu:trusty
Docker run completed.
Retrieving package from the container
Purging the container
distro: ubuntu
distro_release: trusty
major_version: v1
package_version: 1.2.0~+20141114053030-v1.2.0-trusty
file_suffix: deb
major_delim: _
minor_delim: -
Creating aptly.conf
Starting package upload
Syncing down s3://packages.flapjack.io/aptly to aptly
Checking aptly db for errors
Creating all components for the distro release if they don't exist
Adding pkg/ to the flapjack-v1-trusty-experimental repo
Attempting the first publish for all components of the major version of the given distro release
Repository already published, attempting an update
Creating directory index files for published packages
Syncing aptly/public up to s3://packages.flapjack.io/deb
Syncing aptly up to s3://packages.flapjack.io/aptly
Copying candidate package for main to s3
Removing package upload lockfile
19.70user 13.40system 16:47.33elapsed 3%CPU (0avgtext+0avgdata 37500maxresident)k
0inputs+1449744outputs (0major+153752minor)pagefaults 0swaps
Finished v1.2.0 on ubuntu trusty at 2014-11-14T05:47:16Z, duration: 16m 47s

Setup init scripts + directories on package install

  • Symlink the package scripts out of the gem to /etc/init.d
  • Investigate whether service management could be handled more simply with Upstart
  • Create a config directory structure under /etc/flapjack
  • Copy the configuration files in place
  • Create a logging directory structure under /var/log/flapjack

This could all likely be handled out of the postinst script.

1.0 deb repo is a little borked

Earlier on there was a bug in the build script which led to a package being added named flapjack_1.0.0~rc3~20140814083906-master-#{DISTRO_RELEASE}-1_amd64.deb. This was causing errors like this during the apt repo updates:

ERROR: unable to publish: unable to process packages: error linking file to /home/ubuntu/aptly/public/1.0/pool/experimental/f/flapjack/flapjack_1.0.0~rc3~20140814083906-master-#{DISTRO_RELEASE}-1_amd64.deb: file already exists and is different

I looked in the S3 bucket and found the earlier contributed package file, above, and tried deleting it from S3. But of course this was dumb as aptly is now upset that the package file doesn't exist. So still unable to add freshly created packages builds to 1.0 precise experimental:

+ eval aptly -config=aptly.conf publish repo '-architectures="i386,amd64"' '-gpg-key="803709B6"' -component=, flapjack-1.0-precise-main flapjack-1.0-precise-experimental 1.0
+ echo 'Repository already published, attempting an update'
Repository already published, attempting an update
+ aptly -config=aptly.conf -gpg-key=803709B6 publish update precise 1.0
Loading packages...
Generating metadata files and linking package files...
ERROR: unable to publish: unable to process packages: stat /home/ubuntu/aptly/pool/4a/8a/flapjack_1.0.0~rc3~20140814083906-master-#{DISTRO_RELEASE}-1_amd64.deb: no such file or directory
FINISHED  AT Thu Aug 14 11:54:31 UTC 2014
Cloud-init v. 0.7.5 finished at Thu, 14 Aug 2014 11:54:31 +0000. Datasource DataSourceEc2.  Up 830.52 seconds

Error building openssl within ruby 1.9.3 on centos-6.5

make[2]: Entering directory `/var/chef/cache/ruby-1.9.3-p392/ext/openssl'
compiling ossl_pkey_ec.c
make[2]: Leaving directory `/var/chef/cache/ruby-1.9.3-p392/ext/openssl'
make[1]: Leaving directory `/var/chef/cache/ruby-1.9.3-p392'
STDERR: old_thread_select.c: In function 'old_thread_select':
old_thread_select.c:57: warning: 'rb_thread_select' is deprecated (declared at ../../.././include/ruby/intern.h:380)
ossl_pkey_ec.c: In function 'ossl_ec_group_initialize':
ossl_pkey_ec.c:761: warning: assignment makes pointer from integer without a cast
ossl_pkey_ec.c:816: error: 'EC_GROUP_new_curve_GF2m' undeclared (first use in this function)
ossl_pkey_ec.c:816: error: (Each undeclared identifier is reported only once
ossl_pkey_ec.c:816: error: for each function it appears in.)
make[2]: *** [ossl_pkey_ec.o] Error 1
make[1]: *** [ext/openssl/all] Error 2
make: *** [build-ext] Error 2
ossl_pkey_ec.c: In function 'ossl_ec_group_initialize':
ossl_pkey_ec.c:761: warning: assignment makes pointer from integer without a cast
ossl_pkey_ec.c:816: error: 'EC_GROUP_new_curve_GF2m' undeclared (first use in this function)
ossl_pkey_ec.c:816: error: (Each undeclared identifier is reported only once
ossl_pkey_ec.c:816: error: for each function it appears in.)
make[2]: *** [ossl_pkey_ec.o] Error 1
make[1]: *** [ext/openssl/all] Error 2
make: *** [build-ext] Error 2
---- End output of tar zxf ruby-1.9.3-p392.tar.gz

...

Resource Declaration:
---------------------
# In /tmp/vagrant-chef-1/chef-solo-1/cookbooks/omnibus/recipes/ruby.rb

 59:   execute "install ruby-1.9.3" do
 60:     cwd Chef::Config[:file_cache_path]
 61:     command <<-EOH
 62: tar zxf #{ruby_filename}
 63: cd ruby-#{ruby_version}
 64: ./configure --prefix=/opt/ruby1.9
 65: make
 66: make install
 67: EOH
 68:     environment(
 69:       'CFLAGS' => '-L/usr/lib -I/usr/include',
 70:       'LDFLAGS' => '-L/usr/lib -I/usr/include'
 71:     )
 72:     not_if { ::File.exists?("/opt/ruby1.9/bin/ruby") }
 73:   end
 74:



Compiled Resource:
------------------
# Declared in /tmp/vagrant-chef-1/chef-solo-1/cookbooks/omnibus/recipes/ruby.rb:59:in `from_file'

execute("install ruby-1.9.3") do
  action "run"
  retries 0
  retry_delay 2
  command "tar zxf ruby-1.9.3-p392.tar.gz\ncd ruby-1.9.3-p392\n./configure --prefix=/opt/ruby1.9\nmake\nmake install\n"
  backup 5
  cwd "/var/chef/cache"
  environment {"CFLAGS"=>"-L/usr/lib -I/usr/include", "LDFLAGS"=>"-L/usr/lib -I/usr/include"}
  returns 0
  cookbook_name :omnibus
  recipe_name "ruby"
  not_if { #code block }
end



[2014-02-01T13:28:36+00:00] INFO: Forking chef instance to converge...
[2014-02-01T13:31:30+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
Chef never successfully completed! Any errors should be visible in the
output above. Please fix your recipes so that they properly complete.

Failing centos build reports as successful

irc announce:

12:54 Successful build of Flapjack maint/1.x for centos 6 package (experimental) in 6m 30s, log: http://is.gd/phW1Fd

However the log finishes with:

ERROR running docker command, exit status is 1
Successful build of maint/1.x on centos 6 at 2014-12-02T02:24:03Z, duration: 6m 30s

Bring README in line with reality

  • README is mostly Omnibus boilerplate.
  • We do some things differently, and they are detailed at the bottom of the current README.
  • We need to ensure the README details the common package building tasks.

Issues building an RPM with an CentOS 6.5 AWS AMI

Notes while trying to get this working.

Using a 64 bit, CentOS 6.5 AMI in Sydney via the CentOS AMI Marketplace

edit /etc/sysconfig/selinux and change SELINUX=enforcing to SELINUX=disabled, reboot

comment out the following in sudoers:

Defaults requiretty

vagrant provision aws-centos-6.5 now gets a bit further but:

[aws-centos-6.5] Installing Chef 11.12.4
installing with rpm...
...
[aws-centos-6.5] Thank you for installing Chef!
[aws-centos-6.5] Running provisioner: chef_solo...
...
[2014-05-22T01:58:05+00:00] INFO: package[libtool] installing libtool-2.2.6-15.5.el6 from base repository

================================================================================
Error executing action `install` on resource 'package[libyaml-devel]'
================================================================================


Chef::Exceptions::Package
-------------------------
No version specified, and no candidate version available for libyaml-devel


Resource Declaration:
---------------------
# In /tmp/vagrant-chef-1/chef-solo-1/cookbooks/omnibus/recipes/ruby.rb

 45:     package "libyaml-devel"
 46:     package "openssl-devel"



Compiled Resource:
------------------
# Declared in /tmp/vagrant-chef-1/chef-solo-1/cookbooks/omnibus/recipes/ruby.rb:45:in `from_file'

package("libyaml-devel") do
  action :install
  retries 0
  retry_delay 2
  guard_interpreter :default
  package_name "libyaml-devel"
  cookbook_name :omnibus
  recipe_name "ruby"
end



[2014-05-22T01:59:31+00:00] INFO: Running queued delayed notifications before re-raising exception
[2014-05-22T01:59:31+00:00] ERROR: Running exception handlers
[2014-05-22T01:59:31+00:00] ERROR: Exception handlers complete
[2014-05-22T01:59:31+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2014-05-22T01:59:31+00:00] ERROR: package[libyaml-devel] (omnibus::ruby line 45) had an error: Chef::Exceptions::Package: No version specified, and no candidate version available for libyaml-devel
[2014-05-22T01:59:31+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
Chef never successfully completed! Any errors should be visible in the
output above. Please fix your recipes so that they properly complete.

aws build workflow improvements

Currently when we build flapjack packages the workflow looks like this:

  • create a new ec2 launch configuration by cloning an existing one and updating the parameters to the build script within the user data script
  • modify an existing autoscaling group to use the new launch config
  • alter number of desired instances in the asg from 0 to 1 to kick off a build
  • log in to the instance and tail /var/log/cloud-init-output.log to determine success or failure
  • gist the log in case it's helpful later
  • change the number of desired instances in the asg from 1 to 0

This is part of the way to the nirvana described here: https://gist.github.com/jessereynolds/378bdbca81a4fa7b1760

What next to do? Something like:

  • test out an sqs queue that creates a cloud watch alarm when there are jobs on the queue
  • create an asg with a policy that uses this alarm as a trigger
  • work out how to pass params from the sqs job to the build script
  • get a Jenkins server going and a job that receives webhooks from github and creates build jobs on the sqs queue
  • improve maintain ability and more with cloud formation

The first two points were mapped out in some detail by @jdub

Unable to build due to outdated MD5 for cacert.pem in omnibus-software

Building the package fails when chef downloads the CA certificate bundle from http://curl.haxx.se/ca/cacert.pem due to a MD5 mismatch:

[fetcher:net::cacerts] fetching /var/cache/omnibus/cache/cacert.pem from http://curl.haxx.se/ca/cacert.pem
[fetcher:net::cacerts] getting from http://curl.haxx.se/ca/cacert.pem with 10 redirects left
[fetcher:net::cacerts] Invalid MD5 for cacerts
[fetcher:net::cacerts] Expected: 349ba2d6964db9ca558c9e1daf38e428
[fetcher:net::cacerts] Actual:   6253bb1b6696a190fdf7a2062003b21c
Something went wrong...the Omnibus just ran off the road!

Some background is in this gist (and other people hitting the same issue): https://gist.github.com/jessereynolds/6249527

Move to omnibus 4

Omnibus 4 looks fairly close to release, lets slowly start moving towards it.

  • We'll test this out using omnibus-ubuntu with the trusty-omnibus4 tag
  • We'll use the omnibus4 branch on this repo

Upstream omnibus yum problem

Building centos-6 package results in the following

[2014-03-19T19:49:08+00:00] INFO: Running queued delayed notifications before re-raising exception
[2014-03-19T19:49:08+00:00] ERROR: Running exception handlers
[2014-03-19T19:49:08+00:00] ERROR: Exception handlers complete
[2014-03-19T19:49:08+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2014-03-19T19:49:08+00:00] ERROR: package[libyaml-devel](omnibus::ruby line 45) had an error: Chef::Exceptions::Package: No version specified, and no candidate version available for libyaml-devel
[2014-03-19T19:49:08+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
Chef never successfully completed! Any errors should be visible in the
output above. Please fix your recipes so that they properly complete.

Make Flapjack install and run as a non-privileged user

To achieve this we will need to:

Have the package create the Flapjack user + group
Set Flapjack group writable on /var/log/flapjack + /var/run/flapjack
Alter the init script to start Flapjack as the Flapjack user

We're going to defer this from the 1.0 release:

If you're dedicating a machine to Flapjack, the risk isn't that great, but there will still be attack vectors.
Virtual machines are cheap. You probably should be dedicating a machine to Flapjack.
@jessereynolds says "we have no security controls around anything - why should we start now?" :smile:
We will need to address security issues eventually
We're developing with the assumption that there is an external security blanket
We probably should look at API tokens
Lack of focus on security could potentially damage Flapjack's reputation.
We should be really clear about our security strategy in the documentation.

Omnibus build on CentOS fails

You might have noticed I've been trying to get flapjack to build on CentOS with little succes.

For my current attempt I've tried both the vagrant box within the project and my own box.

Both fail at the same time. Well kinda. The omnibus-flapjack centos-6.5 box fails in the shell provisioner
https://gist.github.com/KrisBuytaert/7b8e9a17fc6deaf2b774
After which I can trivially run that shell part manually only to run into exactly the same errors as on my own

@auxesis suggested to look into the docker approach, but I`m kinda sure this will result in the same build errors.

The actual build log + errors can be found in this gist.

https://gist.github.com/KrisBuytaert/a2230a398a77d0374c62

Try Omnibus 3 with Docker

I'm having another go at upgrading to omnibus 3 (3.1.1 currently). Work is in the omnibus3 branch: https://github.com/flapjack/omnibus-flapjack/tree/omnibus3

docker ftw

I'm also making use of a docker image I've created which is ubuntu precise64 with:

  • ruby 1.9
  • omnibus-ruby 3.1.1
  • omnibus-software (latest)
  • omnibus-flapjack

It also has had omnibus build flapjack's dependencies so it's pretty quick to make new flapjack builds when starting from this image.

ruby 2.1.1

I have upgraded the embedded Ruby to 2.1.1, which made for some fun with bundler running from the system ruby (1.9.3) and then failing to find rake, so I've tried just doing a plain gem install from rubygems.org rather than building the gem during the package build. This approach seems to work however we now have some 'behind the scenes' gem executables appearing in /opt/flapjack/bin/ eg gli, rescue-web etc.

redis is sad

it seems the git caching stuff in the latest omnibus (or some other feature) is causing my dodgy redis software definition to fail to include the redis-flapjack init script and the redis-flapjack.conf file in /opt/flapjack and so the package postinstall script is failing ... I've tried a few things with this but I have to stop on this now and pick it up again when I return from holidays

publishing the package

I haven't yet automated the pushing of the generated package to s3 or to an unstable component in the packages.flapjack.io deb repo. I have been reading up on aptly for doing this and was thinking of something like:

  • store the 'local repo' in an s3 bucket and sync that down to the build container
  • update the repo
  • publish it to a local dir, then use aws-cli to sync it back up to s3 (though aptly is supposed to have a version late June that supports publishing to s3)
  • push the 'local repo' to s3 for future use

Perhaps there's a better way. ... Could also stick with the reprepro based script we're already using, at least initially.

0.9.4 build is failing

                [Builder: flapjack] I | Execute: `export gem_home=/`/opt/flapjack/embedded/bin/gem list --all --details flapjack | grep 'Installed at' | sed 's/^.* \///'` ; export installed_gem=`ls -dtr ${gem_home}/gems/flapjack* | tail -1` ; cd ${installed_gem} && ./build.sh': 70.7506s
                [Builder: flapjack] I | Build flapjack: 495.8623s
/var/lib/gems/1.9.1/gems/mixlib-shellout-1.4.0/lib/mixlib/shellout.rb:257:in `invalid!': Expected process to exit with [0], but received '127' (Mixlib::ShellOut::ShellCommandFailed)
---- Begin output of export gem_home=/`/opt/flapjack/embedded/bin/gem list --all --details flapjack | grep 'Installed at' | sed 's/^.* \///'` ; export installed_gem=`ls -dtr ${gem_home}/gems/flapjack* | tail -1` ; cd ${installed_gem} && ./build.sh ----
STDOUT:
STDERR: sh: 1: ./build.sh: not found
---- End output of export gem_home=/`/opt/flapjack/embedded/bin/gem list --all --details flapjack | grep 'Installed at' | sed 's/^.* \///'` ; export installed_gem=`ls -dtr ${gem_home}/gems/flapjack* | tail -1` ; cd ${installed_gem} && ./build.sh ----
Ran export gem_home=/`/opt/flapjack/embedded/bin/gem list --all --details flapjack | grep 'Installed at' | sed 's/^.* \///'` ; export installed_gem=`ls -dtr ${gem_home}/gems/flapjack* | tail -1` ; cd ${installed_gem} && ./build.sh returned 127
    from /var/lib/gems/1.9.1/gems/mixlib-shellout-1.4.0/lib/mixlib/shellout.rb:244:in `error!'
    from /var/lib/gems/1.9.1/bundler/gems/omnibus-d1315b588707/lib/omnibus/util.rb:84:in `shellout!'
    from /var/lib/gems/1.9.1/bundler/gems/omnibus-d1315b588707/lib/omnibus/builder.rb:723:in `_shellout!'
    from /var/lib/gems/1.9.1/bundler/gems/omnibus-d1315b588707/lib/omnibus/builder.rb:84:in `block in command'
    from /var/lib/gems/1.9.1/bundler/gems/omnibus-d1315b588707/lib/omnibus/builder.rb:930:in `instance_eval'
    from /var/lib/gems/1.9.1/bundler/gems/omnibus-d1315b588707/lib/omnibus/builder.rb:930:in `run'
    from /var/lib/gems/1.9.1/bundler/gems/omnibus-d1315b588707/lib/omnibus/builder.rb:741:in `block (3 levels) in execute'
    from /var/lib/gems/1.9.1/bundler/gems/omnibus-d1315b588707/lib/omnibus/builder.rb:766:in `call'
    from /var/lib/gems/1.9.1/bundler/gems/omnibus-d1315b588707/lib/omnibus/builder.rb:766:in `with_retries'
    from /var/lib/gems/1.9.1/bundler/gems/omnibus-d1315b588707/lib/omnibus/builder.rb:740:in `block (2 levels) in execute'
    from /var/lib/gems/1.9.1/bundler/gems/omnibus-d1315b588707/lib/omnibus/instrumentation.rb:23:in `call'
    from /var/lib/gems/1.9.1/bundler/gems/omnibus-d1315b588707/lib/omnibus/instrumentation.rb:23:in `measure'
    from /var/lib/gems/1.9.1/bundler/gems/omnibus-d1315b588707/lib/omnibus/builder.rb:739:in `block in execute'
    from /var/lib/gems/1.9.1/bundler/gems/omnibus-d1315b588707/lib/omnibus/builder.rb:809:in `call'
    from /var/lib/gems/1.9.1/bundler/gems/omnibus-d1315b588707/lib/omnibus/builder.rb:809:in `with_clean_env'
    from /var/lib/gems/1.9.1/bundler/gems/omnibus-d1315b588707/lib/omnibus/builder.rb:738:in `execute'
    from /var/lib/gems/1.9.1/bundler/gems/omnibus-d1315b588707/lib/omnibus/builder.rb:640:in `block (2 levels) in build'
    from /var/lib/gems/1.9.1/bundler/gems/omnibus-d1315b588707/lib/omnibus/builder.rb:639:in `each'
    from /var/lib/gems/1.9.1/bundler/gems/omnibus-d1315b588707/lib/omnibus/builder.rb:639:in `block in build'
    from /var/lib/gems/1.9.1/bundler/gems/omnibus-d1315b588707/lib/omnibus/instrumentation.rb:23:in `call'
    from /var/lib/gems/1.9.1/bundler/gems/omnibus-d1315b588707/lib/omnibus/instrumentation.rb:23:in `measure'
    from /var/lib/gems/1.9.1/bundler/gems/omnibus-d1315b588707/lib/omnibus/builder.rb:638:in `build'
    from /var/lib/gems/1.9.1/bundler/gems/omnibus-d1315b588707/lib/omnibus/software.rb:981:in `execute_build'
    from /var/lib/gems/1.9.1/bundler/gems/omnibus-d1315b588707/lib/omnibus/software.rb:844:in `build_me'
    from /var/lib/gems/1.9.1/bundler/gems/omnibus-d1315b588707/lib/omnibus/project.rb:971:in `block in build_me'
    from /var/lib/gems/1.9.1/bundler/gems/omnibus-d1315b588707/lib/omnibus/project.rb:970:in `each'
    from /var/lib/gems/1.9.1/bundler/gems/omnibus-d1315b588707/lib/omnibus/project.rb:970:in `build_me'
    from /var/lib/gems/1.9.1/bundler/gems/omnibus-d1315b588707/lib/omnibus/cli.rb:72:in `build'
    from /var/lib/gems/1.9.1/gems/thor-0.19.1/lib/thor/command.rb:27:in `run'
    from /var/lib/gems/1.9.1/gems/thor-0.19.1/lib/thor/invocation.rb:126:in `invoke_command'
    from /var/lib/gems/1.9.1/gems/thor-0.19.1/lib/thor.rb:359:in `dispatch'
    from /var/lib/gems/1.9.1/bundler/gems/omnibus-d1315b588707/lib/omnibus/cli/base.rb:33:in `dispatch'
    from /var/lib/gems/1.9.1/bundler/gems/omnibus-d1315b588707/lib/omnibus/cli/deprecated.rb:128:in `dispatch'
    from /var/lib/gems/1.9.1/gems/thor-0.19.1/lib/thor/base.rb:440:in `start'
    from /var/lib/gems/1.9.1/bundler/gems/omnibus-d1315b588707/lib/omnibus/cli.rb:41:in `execute!'
    from /var/lib/gems/1.9.1/bundler/gems/omnibus-d1315b588707/bin/omnibus:11:in `<top (required)>'
    from bin/omnibus:16:in `load'
    from bin/omnibus:16:in `<main>'

0.9.4 precise experimental package doesn't start flapjack

Looks like a 1.x init script has crept into the package installation:

err: /Stage[main]/Flapjack::Install/Package[flapjack]/ensure: change from purged to present failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install flapjack' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  flapjack
0 upgraded, 1 newly installed, 0 to remove and 166 not upgraded.
Need to get 58.6 MB of archives.
After this operation, 183 MB of additional disk space will be used.
Get:1 http://packages.flapjack.io/deb/0.9/ precise/experimental flapjack amd64 0.9.4~+20141007072644-v0.9.4-precise-1 [58.6 MB]
Fetched 58.6 MB in 3min 41s (264 kB/s)
Selecting previously unselected package flapjack.
(Reading database ... 54026 files and directories currently installed.)
Unpacking flapjack (from .../flapjack_0.9.4~+20141007072644-v0.9.4-precise-1_amd64.deb) ...
Setting up flapjack (0.9.4~+20141007072644-v0.9.4-precise-1) ...
Creating /etc/flapjack/flapjack_config.yaml
Creating /etc/init.d/redis-flapjack
Creating /etc/init.d/flapjack
Creating /etc/init.d/flapjack-nagios-receiver
Creating /etc/init.d/flapper
Setting paths
Starting redis-flapjack
Starting redis-server: redis-server.
Starting flapjack
Unknown command provided: 'server'

Usage: flapjack COMMAND [OPTIONS]

Commands
     start                           start flapjack
     stop                            stop flapjack
     restart                         (re)start flapjack
     reload                          reload flapjack configuration
     status                          see if flapjack is running
     version                         display flapjack version and exit
     help                            display this usage info

Options
    -c, --config [PATH]              PATH to the config file to use
    -n, --environment [ENV]          Environment to boot
    -d, --[no-]daemonize             Daemonize?
    -r, --rbtrace                    Enable rbtrace profiling
    -p, --pidfile [PATH]             PATH to the pidfile to write to
    -l, --logfile [PATH]             PATH to the logfile to write to
    -v, --version                    display flapjack version
dpkg: error processing flapjack (--configure):
 subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
 flapjack
E: Sub-process /usr/bin/dpkg returned an error code (1)

Add signing to deb / ubuntu package repo foo

Currently the reprepro based debian repository maintenance steps listed in the README do not involve any signing of the .deb packages. This means consumers have to make a greater leap of faith but installing the package, and put something like the following in /etc/apt/apt.conf.d/99auth:

APT::Get::AllowUnauthenticated yes;

Yurg

Refer to how this is worked around currently in vagrant-flapjack: https://github.com/flpjck/vagrant-flapjack/blob/cae775a6cc027447a29fdb7090fe770f0f909033/dist/manifests/site.pp#L11-L20

nokogiri omnibus 3.2 error - WHITELIST_LIBS is now frozen

My duckpunch of Omnibus::HealthCheck to add libz and linux-gate to WHITELIST_LIBS fails on omnibus 3.2.1 as this array is now frozen:

docker build

[Builder: libffi] W | Detected command `cp'. Consider using the `copy' DSL method. 
�[91m/omnibus-flapjack/config/software/nokogiri.rb:36:in `<class:HealthCheck>'�[0m 
�[91m: �[0m 
�[91mcan't modify frozen Array�[0m 
�[91m (�[0m 
�[91mRuntimeError�[0m 
�[91m)
�[0m 
�[91m   from /omnibus-flapjack/config/software/nokogiri.rb:35:in `<module:Omnibus>'
�[0m 
�[91m   from /omnibus-flapjack/config/software/nokogiri.rb:34:in `evaluate'
�[0m 
�[91m   from /var/lib/gems/1.9.1/bundler/gems/omnibus-ruby-0dfe3badf4c7/lib/omnibus/cleanroom.rb:58:in `instance_eval'
�[0m 

Flapjack 1.2.1 debian package failing to install

When I try to run either an upgrade from 1.2.0 OR a clean install of 1.2.1 on an Ubuntu 14.04.1 system, I get the following:

Preparing to unpack flapjack_1.2.1-trusty_amd64.deb ...
Unpacking flapjack (1.2.1-trusty) ...
dpkg-deb (subprocess): cannot copy archive member from 'flapjack_1.2.1-trusty_amd64.deb' to decompressor pipe: unexpected end of file or stream
dpkg-deb (subprocess): decompressing archive member: lzma error: unexpected end of input
dpkg-deb: error: subprocess <decompress> returned error exit status 2
dpkg: error processing archive flapjack_1.2.1-trusty_amd64.deb (--install):
 cannot copy extracted data for './opt/flapjack/embedded/share/gtk-doc/html/libxml2/libxml2.devhelp' to '/opt/flapjack/embedded/share/gtk-doc/html/libxml2/libxml2.devhelp.dpkg-new': unexpected end of file or stream
 Removing any system startup links for /etc/init.d/flapjack ...
 Removing any system startup links for /etc/init.d/flapjack-nagios-receiver ...
 Removing any system startup links for /etc/init.d/flapper ...
 Removing any system startup links for /etc/init.d/redis-flapjack ...
rm: cannot remove ‘/etc/init.d/flapjack’: No such file or directory
rm: cannot remove ‘/etc/init.d/flapjack-nagios-receiver’: No such file or directory
rm: cannot remove ‘/etc/init.d/flapper’: No such file or directory
rm: cannot remove ‘/etc/init.d/redis-flapjack’: No such file or directory
flapjack has been uninstalled!
Errors were encountered while processing:
 flapjack_1.2.1-trusty_amd64.deb

debian build fails to run post build tests

Tail end of build log:

(Reading database ... 6709 files and directories currently installed.)
Unpacking flapjack (from .../flapjack_1.2.1~rc1~20141204023223-v1.2.1rc1-wheezy-1_amd64.deb) ...
Setting up flapjack (1.2.1~rc1~20141204023223-v1.2.1rc1-wheezy-1) ...
/var/lib/dpkg/info/flapjack.postinst: line 18: adduser: command not found
chown: invalid user: `flapjack:flapjack'
Creating /etc/flapjack/flapjack_config.yaml
Creating /usr/local/lib/flapjackfeeder.o
Creating /etc/init.d/redis-flapjack
Creating /etc/init.d/flapjack
Creating /etc/init.d/flapjack-nagios-receiver
Creating /etc/init.d/flapper
Setting paths
insserv: warning: script 'flapper' missing LSB tags and overrides
insserv: warning: script 'flapjack-nagios-receiver' missing LSB tags and overrides
insserv: warning: script 'flapper' missing LSB tags and overrides
insserv: warning: script 'flapjack-nagios-receiver' missing LSB tags and overrides
Starting redis-flapjack
Starting redis-server: chown: invalid user: `flapjack:flapjack'
dpkg: error processing flapjack (--install):
 subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
 flapjack

Reading package lists... 0%

Reading package lists... 0%

Reading package lists... Done


Building dependency tree... 0%

Building dependency tree... 0%

Building dependency tree... 3%

Building dependency tree... 50%

Building dependency tree... 50%

Building dependency tree... Done

E: Unable to locate package ruby1.9.1-full
E: Couldn't find any package by regex 'ruby1.9.1-full'
E: Unable to locate package git
E: Unable to locate package nagios3
E: Unable to locate package phantomjs
E: Unable to locate package net-tools
STDERR: 
Unable to find image 'debian:wheezy' locally
debian:wheezy: The image you are pulling has been verified
Status: Downloaded newer image for debian:wheezy
ERROR running docker command, exit status is 100

I, [2014-12-04T02:48:55.134248 #20264]  INFO -- : Failed build of v1.2.1rc1 on debian wheezy at 2014-12-04T02:48:55Z, duration: 16m 32s

Build broken - undefined method `default_version' for Omnibus::Software

It seems the build is broken due to omnibus-software now requiring omnibus-ruby >= 2.0, and we're using 1.2.0.

Today I tried to make a build. I got the usual Checksum of downloaded file /var/cache/omnibus/cache/cacert.pem doesn't match expected:

[fetcher:net::cacerts] fetching /var/cache/omnibus/cache/cacert.pem from http://curl.haxx.se/ca/cacert.pem
[fetcher:net::cacerts] getting from http://curl.haxx.se/ca/cacert.pem with 10 redirects left
[fetcher:net::cacerts] Invalid MD5 for cacerts
[fetcher:net::cacerts] Expected: 5d108f8ab86afacc6663aafca8604dd3
[fetcher:net::cacerts] Actual:   9f92a0d9f605e227ae068e605f4c86fa
Something went wrong...the Omnibus just ran off the road!

... and so I did a bundle update to get the latest omnibus-software, with the latest md5 hashes for cacert.pem:

$ bundle update
Updating git://github.com/opscode/omnibus-software.git
Cloning into '/Users/jesse/.rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/bundler/gems/omnibus-software-63f0c8fa76e6'...

And now, when I do a build I get:

...
Installing omnibus (1.2.0)
Using omnibus-software (0.0.1) from git://github.com/opscode/omnibus-software.git (at master)
sing bundler (1.3.5)
Your bundle is complete!
Use `bundle show [gemname]` to see where a bundled gem is installed.
No configuration file `/home/ubuntu/omnibus-flapjack/omnibus.rb', using defaults
No configuration file `/home/ubuntu/omnibus-flapjack/omnibus.rb', using defaults
Something went wrong...the Omnibus just ran off the road!

Error raised was:

    Could not load the Omnibus projects.

Backtrace:

    /opt/ruby1.9/lib/ruby/gems/1.9.1/gems/omnibus-1.2.0/lib/omnibus/cli/base.rb:63:in `rescue in initialize'
    /opt/ruby1.9/lib/ruby/gems/1.9.1/gems/omnibus-1.2.0/lib/omnibus/cli/base.rb:59:in `initialize'
    /opt/ruby1.9/lib/ruby/gems/1.9.1/gems/thor-0.19.1/lib/thor.rb:355:in `new'
    /opt/ruby1.9/lib/ruby/gems/1.9.1/gems/thor-0.19.1/lib/thor.rb:355:in `dispatch'
    /opt/ruby1.9/lib/ruby/gems/1.9.1/gems/thor-0.19.1/lib/thor/invocation.rb:115:in `invoke'
    /opt/ruby1.9/lib/ruby/gems/1.9.1/gems/thor-0.19.1/lib/thor.rb:235:in `block in subcommand'
    /opt/ruby1.9/lib/ruby/gems/1.9.1/gems/thor-0.19.1/lib/thor/command.rb:27:in `run'
    /opt/ruby1.9/lib/ruby/gems/1.9.1/gems/thor-0.19.1/lib/thor/invocation.rb:126:in `invoke_command'
    /opt/ruby1.9/lib/ruby/gems/1.9.1/gems/thor-0.19.1/lib/thor.rb:359:in `dispatch'
    /opt/ruby1.9/lib/ruby/gems/1.9.1/gems/thor-0.19.1/lib/thor/base.rb:440:in `start'
    /opt/ruby1.9/lib/ruby/gems/1.9.1/gems/omnibus-1.2.0/lib/omnibus/cli/application.rb:123:in `start'
    /opt/ruby1.9/lib/ruby/gems/1.9.1/gems/omnibus-1.2.0/bin/omnibus:11:in `<top (required)>'
    bin/omnibus:16:in `load'
    bin/omnibus:16:in `<main>'

Original Error:

    undefined method `default_version' for #<Omnibus::Software:0x000000024b6840>

Original Backtrace:

    /home/ubuntu/.bundler/ruby/1.9.1/omnibus-software-63f0c8fa76e6/config/software/preparation.rb:19:in `initialize'
    /opt/ruby1.9/lib/ruby/gems/1.9.1/gems/omnibus-1.2.0/lib/omnibus/software.rb:90:in `instance_eval'
    /opt/ruby1.9/lib/ruby/gems/1.9.1/gems/omnibus-1.2.0/lib/omnibus/software.rb:90:in `initialize'
    /opt/ruby1.9/lib/ruby/gems/1.9.1/gems/omnibus-1.2.0/lib/omnibus/software.rb:57:in `new'
    /opt/ruby1.9/lib/ruby/gems/1.9.1/gems/omnibus-1.2.0/lib/omnibus/software.rb:57:in `load'
    /opt/ruby1.9/lib/ruby/gems/1.9.1/gems/omnibus-1.2.0/lib/omnibus.rb:296:in `recursively_load_dependency'
    /opt/ruby1.9/lib/ruby/gems/1.9.1/gems/omnibus-1.2.0/lib/omnibus.rb:194:in `block (2 levels) in expand_software'
    /opt/ruby1.9/lib/ruby/gems/1.9.1/gems/omnibus-1.2.0/lib/omnibus.rb:193:in `each'
    /opt/ruby1.9/lib/ruby/gems/1.9.1/gems/omnibus-1.2.0/lib/omnibus.rb:193:in `block in expand_software'
    /opt/ruby1.9/lib/ruby/gems/1.9.1/gems/omnibus-1.2.0/lib/omnibus.rb:192:in `each'
    /opt/ruby1.9/lib/ruby/gems/1.9.1/gems/omnibus-1.2.0/lib/omnibus.rb:192:in `expand_software'
    /opt/ruby1.9/lib/ruby/gems/1.9.1/gems/omnibus-1.2.0/lib/omnibus.rb:213:in `process_dsl_files'
    /opt/ruby1.9/lib/ruby/gems/1.9.1/gems/omnibus-1.2.0/lib/omnibus.rb:86:in `process_configuration'
    /opt/ruby1.9/lib/ruby/gems/1.9.1/gems/omnibus-1.2.0/lib/omnibus/cli/base.rb:60:in `initialize'
    /opt/ruby1.9/lib/ruby/gems/1.9.1/gems/thor-0.19.1/lib/thor.rb:355:in `new'
    /opt/ruby1.9/lib/ruby/gems/1.9.1/gems/thor-0.19.1/lib/thor.rb:355:in `dispatch'
    /opt/ruby1.9/lib/ruby/gems/1.9.1/gems/thor-0.19.1/lib/thor/invocation.rb:115:in `invoke'
    /opt/ruby1.9/lib/ruby/gems/1.9.1/gems/thor-0.19.1/lib/thor.rb:235:in `block in subcommand'
    /opt/ruby1.9/lib/ruby/gems/1.9.1/gems/thor-0.19.1/lib/thor/command.rb:27:in `run'
    /opt/ruby1.9/lib/ruby/gems/1.9.1/gems/thor-0.19.1/lib/thor/invocation.rb:126:in `invoke_command'
    /opt/ruby1.9/lib/ruby/gems/1.9.1/gems/thor-0.19.1/lib/thor.rb:359:in `dispatch'
    /opt/ruby1.9/lib/ruby/gems/1.9.1/gems/thor-0.19.1/lib/thor/base.rb:440:in `start'
    /opt/ruby1.9/lib/ruby/gems/1.9.1/gems/omnibus-1.2.0/lib/omnibus/cli/application.rb:123:in `start'
    /opt/ruby1.9/lib/ruby/gems/1.9.1/gems/omnibus-1.2.0/bin/omnibus:11:in `<top (required)>'
    bin/omnibus:16:in `load'
    bin/omnibus:16:in `<main>'

Build Go bits as part of the package build

We need to add bits to the config/software/flapjack.rb that do the build of Go components (pandik, httpbroker, oneoff).

It should just be a case of:

  • Installing Go
  • Calling ./build.sh from within the Flapjack gem install path.

omnibus 3.2.1 compatability - undefined method `install_path' for Omnibus::Project

The omnibus-ubuntu docker image build is failing now that omnibus 3.2.1 is available (previous working build used version 3.1.1 of omnibus)

...
Using omnibus 3.2.1 from git://github.com/opscode/omnibus-ruby.git (at master) 
Using omnibus-software 0.0.1 from git://github.com/opscode/omnibus-software.git (at master) 
Using bundler 1.6.5 
Your bundle is complete! 
Use `bundle show [gemname]` to see where a bundled gem is installed. 
�[91m/omnibus-flapjack/config/projects/flapjack-dependencies.rb:7:in `evaluate': undefined method `install_path' for #<Omnibus::Project (Cleanroom)> (NoMethodError)
    from /var/lib/gems/1.9.1/bundler/gems/omnibus-ruby-0dfe3badf4c7/lib/omnibus/cleanroom.rb:58:in `instance_eval'
�[0m 
�[91m   from /var/lib/gems/1.9.1/bundler/gems/omnibus-ruby-0dfe3badf4c7/lib/omnibus/cleanroom.rb:58:in `evaluate'
    from /var/lib/gems/1.9.1/bundler/gems/omnibus-ruby-0dfe3badf4c7/lib/omnibus/cleanroom.rb:44:in `evaluate_file'
    from /var/lib/gems/1.9.1/bundler/gems/omnibus-ruby-0dfe3badf4c7/lib/omnibus/cleanroom.rb:125:in `evaluate_file'
    from /var/lib/gems/1.9.1/bundler/gems/omnibus-ruby-0dfe3badf4c7/lib/omnibus/project.rb:51:in `load'
    from /var/lib/gems/1.9.1/bundler/gems/omnibus-ruby-0dfe3badf4c7/lib/omnibus.rb:190:in `project'
    from /var/lib/gems/1.9.1/bundler/gems/omnibus-ruby-0dfe3badf4c7/lib/omnibus/cli.rb:68:in `build'
    from /var/lib/gems/1.9.1/gems/thor-0.19.1/lib/thor/command.rb:27:in `run'
    from /var/lib/gems/1.9.1/gems/thor-0.19.1/lib/thor/invocation.rb:126:in `invoke_command'
    from /var/lib/gems/1.9.1/gems/thor-0.19.1/lib/thor.rb:359:in `dispatch'
    from /var/lib/gems/1.9.1/bundler/gems/omnibus-ruby-0dfe3badf4c7/lib/omnibus/cli/base.rb:33:in `dispatch'
    from /var/lib/gems/1.9.1/gems/thor-0.19.1/lib/thor/base.rb:440:in `start'
    from /var/lib/gems/1.9.1/bundler/gems/omnibus-ruby-0dfe3badf4c7/lib/omnibus/cli.rb:41:in `execute!'
    from /var/lib/gems/1.9.1/bundler/gems/omnibus-ruby-0dfe3badf4c7/bin/omnibus:11:in `<top (required)>'
    from bin/omnibus:16:in `load'
    from bin/omnibus:16:in `<main>'
�[0m 
The command [/bin/sh -c git clone --branch omnibus3 https://github.com/flapjack/omnibus-flapjack.git &&     cd omnibus-flapjack &&     bundle install --binstubs &&     bin/omnibus build flapjack-dependencies] returned a non-zero code: 1

nokogiri 1.6.0 is depending on system libz.so.1

I think this is because flapjack depends on nokogiri 1.6.0 but the omnibus-software nokigiri definition is building 1.5.x for us.

To further confuse the issue, my local Gemfile.lock reads like flapjack is depending on both nokogiri versions, but that doesn't make sense:

GEM
  remote: https://rubygems.org/
  specs:
    blather (0.8.7)
      niceogiri (~> 1.0)
      nokogiri (~> 1.5, >= 1.5.6)
    niceogiri (1.1.2)
      nokogiri (~> 1.5)
    nokogiri (1.6.0)

Make package pushing optional?

Using the latest version of the omnibus-flapjack repo I get the following when using the 'centos-6' box. The rpm builds fine but throws the following error.

[SNIP]
timestamp=>"2014-01-31T21:28:53.085948+0000", :message=>"no value for epoch is set, defaulting to nil", :level=>:warn}
{:timestamp=>"2014-01-31T21:31:19.516245+0000", :message=>"Created rpm", :path=>"flapjack-0.8.4+20140131211658-1.el6.x86_64.rpm"}

Stderr from the command:

mkdir -p /var/cache/omnibus/build/flapjack
mkdir -p /var/cache/omnibus/src
mkdir -p /var/cache/omnibus/cache
mkdir -p /var/cache/omnibus/src/preparation
touch /var/cache/omnibus/build/flapjack/preparation.fetch
touch /var/cache/omnibus/build/flapjack/preparation.manifest
mkdir -p /var/cache/omnibus/src/version-manifest
touch /var/cache/omnibus/build/flapjack/version-manifest.fetch
touch /var/cache/omnibus/build/flapjack/version-manifest.manifest
mkdir -p /var/cache/omnibus/src/libgcc
touch /var/cache/omnibus/build/flapjack/libgcc.fetch
touch /var/cache/omnibus/build/flapjack/libgcc.manifest
mkdir -p /var/cache/omnibus/src/zlib-1.2.6
touch /var/cache/omnibus/build/flapjack/zlib.fetch
touch /var/cache/omnibus/build/flapjack/zlib.manifest
mkdir -p /var/cache/omnibus/src/ncurses-5.9
touch /var/cache/omnibus/build/flapjack/ncurses.fetch
touch /var/cache/omnibus/build/flapjack/ncurses.manifest
mkdir -p /var/cache/omnibus/src/libedit-20120601-3.0
touch /var/cache/omnibus/build/flapjack/libedit.fetch
touch /var/cache/omnibus/build/flapjack/libedit.manifest
mkdir -p /var/cache/omnibus/src/cacerts-2014.01.28
touch /var/cache/omnibus/build/flapjack/cacerts.fetch
touch /var/cache/omnibus/build/flapjack/cacerts.manifest
mkdir -p /var/cache/omnibus/src/openssl-1.0.1f
touch /var/cache/omnibus/build/flapjack/openssl.fetch
touch /var/cache/omnibus/build/flapjack/openssl.manifest
mkdir -p /var/cache/omnibus/src/yaml-0.1.4
touch /var/cache/omnibus/build/flapjack/libyaml.fetch
touch /var/cache/omnibus/build/flapjack/libyaml.manifest
mkdir -p /var/cache/omnibus/src/libiconv-1.14
touch /var/cache/omnibus/build/flapjack/libiconv.fetch
touch /var/cache/omnibus/build/flapjack/libiconv.manifest
mkdir -p /var/cache/omnibus/src/ruby-1.9.3-p484
touch /var/cache/omnibus/build/flapjack/ruby.fetch
touch /var/cache/omnibus/build/flapjack/ruby.manifest
mkdir -p /var/cache/omnibus/src/redis-2.8.3
touch /var/cache/omnibus/build/flapjack/redis.fetch
touch /var/cache/omnibus/build/flapjack/redis.manifest
mkdir -p /var/cache/omnibus/src/rubygems-1.8.24
touch /var/cache/omnibus/build/flapjack/rubygems.fetch
touch /var/cache/omnibus/build/flapjack/rubygems.manifest
mkdir -p /var/cache/omnibus/src/yajl-ruby
touch /var/cache/omnibus/build/flapjack/yajl.fetch
touch /var/cache/omnibus/build/flapjack/yajl.manifest
mkdir -p /var/cache/omnibus/src/libxml2-2.7.8
touch /var/cache/omnibus/build/flapjack/libxml2.fetch
touch /var/cache/omnibus/build/flapjack/libxml2.manifest
mkdir -p /var/cache/omnibus/src/libxslt-1.1.26
touch /var/cache/omnibus/build/flapjack/libxslt.fetch
touch /var/cache/omnibus/build/flapjack/libxslt.manifest
mkdir -p /var/cache/omnibus/src/nokogiri
touch /var/cache/omnibus/build/flapjack/nokogiri.fetch
touch /var/cache/omnibus/build/flapjack/nokogiri.manifest
mkdir -p /var/cache/omnibus/src/bundler
touch /var/cache/omnibus/build/flapjack/bundler.fetch
touch /var/cache/omnibus/build/flapjack/bundler.manifest
mkdir -p /var/cache/omnibus/src/flapjack
touch /var/cache/omnibus/build/flapjack/flapjack.fetch
touch /var/cache/omnibus/build/flapjack/flapjack.manifest
mkdir -p /var/cache/omnibus/pkg
mkdir -p pkg
hacks/configure_awscli:28:in <main>': --aws-access-key-id must be supplied (RuntimeError) ls: cannot access /home/vagrant/omnibus-flapjack/pkg/flapjack*deb: No such file or directory hacks/push_package_to_s3:17:in

': no deb found at "" (RuntimeError)

Is Debian package will be avialable?

We using Debian Stable in production, so would be great to see debian packages. I think there is not so much work to port existing ubuntu packages.

on version strings for debian packages

In the omnibus3 branch we've changed the flapjack software definition so that it pulls flapjack from github, rather than from rubygems.org. This has the advantage that we can now build flapjack packages from master, or any tag, without having to first publish the tagged version on rubygems.org.

But the version identifier needs some changes. Here's some points identified when discussing this with @Hobbsee, @ali-graham:

VERSION[~VERSIONMOD][~TIMESTAMP-SHA1]-DEBIAN_REVISION

  • when building from a prerelease tag, eg a release candidate, insert ~ before the first alpha char, eg tag v1.0.0rc3 => 1.0.0~rc3 - this is so the release candidate sorts earlier than the full release of 1.0.0 (so that installing 1.0.0 over the top of 1.0.0rc3 would work)
  • ~TIMESTAMP-SHA1 should be included when building from a branch name or commit sha1. It's not to be included when building from a tag. The SHA1 part is the first 7 chars of the SHA-1 hash of the commit being built from
  • for branches, the version string needs to be fetched from Flapjack::VERSION (which will correspond to the previous release unless the version has been bumped in readiness for a new version}. So, if a timestamp and sha-1 is present, it means the version doesn't correspond to a tag, but rather an arbitrary commit.
  • TIMESTAMP is the time the build commences, UTC (not applicable when building from a tag)

I think this is going to require a wrapper script in omnibus-flapjack that calculates the version string to use, and then runs the docker build command with it set up in env vars:

eg:

$ package_build master
=>
FLAPJACK_BUILD_REF="9c027de"
FLAPJACK_PACKAGE_VERSION="1.0.0~rc3~20140724T180446-9c027de-1"
...
$ package_build v1.0.0rc3
=>
FLAPJACK_BUILD_REF="v1.0.0rc3"
FLAPJACK_PACKAGE_VERSION="1.0.0~rc3-1"
...
$ package_build 75a209e
=>
FLAPJACK_BUILD_REF="75a209e"
FLAPJACK_PACKAGE_VERSION="1.0.0~rc2~20140724T180446-75a209e-1"

The flapjack version string can be retrieved by the wrapper script like so:

ruby -e 'load "lib/flapjack/version.rb" ; puts Flapjack::VERSION'

Sign RPM packages

While we have easy deb signing, our RPM packages are currently unsigned.

This should be fixed.

promote rake task is failing on add_to_deb_repo

per discussion with @Hobbsee we need to fix the Package#version method so it works when truth is from filename. There's a regex to do this currently in the promote rake task that can be moved to the package class.

Rpm package name configuration

It would be nice to be able to control the name of the created package or have different package names for deb's and rpm's. Currently an rpm package looks like this (the version in the rpm is 1.2.0rc220141017115520_v1.2.0rc2_6)

flapjack-1.2.0rc220141017115520_v1.2.0rc2_6-1.el6.x86_64.rpm

It would be ideal if the final 1.2 release rpm looked liked something like the following
flapjack-1.2.0-1.el6.x86_64.rpm

with the version string in the package being 1.2.0

Thanks. Derek.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.