webtrends / storm

Chef cookbook for storm

Description

Installs Twitter's Storm distributed computation platform. Includes recipes for installing both the Nimbus / Web UI component and the Supervisor component.

Requirements

• Ubuntu 10.04 / 12.04

• May function on other distributions, but has not been tested

• java cookbook

• runit cookbook

Attributes

Usage

This recipe relies on two setup components that need to be noted as they are not used in many (or any) community cookbooks.

Role Based Cluster Setup: This cookbook relies on a cluster identification role to allow more than one storm cluster to run in a single Chef environment, while not breaking Chef search. Create a role with a name of your choosing. The role may be left empty or you may use it to apply the your application's topology and all necessary JARs within your topology. You will need to specify the name of this role using the node attribute ['storm']['cluster_role'], which is empty by default. You will need to apply this cluster role to both supervisor and the nimbus/UI node in your cluster

Deploy Flag: This cookbook uses a deploy flag to prevent the application from deploying unless desired and allows for an undeploy recipe to run prior to the deploy. The deploy recipe will also cleanup the state of storm and is sufficient to wipe clean any topology deploy, although it does not stop the actual topology (that's in the works). Once you've applied the supervisor or nimbus recipes to a node you need to have "deploy_build=true" set in your shell. "sudo deploy_build=true chef-client" can be used to set the environment variable and run Chef in a single command.

storm's Issues

zeromq package not available in ubuntu precise

ERROR: package[zeromq](storm::default line 32) had an error: Chef::Exceptions::Package: No version specified, and no candidate version available for zeromq

sudo apt-get install zeromq
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package zeromq

libzmq-dev does however:

sudo apt-get install libzmq-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
libpgm-5.1-0 libzmq1
The following NEW packages will be installed:
libpgm-5.1-0 libzmq-dev libzmq1
0 upgraded, 3 newly installed, 0 to remove and 127 not upgraded.
Need to get 526 kB of archives.
After this operation, 1,548 kB of additional disk space will be used

support RHEL

I've already got this work started, but I had a question so I thought I'd start with where I am. I want to converge what we have grown in-house with this cookbook, and first thing blocking me is redhat support.

I see you have .debs in /debs, and I took a similar approach, creating an /rpms folder -- I'm not finding anything in the chef doc on how it would pick up those files. Are you just source controlling them for your convenience? How are you getting chef to use those files as the packages it installs with the package provider?

Compile Error in recipes/default.rb

[2014-03-30T13:06:31+00:00] INFO: Forking chef instance to converge...
[2014-03-30T13:06:31+00:00] INFO: *** Chef 11.10.4 ***
[2014-03-30T13:06:31+00:00] INFO: Chef-client pid: 1929
[2014-03-30T13:06:31+00:00] INFO: Setting the run_list to ["role[development]"] from JSON
[2014-03-30T13:06:31+00:00] INFO: Run List is [role[development]]
[2014-03-30T13:06:31+00:00] INFO: Run List expands to [nodejs, storm]
[2014-03-30T13:06:31+00:00] INFO: Starting Chef Run for catapult.dev
[2014-03-30T13:06:31+00:00] INFO: Running start handlers
[2014-03-30T13:06:31+00:00] INFO: Start handlers complete.
[2014-03-30T13:06:32+00:00] WARN: Failed to read the private key /etc/chef/client.pem: #<Errno::ENOENT: No such file or directory - /etc/chef/client.pem>

Recipe Compile Error in /tmp/vagrant-chef-1/chef-solo-1/cookbooks/storm/recipes/default.rb