cph / heroku-log-store

An http-based log drain. Drain your Heroku app's logs to a Postgres database...also on Heroku.

heroku-log-store

heroku-log-store is an evented Ruby service for receiving and storing syslog messages over HTTP. Its purpose is to function as a Heroku https log drain which stores log data in a Postgres database for long-term log storage and very simple querying.

heroku-log-store makes no assumptions about the format of your data, merely dumping payload data into a Postgres table which you can then query yourself. Think of heroku-log-store as a long-term but queryable place to store your log data. Future versions may built simple introspection functionality on top of this data, but I reserve the right to keep it real dumb.

Currently, heroku-log-store is seeing lite action in production. Please report any issues in the project's Github issue tracker.

Deploy

To have heroku-log-store store your logs, deploy an instance of heroku-log-store to Heroku. Yes, a log-drain app on Heroku, draining logs from another app on Heroku. Just don't create a circular reference or else the universe might rip.

Note: This assumes you have the Heroku Toolbelt installed locally. Also, these steps install the dev heroku-postgresql add-on which is limited to 10K rows. This is not a lot of capacity as logs can be quite chatty. It is recommended that you use at least the basic postgres database. If you want to do this just specify heroku-postgresql:basic below instead.

$ git clone git://github.com/rwdaigle/heroku-log-store.git
$ cd heroku-log-store
$ heroku create log-store
$ heroku config:set RACK_ENV=production
$ heroku addons:add heroku-postgresql:dev
$ git push heroku master

Your drain should now be deployed and running on Heroku, although it will need to be configured before use.

Configure

To configure your heroku-log-store instance you need to create the basic table structure in the database. There is a process type called migrate you can invoke with heroku run that will do this for you.

$ heroku run migrate
Running `migrate` attached to terminal... up, run.4179
WARNING:  => is deprecated as an operator name
DETAIL:  This name may be disallowed altogether in future versions of PostgreSQL.

Now open the app to see instance-specific instructions such as setting up http basic authentication and setting the instance to be a log drain for a source app.

$ heroku open
Opening log-store... done

You will be prompted for a username/password. Until the credentials are explicitly set, you can login with anything.

Leave both fields blank and click "Log in". You should now see instructions similar to:

Follow the instructions in the app to properly setup security and to set the app as a log drain for another app. If you weren't able to reach this point, please submit an issue to the project's Github issue tracker.

Querying

heroku-log-store is not a replacement for Papertrail or any other true log-aggregation and eventing service. As such, its interface is limited to SQL.

Once your heroku-log-store is receiving log data you can query/filter it by opening a Postgres shell from the log-drain app.

$ heroku pg:psql
u7ejsaid=#

The events table is where the goods are. Here is its schema:

=# \d events
                                      Table "public.events"
     Column      |           Type           |                      Modifiers                      
-----------------+--------------------------+-----------------------------------------------------
 id              | bigint                   | not null default nextval('events_id_seq'::regclass)
 emitted_at      | timestamp with time zone | 
 received_at     | timestamp with time zone | default now()
 priority        | integer                  | 
 syslog_version  | integer                  | 
 hostname        | character(255)           | 
 appname         | character(255)           | 
 proc_id         | character(255)           | 
 msg_id          | character(255)           | 
 structured_data | text                     | 
 message         | text                     | 
Indexes:
    "events_pkey" PRIMARY KEY, btree (id)
    "events_emitted_at" btree (emitted_at)

The rest is up to you.

-- @rwdaigle