jbarnette / pinglish

A simple Rack middleware for checking application health.

Pinglish

A simple Rack middleware for checking application health. Pinglish exposes a /_ping resource via HTTP GET, returning JSON that conforms to the spec below.

The Spec

0. The application must respond to GET /_ping as an HTTP request.

1. The request handler should check the health of all services the application depends on, answering questions like, "Can I query against my MySQL database," "Can I create/read keys in Redis," or "How many docs are in my ElasticSearch index?"

2. The response must return within 29 seconds. This is one second less than the default timeout for many monitoring services.

3. The response must return an HTTP 200 OK status code if all health checks pass.

4. The response must return an HTTP 503 SERVICE UNAVAILABLE status code if any health checks fail.

5. The response must be of Content-Type application/json; charset=UTF-8.

6. The response must be valid JSON no matter what, even if JSON serialization or other fundamental code fails.

7. The response must contain a "status" key set either to "ok" or "failures".

8. The response must contain a "now" key set to the current server's time in seconds since epoch as a string.

9. If the "status" key is set to "failures", the response may contain a "failures" key set to an Array of string names representing failed checks.

10. If the "status" key is set to "failures", the response may contain a "timeouts" key set to an Array of string names representing checks that exceeded an implementation-specific individual timeout.

11. The response body may contain any other top-level keys to supply additional data about services the application consumes, but all values must be strings, arrays of strings, or hashes where both keys and values are strings.

An Example Response

{

  // These two keys will always exist.

  "now": "1359055102",
  "status": "failures",

  // This key may only exist when a named check has failed.

  "failures": ["db"],

  // This key may only exist when a named check exceeds its timeout.

  "timeouts": ["really-long-check"],

  // Keys like this may exist to provide extra information about
  // healthy services, like the number of objects in an S3 bucket.

  "s3": "127"
}

The Middleware

require "pinglish"

use Pinglish do |ping|

  # A single unnamed check is the simplest possible way to use
  # Pinglish, and you'll probably never want combine it with other
  # named checks. An unnamed check contributes to overall success or
  # failure, but never adds additional data to the response.

  ping.check do
    App.healthy?
  end

  # A named check like this can provide useful summary information
  # when it succeeds. In this case, a top-level "db" key will appear
  # in the response containing the number of items in the database. If
  # a check returns nil, no key will be added to the response.

  ping.check :db do
    App.db.items.size
  end

  # By default, checks time out after one second. You can override
  # this with the :timeout option, but be aware that no combination of
  # checks is ever allowed to exceed the overall 29 second limit.

  ping.check :long, :timeout => 5 do
    App.dawdle
  end

  # Signal check failure by raising an exception. Any exception will do.
  ping.check :fails do
    raise "Everything's ruined."
  end

  # Additionally, any check that returns false is counted as a failure.
  ping.check :false_fails do
    false
  end
end

pinglish's Issues

May return 404 to be intentionally excluded from load balancing

Hey, nice project here. I saw some tweeting between @qrush and @jbarnette, and the idea of some standardization of health check conventions appeals to me.

I'd like to propose an addition to the spec:

• The response may return a 404 to indicate that it should be intentionally excluded from upstream load balancers.

This is based on HAProxy's http_check_disable_on_404 option, used with its httpchk.

When this option is set, a server which returns an HTTP code 404 will be excluded from further load-balancing, but will still receive persistent connections.

Essentially, ‘nothing is failing, but stop sending traffic until indicated otherwise.’ This is a convention I use a lot with my own apps for graceful shutdowns and restarts.

Not quite sure how to fit that in with the existing syntax and concepts. I'll have to play around with Pinglish a bit more before I have a more concrete idea there.

Perhaps a ping.enabled analogous to ping.check?

Public release

• extract tests
• check a few ruby versions
• expand the README

Regarding failures and timeouts

With regards to the following in the spec:

1. If the "status" key is set to "fail", the response may
   contain a "failures" key set to an Array of string names
   representing failed checks.
2. If the "status" key is set to "fail", the response may
   contain a "timeouts" key set to an Array of string names
   representing checks that exceeded an implementation-specific
   individual timeout.

Wouldn't it be more straightforward to, rather than include a "timeouts" key, just include the keys that failed and their value the corresponding failure message (for which a timeout would be considered a failure message)?

I apologize, I do not remember the term for this type of "crawling" a dataset, but it proves very useful for those receiving the response and provides a clear mechanism to act on messages without a complex decision set based on the spec's rules.

I am not sure I am being clear, so perhaps this example will explain what it is I am trying to say:

{
  "status": "failures",
  "failures": ["db","s3"],
  "db": "timed out",
  "s3": "Failed to determine number of objects",
  "redis": "Hi Mom!"
}

Suggestion for additional endpoints

❤️ this so much. I'd like to propose a couple extra endpoints I use in other apps. Some of them are in use in github/github as well:

• /sha — Return the SHA1 that's currently running.
• /ruby — Returns the RUBY_VERSION that's currently running.
• /boomtown — Raise and exception, and return a 500. Useful for testing that the exception reporting mechanism works.

/cc @jbarnette

Include Rails examples

Trying to add pinglish to a Rails app, but there doesn't seem to be a 'preferred' way of adding. Methods I've seen:

• in config.ru, like any Rack app
• in config/application.rb using config.middleware.use Pinglish
• in config/initializers/pinglish.rb using config.middleware.insert_after ActionDispatch::Static, Pinglish

I think the way I'd prefer to use it is like:

• mount Pinglish in config/routes.rb, since that is where you expect to find URLs)
• configure checks in config/initializers/pinglish.rb, since that is where you'd expect to configure things

I'm just not sure if you'd be able to get at the Pinglish instance in an initializer after you've created it in routes.rb.

A minor benefit of using mount, is you'd be able to mount Pinglish, as: 'ping' to have it create a ping_url helper.

Takes over 500 page in Sinatra development

When working on a Sinatra project, whenever I do something that causes a 500 error, instead of getting the Sinatra exception page with the backtrace I get a pinglish failed response, i.e. { "status": "fail", "now": "1366307578" }

Include example of class based pinglish

I was building a pinglish check, and wanted to include more application logic, but was worried about including it in config/application.rb. I figured out a way to include this in app/middleware as it's own class, as well as a way to test it.

Here's the example:

# app/middleware/fancy_pinglish.rb
class FancyPinglish < Pinglish

  def initialize(app, options = nil, &block)
    super

    check :fancy do
      # logic here
    end

    check :exquisite do
      # logic here
    end

    check :epic do
      # logic
    end
  end

end

# config/application.rb
config.middleware.use "FancyPinglish", max: 15, path: '/_ping/fancy'

# spec/middleware/fancy_pinglish_spec.rb
describe FancyPinglish, type: :request do

  context "when all is clear" do
    before do
      # stub logic
    end

    it "returns 200" do
      get "/_ping/fancy"

      expect(response.status).to eq(200)
    end
  end

  context "when pagerduty is failing" do
    before do
      # stub logic
    end

    it "returns 503" do
      get "/_ping/fancy"

      expect(response.status).to eq(503)
    end
  end
  # other variations here
end