redborder / druid_config

A Ruby gem to access the information about Druid cluster status

DruidConfig

DruidConfig is a gem to access the information about Druid cluster status. You can check a node capacity, number of segments, tiers... It uses zookeeper to get coordinator and overlord URIs.

To use in your application, add this line to your Gemfile:

gem 'druid_config'

Query Druid data

The purpose of this gem is to query Druid cluster status. If you want to query Druid data, I recommend you to use ruby-druid gem.

Initialization

Cluster is the base class to perform queries. To initialize it send the Zookeeper URI and options as arguments:

cluster = DruidConfig::Cluster.new(zookeeper_uri, options)

Available options:

• discovery_path: string with the discovery path of druid inside Zookeeper directory structure.

Usage

Call methods defined in DruidConfig::Cluster to access to the data. To get more information about data returned in methods, check Druid documentation.

• leader: leader
• load_status: load status
• load_status: load queue
• metadata_datasources: Hash with metadata of datasources
• metadata_datasources_segments: Hash with metadata of segments
• datasources: all data sources
• datasource: a concrete data source
• rules: all rules defined in the cluster
• tiers: tiers
• servers or nodes: all nodes of the cluster
• physical_servers or physical_nodes: array of URIs of nodes
• historicals: historical nodes
• realtimes: realtime nodes
• workers: worker nodes
• physical_workers: array of URIs of worker nodes
• running_tasks, pending_tasks, waiting_tasks, complete_tasks: tasks in the cluster based in their status
• task: load a task based on an identifier
• services: Hash with physical nodes and the services they are running

Entities

Some methods return an instance of an Entity class. These entities provide multiple methods to access data. Defined entities are inside druid_config/entities folder.

• DataSource
• Segment
• Tier
• Node
• Worker
• Task

Exceptions

DruidConfig::Exceptions::NotAvailableNodes

This exception will be raised when you try to perform a query to a Druid Coordinator or Overlord but there aren't any node of that type available.

DruidConfig::Exceptions::DruidApiError

Sometimes the Gem have available nodes, but it can't access to Druid API. In this case, the gem automatically will reset the Zookeeper connection and retry the query. If second query fails too, a DruidConfig::Exceptions::DruidApiError exception will be raised.

Collaborate

To contribute DruidConfig:

• Create an issue with the contribution: bug, enhancement or feature
• Fork the repository and make all changes you need
• Write test on new changes
• Create a pull request when you finish

License

DruidConfig gem is released under the Affero GPL license. Copyright redBorder

druid_config's Issues

Bug trying to get rules from data source

E, [2015-12-14T10:15:41.698107 #11990] ERROR -- : #<NoMethodError: undefined method `match' for nil:NilClass>
E, [2015-12-14T10:15:41.698699 #11990] ERROR -- : /usr/local/rvm/gems/ruby-2.1.2/gems/iso8601-0.8.7/lib/iso8601/duration.rb:208:in `atomize'
/usr/local/rvm/gems/ruby-2.1.2/gems/iso8601-0.8.7/lib/iso8601/duration.rb:45:in `initialize'
/usr/local/rvm/gems/ruby-2.1.2/gems/druid_config-0.4.0/lib/druid_config/entities/rule.rb:70:in `new'
/usr/local/rvm/gems/ruby-2.1.2/gems/druid_config-0.4.0/lib/druid_config/entities/rule.rb:70:in `initialize'
/usr/local/rvm/gems/ruby-2.1.2/gems/druid_config-0.4.0/lib/druid_config/entities/rule.rb:39:in `new'
/usr/local/rvm/gems/ruby-2.1.2/gems/druid_config-0.4.0/lib/druid_config/entities/rule.rb:39:in `parse'
/usr/local/rvm/gems/ruby-2.1.2/gems/druid_config-0.4.0/lib/druid_config/entities/data_source.rb:85:in `block (2 levels) in rules'
/usr/local/rvm/gems/ruby-2.1.2/gems/httparty-0.13.1/lib/httparty/response.rb:62:in `each'
/usr/local/rvm/gems/ruby-2.1.2/gems/httparty-0.13.1/lib/httparty/response.rb:62:in `method_missing'
/usr/local/rvm/gems/ruby-2.1.2/gems/druid_config-0.4.0/lib/druid_config/entities/data_source.rb:84:in `block in rules'
/usr/local/rvm/gems/ruby-2.1.2/gems/druid_config-0.4.0/lib/druid_config/util.rb:21:in `secure_query'
/usr/local/rvm/gems/ruby-2.1.2/gems/druid_config-0.4.0/lib/druid_config/entities/data_source.rb:83:in `rules'

Improve documentation

We need to create documentation for every entity element and more examples of usage.

Create Rule class

Rules allow the user to define where and how much time data will be stored. These rules are applied to data sources. Check the type of rules and format at Druid docs

We need to create a new class to manage rules. An example of load rule format:

{
  "type" : "loadByInterval",
  "interval": "2012-01-01/2013-01-01",
  "tieredReplicants": {
    "hot": 1,
    "_default_tier" : 1
  }
}

Create RuleCollection class

A collection of data source rules. This method contains an Array of elements (rules) and a method to check consistence of them. For example:

For example, if you define the following rules:

1 - LOAD ByPeriod PT12M replicants -> { _default_tier => 1 }
2 - LOAD ByPeriod PT3M replicants -> { _default_tier => 1 }

The second rule never be applied. The method don't raise any exception and your Druid installation will work unless there are an inconsistence. The purpose of this method is to advise you could be wrong about rules.

Enumerable methods will me delegated to the Array.

Add disable and destroy methods to data source entity

Druid documentation to implement: Coordinator#delete

Improve tests

We must define an architecture to test the gem. The best scenario to test is a real environment, so we can use Docker to create a Druid cluster and perform tests to them