vkhatri / chef-elasticsearch-cluster Goto Github PK

View Code? Open in Web Editor NEW

10.0 4.0 8.0 197 KB

Chef Cookbook to Manage ElasticSearch Cluster

Home Page: https://supermarket.chef.io/cookbooks/elasticsearch-cluster

License: Apache License 2.0

Ruby 58.47% HTML 14.12% Shell 27.40%

chef-elasticsearch-cluster's Introduction

elasticsearch-cluster Cookbook

This is a Chef cookbook to manage ElasticSearch Cluster.

More features and attributes will be added over time, feel free to contribute what you find missing!

For Production environment, always prefer the most recent release.

Dependency

Most Recent Release

cookbook 'elasticsearch-cluster', '~> 0.5.6'

From Git

cookbook 'elasticsearch-cluster', github: 'vkhatri/chef-elasticsearch-cluster',  tag: 'v0.5.6'

Repository

https://github.com/vkhatri/chef-elasticsearch-cluster

Supported OS

This cookbook was tested on Amazon (2015-03) & Ubuntu (14.04) Linux and expected to work on similar platform family OS.

Supported ElasticSearch Version

This cookbook was tested for ElasticSearch v1.6.0+.

Supported ElasticSearch Deployment

This cookbook supports both Package and Tarball based installation.

Cookbook Dependencies

java cookbook
yum cookbook
apt cookbook

TODO

add more locations to nginx vhost
add specs

Recipes

elasticsearch-cluster::default - default recipe (use it for run_list)
elasticsearch-cluster::install - install elasticsearch via package or tarball
elasticsearch-cluster::java - install java using java cookbook
elasticsearch-cluster::package - install elasticsearch using repository package
elasticsearch-cluster::tarball - install elasticsearch using tarball
elasticsearch-cluster::user - create elasticsearch user/group when node['elasticsearch']['install_method'] is set to tarball
elasticsearch-cluster::config - configure elasticsearch
elasticsearch-cluster::plugins - install / remove plugins using node attribute node['elasticsearch']['plugins']
elasticsearch-cluster::scripts - install / remove scripts using node attribute node['elasticsearch']['scripts']

HWRP elasticsearch_plugin

HWRP elasticsearch_plugin manages elasticsearch plugins install / remove. It is a wrapper for plugin binary.

For plugins validation (exists or not), it uses node API call to verify wheter required plugin or plugin version is installed.

version is evaluated to verify the correct installed version. Github plugins version vX.Y.Z is not the same as plugin installed version X.Y.Z discovered by provider. Check provider for more details.

Install Plugin

elasticsearch_plugin 'cloud-aws' do
  install_source   'elasticsearch/elasticsearch-cloud-aws'
  version        '2.7.0'
end

elasticsearch_plugin 'kopf' do
  install_source 'lmenezes/elasticsearch-kopf'
end

Above HWRP resource will install two plugins: cloud-aws and kopf.

Remove Plugin

elasticsearch_plugin 'kopf' do
  action :remove
end

Above HWRP resource will remove the kopf plugin.

HWRP Options

action (optional) - default :install, options: :install, :remove, :nothing
host (optional, String) - elasticsearch host (default: node['elasticsearch']['config']['network.bind_host'])
port (optional, Integer) - elasticsearch port (default: node['elasticsearch']['config']['http.port'])
install_source (optional, String) - plugin source to install, e.g. to install kopf plugin, install_source would be lmenezes/elasticsearch-kopf
url (optional, String) - plugin url
timeout (optional, String) - plugin install timeout
notify_restart (optional, TrueClass, FalseClass) - whether to restart elasticsearch service post plugin install (default: false)
ignore_error (optional, TrueClass,FlaseClass) - whether to ignore error like unable to fetch installed plugins when service is down, could ignore other errors in the future (default: true)
version (optional, String) - plugin install version

HWRP elasticsearch_script

HWRP elasticsearch_script manages elasticsearch scripts under directory node['elasticsearch']['scripts_dir']. It is a wrapper for cookbook_file resource to manage scripts.

Install Script

elasticsearch_script 'foo' do
  cookbook 'cookbook'
  source 'source'
end

Above HWRP resource will add script foo.

Remove Script

elasticsearch_script 'foo' do
  action :delete
end

Above HWRP resource will remove script foo.

HWRP Options

action (optional) - default :add, options: :add, :delete, :nothing
cookbook (optional, String) - script cookbook name for cookbook_file resource, required for action :add
source (optional, String) - default :name, script cookbook file source for cookbook_file resource

Manage Plugins using Node attribute

Using recipe plugins, plugins can be installed or removed by node attribute node['elasticsearch']['plugins']. Below is the role attributes to install and remove plugins.

  "default_attributes": {
    "elasticsearch": {
      "plugins": {
        "bigdesk": {
          "install_source": "lukas-vlcek/bigdesk"
        },
        "kopf": {
          "install_source": "lmenezes/elasticsearch-kopf"
        },
        "head": {
          "action": "remove"
        }
      }
    }
  }

Check out HWRP elasticsearch_plugin or recipe plugins for more info on attributes.

Manage Scripts using Node attribute

Using recipe scripts, scripts can be added or deleted by node attribute node['elasticsearch']['scripts']. Below is the role attributes to add and delete scripts.

  "default_attributes": {
    "elasticsearch": {
      "plugins": {
        "script-name": {
          "cookbook": "foo",
          "source": "foo"
        }
      }
    }
  }

Check out HWRP elasticsearch_script or recipe scripts for more info on attributes.

Cookbook Advanced Attributes

default['elasticsearch']['config']['cluster.name'] (default: nil): elasticsearch cluster name, required to setup elasticsearch cluster
default['elasticsearch']['config_v5']['cluster.name'] (default: nil): elasticsearch cluster name, required to setup elasticsearch cluster
default['elasticsearch']['install_method'] (default: package): elasticsearch install method, options: package tarball
default['elasticsearch']['install_java'] (default: true): whether to install java using cookbook java
default['elasticsearch']['service_action'] (default: [:enable, :start]): elasticsearch service resource action
default['elasticsearch']['notify_restart'] (default: false): whether to notify elasticsearch service on any config change
default['elasticsearch']['setup_user'] (default: true): whether to setup elasticsearch service user when installing via tarball
default['elasticsearch']['cookbook'] (default: elasticsearch-cluster): cookbook to use for elasticsearch configuration file/template source
default['elasticsearch']['plugins'] (default: {}): node Hash attribute to install/remove elasticsearch plugins.
default['elasticsearch']['scripts'] (default: {}): node Hash attribute to install/remove elasticsearch scripts.
default['elasticsearch']['auto_java_memory'] (default: false): whether to allocate maximum possible heap size
default['elasticsearch']['auto_system_memory'] (default: 768): minimum memory to keep for OS while allocating maximum possible memory to elasticsearch, used with node['elasticsearch']['auto_java_memory']
default['elasticsearch']['use_chef_search'] (default: false): whether to use Chef Search to find elasticsearch cluster nodes in an environment for a given chef elasticsearch role
default['elasticsearch']['search_role_name'] (default: elasticsearch_cluster): chef role associated with elasticsearch nodes which add this cookbook to elasticsearch nodes run list
default['elasticsearch']['search_cluster_name_attr'] (default: nil): node attribute to match elasticsearch cluster name, not necessary applicable to all
default['elasticsearch']['tarball_purge'] (default: false): purge older installed versions for tarball install method
default['elasticsearch']['enable_sensitive'] (default: true): enable sensitive resource attribute for configuration resources

Cookbook Core Attributes

default['elasticsearch']['version'] (default: 5.4.2): elasticsearch version to install
default['elasticsearch']['sha256sum'] (default: nil): allow users to provide sha256sum for unsupported elasticsearch version tarball file
default['elasticsearch']['version_suffix'] (default: calculated): elasticsearch package version suffix
default['elasticsearch']['user'] (default: elasticsearch): elasticsearch service user
default['elasticsearch']['group'] (default: elasticsearch): elasticsearch service group
default['elasticsearch']['conf_dir'] (default: /etc/elasticsearch): elasticsearch configuration directory
default['elasticsearch']['data_dir'] (default: /var/lib/elasticsearch): elasticsearch data directory
default['elasticsearch']['log_dir'] (default: /var/log/elasticsearch): elasticsearch log directory
default['elasticsearch']['work_dir'] (default: /tmp/elasticsearch): elasticsearch temporary files directory
default['elasticsearch']['plugins_dir'] (default: calcualted): elasticsearch plugins directory
default['elasticsearch']['conf_file'] (default: /etc/elasticsearch/elasticsearch.yml): elasticsearch configuration file
default['elasticsearch']['logging_conf_file'] (default: /etc/elasticsearch/logging.yml): elasticsearch logging configuration file
default['elasticsearch']['tarball_url'] (default: auto): elasticsearch tarball download url
default['elasticsearch']['tarball_checksum'] (default: versions): elasticsearch tarball version source sha256sum
default['elasticsearch']['parent_dir'] (default: /usr/local/elasticsearch): elasticsearch directory for tarball based installation
default['elasticsearch']['install_dir'] (default: /usr/local/elasticsearch/elasticsearch): elasticsearch symlink to current version source directory (for tarball based installation)
default['elasticsearch']['source_dir'] (default: calculated): elasticsearch current version directory (for tarball based installation)
default['elasticsearch']['home_dir'] (default: calculated): elasticsearch home directory
default['elasticsearch']['sysconfig_file'] (default: calculated): elasticsearch service config file location
default['elasticsearch']['umask'] (default: 0022): file/directory umask for execute resource (for tarball based installation)
default['elasticsearch']['mode'] (default: 0755): file/directory resource mode

Service Attributes

default['elasticsearch']['sysconfig']['ES_HEAP_NEWSIZE'] (default: calculated): elasticsearch service config file parameter
default['elasticsearch']['sysconfig']['ES_DIRECT_SIZE'] (default: nil): elasticsearch service config file parameter
default['elasticsearch']['sysconfig']['ES_JAVA_OPTS'] (default: nil): elasticsearch service config file parameter
default['elasticsearch']['sysconfig']['MAX_OPEN_FILES'] (default: 65_536): elasticsearch service config file parameter
default['elasticsearch']['sysconfig']['MAX_LOCKED_MEMORY'] (default: unlimited): elasticsearch service config file parameter
default['elasticsearch']['sysconfig']['MAX_MAP_COUNT'] (default: 262_144): elasticsearch service config file parameter
default['elasticsearch']['sysconfig']['ES_RESTART_ON_UPGRADE'] (default: true): elasticsearch service config file parameter
default['elasticsearch']['sysconfig']['ES_HEAP_SIZE'] (default: calculated): elasticsearch service config file parameter

ElasticSearch v1.x, 2.x Configuration File Attributes

default['elasticsearch']['config']['cluster.name'] (default: nil): elasticsearch configuration parameter
default['elasticsearch']['config']['node.name'] (default: node.name): elasticsearch configuration parameter
default['elasticsearch']['config']['node.master'] (default: true): elasticsearch configuration parameter
default['elasticsearch']['config']['node.data'] (default: true): elasticsearch configuration parameter
default['elasticsearch']['config']['node.max_local_storage_nodes'] (default: 1): elasticsearch configuration parameter
default['elasticsearch']['config']['index.number_of_shards'] (default: 5): elasticsearch configuration parameter
default['elasticsearch']['config']['index.number_of_replicas'] (default: 1): elasticsearch configuration parameter
default['elasticsearch']['config']['path.conf'] (default: node['elasticsearch']['conf_dir']): elasticsearch configuration parameter
default['elasticsearch']['config']['path.data'] (default: node['elasticsearch']['data_dir']): elasticsearch configuration parameter
default['elasticsearch']['config']['path.work'] (default: node['elasticsearch']['work_dir']): elasticsearch configuration parameter
default['elasticsearch']['config']['path.logs'] (default: node['elasticsearch']['log_dir']): elasticsearch configuration parameter
default['elasticsearch']['config']['path.plugins'] (default: node['elasticsearch']['plugins_dir']): elasticsearch configuration parameter
default['elasticsearch']['config']['bootstrap.mlockall'] (default: true): elasticsearch configuration parameter
default['elasticsearch']['config']['network.bind_host'] (default: node['ipaddress']): elasticsearch configuration parameter
default['elasticsearch']['config']['network.publish_host'] (default: node['ipaddress']): elasticsearch configuration parameter
default['elasticsearch']['config']['transport.tcp.port'] (default: 9300): elasticsearch configuration parameter
default['elasticsearch']['config']['transport.tcp.compress'] (default: true): elasticsearch configuration parameter
default['elasticsearch']['config']['http.port'] (default: 9200): elasticsearch configuration parameter
default['elasticsearch']['config']['discovery.zen.minimum_master_nodes'] (default: 1): elasticsearch configuration parameter
default['elasticsearch']['config']['discovery.zen.ping.timeout'] (default: 3s): elasticsearch configuration parameter
default['elasticsearch']['config']['discovery.zen.ping.multicast.enabled'] (default: false): elasticsearch configuration parameter
default['elasticsearch']['config']['discovery.zen.ping.unicast.hosts'] (default: []): elasticsearch configuration parameter
default['elasticsearch']['config']['action.auto_create_index'] (default: true): elasticsearch configuration parameter
default['elasticsearch']['config']['action.disable_delete_all_indices'] (default: true): elasticsearch configuration parameter
default['elasticsearch']['config']['script.disable_dynamic'] (default: true): elasticsearch configuration parameter
default['elasticsearch']['config']['script.inline'] (default: script.inline): elasticsearch configuration parameter
default['elasticsearch']['config']['script.indexed'] (default: script.indexed): elasticsearch configuration parameter
default['elasticsearch']['config']['script.file'] (default: on): elasticsearch configuration parameter

ElasticSearch v5.x Configuration File Attributes

default['elasticsearch']['config_v5']['cluster.name'] (default: nil): elasticsearch configuration parameter
default['elasticsearch']['config_v5']['node.name'] (default: node.name): elasticsearch configuration parameter
default['elasticsearch']['config_v5']['node.master'] (default: true): elasticsearch configuration parameter
default['elasticsearch']['config_v5']['node.data'] (default: true): elasticsearch configuration parameter
default['elasticsearch']['config_v5']['node.ingest'] (default: true): elasticsearch configuration parameter
default['elasticsearch']['config_v5']['node.attr.rack'] (default: nil): elasticsearch configuration parameter
default['elasticsearch']['config_v5']['node.attr.dc'] (default: nil): elasticsearch configuration parameter
default['elasticsearch']['config_v5']['node.attr.size'] (default: nil): elasticsearch configuration parameter
default['elasticsearch']['config_v5']['path.data'] (default: nil): elasticsearch configuration parameter
default['elasticsearch']['config_v5']['path.logs'] (default: /var/log/elasticsearch): elasticsearch configuration parameter
default['elasticsearch']['config_v5']['bootstrap.memory_lock'] (default: true): elasticsearch configuration parameter
default['elasticsearch']['config_v5']['network.host'] (default: node['ipaddress']): elasticsearch configuration parameter
default['elasticsearch']['config_v5']['transport.tcp.port'] (default: 9300): elasticsearch configuration parameter
default['elasticsearch']['config_v5']['transport.tcp.compress'] (default: true): elasticsearch configuration parameter
default['elasticsearch']['config_v5']['http.port'] (default: 9200): elasticsearch configuration parameter
default['elasticsearch']['config_v5']['discovery.zen.ping.unicast.hosts'] (default: []): elasticsearch configuration parameter
default['elasticsearch']['config_v5']['discovery.zen.minimum_master_nodes'] (default: 1): elasticsearch configuration parameter
default['elasticsearch']['config_v5']['discovery.zen.fd.ping_timeout'] (default: 3s): elasticsearch configuration parameter
default['elasticsearch']['config_v5']['node.max_local_storage_nodes'] (default: 1): elasticsearch configuration parameter
default['elasticsearch']['config_v5']['action.destructive_requires_name'] (default: true): elasticsearch configuration parameter
default['elasticsearch']['config_v5']['action.auto_create_index'] (default: true): elasticsearch configuration parameter
default['elasticsearch']['config_v5']['script.inline'] (default: false): elasticsearch configuration parameter
default['elasticsearch']['config_v5']['script.stored'] (default: false): elasticsearch configuration parameter
default['elasticsearch']['config_v5']['script.file'] (default: true): elasticsearch configuration parameter

ElasticSearch Configuration From Data Bag Item

default['elasticsearch']['databag_configs'] (default: nil): elasticsearch configuration parameters from data bag
- For example, following configuration will get data bag named 'aws_keys' and data bag item 'elasticsearch'.
- The key of 'config_items' will be used as configuration name.
- The value of 'config_items' will be used for fetching value from data bag.

'elasticsearch' => {
    'databag_configs' => [
      {
        'name' => 'aws_keys',
        'item' => 'elasticsearch',
        'config_items' => {
          'cloud.aws.access_key' => 'data_bag_key_for_access_key',
          'cloud.aws.secret_key' => 'data_bag_key_for_secret_key'
        }
      }
    ]
}

Elasticsearch YUM/APT Repository Attributes

default['elasticsearch']['yum']['description'] (default: calculated): elasticsearch yum reporitory attribute
default['elasticsearch']['yum']['gpgcheck'] (default: true): elasticsearch yum reporitory attribute
default['elasticsearch']['yum']['enabled'] (default: true): elasticsearch yum reporitory attribute
default['elasticsearch']['yum']['baseurl'] (default: calculated): elasticsearch yum reporitory attribute
default['elasticsearch']['yum']['gpgkey'] (default: https://packages.elasticsearch.org/GPG-KEY-elasticsearch): elasticsearch yum reporitory attribute
default['elasticsearch']['yum']['action'] (default: :create): elasticsearch yum reporitory attribute
default['elasticsearch']['apt']['description'] (default: calculated): elasticsearch apt reporitory attribute
default['elasticsearch']['apt']['components'] (default: ['stable', 'main']): elasticsearch apt reporitory attribute
default['elasticsearch']['apt']['uri'] (default: calculated): elasticsearch apt reporitory attribute
default['elasticsearch']['apt']['key'] (default: http://packages.elasticsearch.org/GPG-KEY-elasticsearch): elasticsearch apt reporitory attribute
default['elasticsearch']['apt']['action'] (default: :add): elasticsearch apt reporitory attribute

ElasticSearch v5.x Configuration File Attributes

default['elasticsearch']['logging']['status'] (default: error): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['logger.action.name'] (default: org.elasticsearch.action): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['logger.action.level'] (default: debug): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.console.type'] (default: Console): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.console.name'] (default: console): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.console.layout.type'] (default: PatternLayout): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.console.layout.pattern'] (default: [%d{ISO8601}][%-5p][%-25c{1.}] %marker%m%n): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.rolling.type'] (default: RollingFile): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.rolling.name'] (default: rolling): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.rolling.fileName'] (default: ${sys:es.logs}.log): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.rolling.layout.type'] (default: PatternLayout): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.rolling.layout.pattern'] (default: [%d{ISO8601}][%-5p][%-25c{1.}] %marker%.-10000m%n): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.rolling.filePattern'] (default: ${sys:es.logs}-%d{yyyy-MM-dd}.log): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.rolling.policies.type'] (default: Policies): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.rolling.policies.time.type'] (default: TimeBasedTriggeringPolicy): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.rolling.policies.time.interval'] (default: 1): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.rolling.policies.time.modulate'] (default: true): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['rootLogger.level'] (default: info): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['rootLogger.appenderRef.console.ref'] (default: console): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['rootLogger.appenderRef.rolling.ref'] (default: rolling): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.deprecation_rolling.type'] (default: RollingFile): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.deprecation_rolling.name'] (default: deprecation_rolling): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.deprecation_rolling.fileName'] (default: ${sys:es.logs}_deprecation.log): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.deprecation_rolling.layout.type'] (default: PatternLayout): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.deprecation_rolling.layout.pattern'] (default: [%d{ISO8601}][%-5p][%-25c{1.}] %marker%.-10000m%n): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.deprecation_rolling.filePattern'] (default: ${sys:es.logs}_deprecation-%i.log.gz): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.deprecation_rolling.policies.type'] (default: Policies): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.deprecation_rolling.policies.size.type'] (default: SizeBasedTriggeringPolicy): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.deprecation_rolling.policies.size.size'] (default: 1GB): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.deprecation_rolling.strategy.type'] (default: DefaultRolloverStrategy): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.deprecation_rolling.strategy.max'] (default: 1): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['logger.deprecation.name'] (default: org.elasticsearch.deprecation): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['logger.deprecation.level'] (default: warn): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['logger.deprecation.appenderRef.deprecation_rolling.ref'] (default: deprecation_rolling): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['logger.deprecation.additivity'] (default: false): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.index_search_slowlog_rolling.type'] (default: RollingFile): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.index_search_slowlog_rolling.name'] (default: index_search_slowlog_rolling): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.index_search_slowlog_rolling.fileName'] (default: ${sys:es.logs}_index_search_slowlog.log): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.index_search_slowlog_rolling.layout.type'] (default: PatternLayout): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.index_search_slowlog_rolling.layout.pattern'] (default: [%d{ISO8601}][%-5p][%-25c] %marker%.-10000m%n): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.index_search_slowlog_rolling.filePattern'] (default: ${sys:es.logs}_index_search_slowlog-%d{yyyy-MM-dd}.log): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.index_search_slowlog_rolling.policies.type'] (default: Policies): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.index_search_slowlog_rolling.policies.time.type'] (default: TimeBasedTriggeringPolicy): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.index_search_slowlog_rolling.policies.time.interval'] (default: 1): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.index_search_slowlog_rolling.policies.time.modulate'] (default: true): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['logger.index_search_slowlog_rolling.name'] (default: index.search.slowlog): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['logger.index_search_slowlog_rolling.level'] (default: trace): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['logger.index_search_slowlog_rolling.appenderRef.index_search_slowlog_rolling.ref'] (default: index_search_slowlog_rolling): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['logger.index_search_slowlog_rolling.additivity'] (default: false): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.index_indexing_slowlog_rolling.type'] (default: RollingFile): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.index_indexing_slowlog_rolling.name'] (default: index_indexing_slowlog_rolling): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.index_indexing_slowlog_rolling.fileName'] (default: ${sys:es.logs}_index_indexing_slowlog.log): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.index_indexing_slowlog_rolling.layout.type'] (default: PatternLayout): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.index_indexing_slowlog_rolling.layout.pattern'] (default: [%d{ISO8601}][%-5p][%-25c] %marker%.-10000m%n): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.index_indexing_slowlog_rolling.filePattern'] (default: ${sys:es.logs}_index_indexing_slowlog-%d{yyyy-MM-dd}.log): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.index_indexing_slowlog_rolling.policies.type'] (default: Policies): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.index_indexing_slowlog_rolling.policies.time.type'] (default: TimeBasedTriggeringPolicy): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.index_indexing_slowlog_rolling.policies.time.interval'] (default: 1): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['appender.index_indexing_slowlog_rolling.policies.time.modulate'] (default: true): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['logger.index_indexing_slowlog.name'] (default: index.indexing.slowlog.index): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['logger.index_indexing_slowlog.level'] (default: trace): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['logger.index_indexing_slowlog.appenderRef.index_indexing_slowlog_rolling.ref'] (default: index_indexing_slowlog_rolling): elasticsearch 5.x log4j configuration parameter
default['elasticsearch']['logging']['logger.index_indexing_slowlog.additivity'] (default: false): elasticsearch 5.x log4j configuration parameter

Contributing

Fork the repository on Github
Create a named feature branch (like add_component_x)
Write your change
Write tests for your change (if applicable)
Run the tests (rake & rake knife), ensuring they all pass
Write new resource/attribute description to README.md
Write description about changes to PR
Submit a Pull Request using Github

Copyright & License

Authors:: Virender Khatri and Contributors

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

chef-elasticsearch-cluster's People

Contributors

Stargazers

Watchers

Forkers

michaelklishin shortdudey123 automatedops ramyakailas bap-pa claudiouv sumit0k balazsadam

chef-elasticsearch-cluster's Issues

update sysv init templates for v1.7.2 bug fixes

v1.7.2 has major bug fixes for sysv init scripts.

allocate half of the available memory to ES by default

set attribute default['elasticsearch']['sysconfig']['ES_HEAP_SIZE'] with half of the total memory as per ES guidelines.

https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html

add attribute ignore_error to plugin resource

set ignore_error default value to true

this will ignore error while ES service is down to fetch installed plugins

move common directory resources to recipe install

bump elasticsearch version to 1.7.0

test with wrapper cookbook for default attributes update

cookbook was tested with role/environment for attributes update, but have not tested when using wrapper cookbook to update default attributes

update elasticsearch service resource to delayed start

update init.d scripts resource type from template to cookbook_file

init.d scripts are static files and parameters updates are solely managed via env config file.

there are some changes in the init.d scripts (for v1 & v2), maintaining same initd script shipped with package sound reasonable than patching in the cookbook.

will have separate initd scripts for v1 and v2.

bump elasticsearch version to 1.7.1

$ sha256sum elasticsearch-1.7.1.tar.gz.1
86a0c20eea6ef55b14345bff5adf896e6332437b19180c4582a346394abde019 elasticsearch-1.7.1.tar.gz.1

move install recipe directory resources to install method recipe

Recipe: elasticsearch-cluster::install
  * directory[/mnt/elasticsearch] action create
    * cannot determine user id for 'elasticsearch', does the user exist on this system?
    ================================================================================
    Error executing action `create` on resource 'directory[/mnt/elasticsearch]'
    ================================================================================

    Chef::Exceptions::UserIDNotFound
    --------------------------------
    cannot determine user id for 'elasticsearch', does the user exist on this system?

    Resource Declaration:
    ---------------------
    # In /var/chef/cache/cookbooks/elasticsearch-cluster/recipes/install.rb

     28:   directory dir do
     29:     owner node['elasticsearch']['user']
     30:     group node['elasticsearch']['group']
     31:     mode node['elasticsearch']['mode']
     32:     recursive true
     33:   end
     34: end

    Compiled Resource:
    ------------------
    # Declared in /var/chef/cache/cookbooks/elasticsearch-cluster/recipes/install.rb:28:in `block in from_file'

    directory("/mnt/elasticsearch") do
      action [:create]
      retries 0
      retry_delay 2
      default_guard_interpreter :default
      path "/mnt/elasticsearch"
      recursive true
      declared_type :directory
      cookbook_name "elasticsearch-cluster"
      recipe_name "install"
      owner "elasticsearch"
      group "elasticsearch"
      mode "0755"
    end
111

plugin syntax changed in es 2.x

link node['elasticsearch']['install_dir'] should always notify service restart

plugin version is not the same as installed version

plugin kopf resource version:

v1.5.5

provider discovered version:

{
  "bigdesk": "NA",
  "kopf": "1.5.5"
}

kopf and bigdesk are always installed

When I have node attributes that look like this:

          plugins: {
            "cloud-aws" => {
              "install_name" => "elasticsearch/elasticsearch-cloud-aws"
            }
          }

the cookbook still installs kopf and bigdesk. I don't want any of that.

release new cookbook version to supermarket

manage default path.scripts directory

exception in v2.0.0-beta2

java.lang.IllegalStateException: Unable to access 'path.scripts' (/etc/elasticsearch/scripts)
    at org.elasticsearch.bootstrap.Security.addPath(Security.java:204)
    at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:169)
    at org.elasticsearch.bootstrap.Security.configure(Security.java:101)
    at org.elasticsearch.bootstrap.Bootstrap.setupSecurity(Bootstrap.java:179)
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:157)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:266)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/scripts
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
    at java.nio.file.Files.createDirectory(Files.java:674)
    at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
    at java.nio.file.Files.createDirectories(Files.java:767)
    at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:225)
    at org.elasticsearch.bootstrap.Security.addPath(Security.java:202)
    ... 6 more

Plugin installer requires ES to be running

Plugin installation will fail when the service isn't running because the provider tries to query node info for the list of plugins. I don't know if this really can be avoided but it can lead to pretty obscure issues and is quite confusing.

move tarball checksum to helper method

publish new cookbook v0.2.7

add attribute notify_restart to plugin resource

this attribute will trigger ES service restart

ES_JAVA_OPTS needs to be quoted

udpate tarball url to download.elasticsearch.org

more appropriate name for resource plugin attribute install_name

Considering changes introduced in ES v2.x plugin binary, changing attribute name install_name to install_source seems more appropriate.

This breaks backward compatibility, will add a note to section Major Changes.

default['elasticsearch']['config']['script.disable_dynamic'] is not supported for v2

es 2.0 invalid parameter CONF_FILE

CONF_FILE setting is no longer supported. elasticsearch.yml must be placed in the config directory and cannot be renamed.

This parameter is valid for version < 2.x

setup kitchen

add config attribute to pick elasticsearch.yml from another attribute

e.g. default['elasticsearch']['config_attribute'] = 'config'

manage logging.yml configuration file

due to missing logging.yml file tarball install_method does not create log files.

log4j:WARN No appenders could be found for logger (node).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

make sensitive optional for elasticsearch.yml

making it optional, true by default. would be a great help for why run mode when making new config changes.

add feature to purge old tarball es revisions

add hwrp resource for scripts

hwrp to manage .groovy scripts using cookbook_file

Consider using directory-based plugin detection

Currently this cookbook tries to use bind_host to contact ES node to fetch installed plugin list. This has plenty of edge cases that are completely ignored:

When deploying an AMI build with Packer and chef-solo, bind_host will be a local domain on first Chef run.
The value can be 0.0.0.0
ElasticSearch supports special values that are not valid IP addresses at all

I think the right thing to do would be to use directory-driven plugin detection instead of trying to contact the node.

plugins dir should be created under install_dir for tarball installation

add attribute host & port to plugin resource

es 2.x repo url change

[elasticsearch-2.x]
name=Elasticsearch repository for 2.x packages
baseurl=http://packages.elastic.co/elasticsearch/2.x/centos
gpgcheck=1
gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1

bump es version to 2.0.0

add ES v2.0 support

manage es config dir in favor of tarball install method

  * template[/etc/elasticsearch/elasticsearch.yml] action create
    * Parent directory /etc/elasticsearch does not exist.
    ================================================================================
    Error executing action `create` on resource 'template[/etc/elasticsearch/elasticsearch.yml]'
    ================================================================================

    Chef::Exceptions::EnclosingDirectoryDoesNotExist
    ------------------------------------------------
    Parent directory /etc/elasticsearch does not exist.

    Resource Declaration:
    ---------------------
    suppressed sensitive resource output

    Compiled Resource:
    ------------------
    suppressed sensitive resource output

add attribute default['elasticsearch']['bin_dir']

package install needs to manage ES data, work and log dir

elasticsearch-cluster::default's implicit dependency on elasticsearch-cluster::user

When installation method is package, there is an implicit dependency on having the user created.

I've noticed that elasticsearch-cluster::default does not include elasticsearch-cluster::user. This is easy to work around by including the user recipe into the run list but is this how things are intended to be used?

If so, I don't think it's documented.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.