netflix / raigad

Co-Process for backup/recovery, Auto Deployments and Centralized Configuration management for ElasticSearch

Raigad is a process/tool that runs alongside Elasticsearch to automate the following:

• Snapshot backup and restore.
• Configured deployments for a dedicated master/data/search approach.
• Tribe node deployments.
• Publishing Elasticsearch monitoring metrics.
• Support for AWS environment.

Elasticsearch is open source, distributed, real-time search and analytics engine: https://www.elastic.co/products/elasticsearch/

The name "Raigad" refers to the name of an Indian fort in state of Maharashtra.

Raigad is actively developed and used at Netflix.

Features:

• Support multi-region Elasticsearch deployment in AWS via public IP.
• Automated security group update in multi-region environment.
• REST APIs for backup/restore and other operations.

raigad's Issues

integrate into https://github.com/Netflix-Skunkworks/zerotodocker

would be great to try this out easily with docker. Maybe your teams could collaborate?

Build issue

Running into build issues with Raigad. Build fails while executing test cases present in TestElasticsearchConfig. Does your test cases require Cassandra host to be up at the time of build? I am running elasticsearch 1.4 on ubuntu.

Here is the stack trace>>

com.netflix.raigad.resources.TestElasticsearchConfig > getNodes_handlesUnknownHostException FAILED
java.lang.IllegalStateException at TestElasticsearchConfig.java:80

31 tests completed, 1 failed
:raigad:test FAILED

FAILURE: Build failed with an exception.

How well do tribe nodes work?

@sagarl How are tribe nodes working for you? Have you found any gotchas? Do you have any learnings about distributed scoring across clusters?

We're looking into tribe nodes at GitHub and Netflix is the only company I know of using them in a production environment.

/cc @TwP

raigad-es-extensions usage

@sagarl :

i'm running following services locally

1)Eureka ###(spring-cloud-starter-eureka)
2) elasticsearch
3) Nodejs app (my web app)

i want to get the info from elasticsearch from nodejs app, i'm able to register nodejs app with eureka, but seems there is no plugin for elasticsearch to register with eureka except Raigad.

can we use raigad-es-extensions directly (copying to $ELASTICSEARCH_HOME/lib) and elasticsearch registers with Eureka

aws is to much for now to me

Help is appreciated!!

Add support for Search Guard

Add support for Search Guard (https://github.com/floragunncom/search-guard)

Elasticsearch 1.4+ Support

I understand Raigad Supports 1.3.2, are there any plans to support Elasticsearch 1.4+

Compatibility?

I've checked the wiki, and it says 1.3.2. Is that still the case?

Guice creation errors

When attempting to run the latest raigad-web, I am encountering the following problem when deploying Raiged.war:

14 errors
com.google.inject.CreationException: Guice creation errors:

1) Error injecting constructor, java.lang.NullPointerException
  at com.netflix.raigad.configuration.RaigadConfiguration.<init>(RaigadConfiguration.java:299)
  at com.netflix.raigad.configuration.RaigadConfiguration.class(RaigadConfiguration.java:36)
  while locating com.netflix.raigad.configuration.RaigadConfiguration
  while locating com.netflix.raigad.configuration.IConfiguration
    for field at com.netflix.raigad.identity.CassandraInstanceFactory.config(CassandraInstanceFactory.java:30)
  at com.netflix.raigad.identity.CassandraInstanceFactory.class(CassandraInstanceFactory.java:30)
  while locating com.netflix.raigad.identity.CassandraInstanceFactory
  while locating com.netflix.raigad.identity.IRaigadInstanceFactory
Caused by: java.lang.NullPointerException
    at com.netflix.raigad.configuration.RaigadConfiguration.getDefaultCredentialProvider(RaigadConfiguration.java:853)
    at com.netflix.raigad.configuration.RaigadConfiguration.<init>(RaigadConfiguration.java:221)
    at com.netflix.raigad.configuration.RaigadConfiguration$$FastClassByGuice$$a6937af9.newInstance(<generated>)
    at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)

The issue appears to be due to the code on line 221 (and the similar initializations below that):

private final DynamicStringProperty CREDENTIAL_PROVIDER = DynamicPropertyFactory.getInstance().getStringProperty(CONFIG_CREDENTIAL_PROVIDER, getDefaultCredentialProvider());

This code seems to get executed before the constructor initializes the config field with the injected IConfigSource. I have debugged the dependency injection and it does appear that the injector has indeed located an instance of the default RaigadConfigSource and has passed it to the RaigadConfiguration constructor. So if the code on line 221 was evaluated after the class construction things would be fine. But this appears to happen out of order, hence the NPE.

Any ideas what I'm doing wrong or if this is a known issue?

.travis.yml: The 'sudo' tag is now deprecated in Travis CI

Travis are now recommending removing the sudo tag.

"If you currently specify sudo: false in your .travis.yml, we recommend removing that configuration"

Compatibility with ES5 and non AWS Clusters

Hi

I found in the wiki that Raigad does only support ES 1.5.2. Are there any plans to support newer versions?

And does it work with clusters which aren't hosted on AWS too. I'm looking for the functionalities "backup and restore" , "Automated Index Management" and "monitoring"

Thanks for your help