github / vulcanizer

GitHub's ops focused Elasticsearch library

vulcanizer

GitHub's ops focused Elasticsearch library

This project is a golang library for interacting with an Elasticsearch cluster. It's goal is to provide a high level API to help with common tasks that are associated with operating an Elasticsearch cluster such as querying health status of the cluster, migrating data off of nodes, updating cluster settings, etc.

This project does not aim to be a fully-featured API client for querying or indexing to Elasticsearch.

Go API

You can perform custom operations in your Go application.

import "github.com/github/vulcanizer"

v = vulcanizer.NewClient("localhost", 9200)
oldSetting, newSetting, err := v.SetSetting("indices.recovery.max_bytes_per_sec", "1000mb")

Command line application

This project produces a vulcanizer binary that is a command line application that can be used to manage your Elasticsearch cluster.

$ vulcanizer help
Usage:
  vulcanizer [command]

Available Commands:
  aliases         Interact with aliases of the cluster.
  allocation      Set shard allocation on the cluster.
  analyze         Analyze text given an analyzer or a field and index.
  drain           Drain a server or see what servers are draining.
  fill            Fill servers with data, removing shard allocation exclusion rules.
  health          Display the health of the cluster.
  heap            Display the node heap stats.
  help            Help about any command
  hotthreads      Display the current hot threads by node in the cluster.
  indices         Display the indices of the cluster.
  mappings        Display the mappings of the specified index.
  nodeallocations Display the nodes of the cluster and their disk usage/allocation.
  nodes           Display the nodes of the cluster.
  repository      Interact with the configured snapshot repositories.
  setting         Interact with cluster settings.
  settings        Display all the settings of the cluster.
  shards          Get shard data by cluster node(s).
  snapshot        Interact with a specific snapshot.

Flags:
      --cacert string       Path to the certificate to check the cluster certificates against
      --cert string         Path to the certificate to use for client certificate authentication
  -c, --cluster string      Cluster to connect to defined in config file
  -f, --configFile string   Configuration file to read in (default to "~/.vulcanizer.yaml")
  -h, --help                help for vulcanizer
      --host string         Host to connect to (default "localhost")
      --key string          Path to the key to use for client certificate authentication
      --password string     Password to use during authentication
      --path string         Path to prepend to queries, in case Elasticsearch is behind a reverse proxy
  -p, --port int            Port to connect to (default 9200)
      --protocol string     Protocol to use when querying the cluster. Either 'http' or 'https'. Defaults to 'http' (default "http")
  -k, --skipverify string   Skip verifying server's TLS certificate. Defaults to 'false', ie. verify the server's certificate (default "false")
      --user string         User to use during authentication

Use "vulcanizer [command] --help" for more information about a command.

Roadmap and future releases

The proposed future for vulcanizer can be found in our ROADMAP.

Configuration and connection information

All commands take --cluster <name> to look up information in a configuration file in ~/.vulcanizer.yaml. The configuration should be in the form of

local:
  host: localhost
  port: 9200
staging:
  host: 10.10.2.1
  port: 9201
production:
  host: 10.10.1.1
  port: 9202

Alternatively, all commands take --host and --port for the connection information.

For example:

# Query for cluster health on the "local" cluster
vulcanizer health --cluster local

# Query for nodes against the node 10.10.2.1 and port 9202
vulcanizer nodes --host 10.10.2.1 --port 9202

Development

./script/build will compile the project and install the vulcanizer binary to $GOPATH/bin.

./script/test will run the tests in the project.

Supported Elasticsearch versions

Integration tests are set up to run against the latest v5 and v6 versions of Elasticsearch.

Name

Vulcanization is the process of making rubber more elastic, so vulcanizer is the library that makes Elasticsearch easier to work with!

Project status

This project is under active development.

Contributing

This repository is open to contributions. Please also see code of conduct

To get up and running, install the project into your $GOPATH and run the set up scripts.

go get github.com/github/vulcanizer

cd $GOPATH/src/github.com/github/vulcanizer

./script/bootstrap
./script/test

And the test suite should execute correctly.

License

This project is released under the MIT LICENSE. Please note it includes 3rd party dependencies release under their own licenses; dependencies are listed in our go.mod file. When using the GitHub logos, be sure to follow the GitHub logo guidelines.

Authors

Authored by GitHub Engineering

vulcanizer's Issues

Snapshot list show incorrect finished date while running a snapshot

I've been trying the snapshot functionallity of vulcanizer but when I run list command while a snapshot process is running the FINISHED and DURATION columns show this information:

$ vulcanizer snapshot -c c0 list -r r0
+-------------+-----------+----------------------+-------------------+
|    STATE    |   NAME    |       FINISHED       |     DURATION      |
+-------------+-----------+----------------------+-------------------+
| IN_PROGRESS | test      | 1970-01-01T00:00:00Z | -431225h3m27.241s |
+-------------+-----------+----------------------+-------------------+

If I curl the _snapshot endpoint of the Elasticsearch cluster I can see that end_time is actually wrong:

$ curl -X GET "localhost:9200/_snapshot/r0/_current" | jq .  
{
  "snapshots": [
    {
      "snapshot": "test",
      "uuid": "f3jm-sDnTHqmLrEq-JLEpw",
      "version_id": 5061099,
      "version": "5.6.10",
      "indices": [
        ".ltrstore",
        "live",
        "settings"
      ],
      "state": "IN_PROGRESS",
      "start_time": "2019-03-12T17:03:27.241Z",
      "start_time_in_millis": 1552410207241,
      "end_time": "1970-01-01T00:00:00.000Z",
      "end_time_in_millis": 0,
      "duration_in_millis": -1552410207241,
      "failures": [],
      "shards": {
        "total": 0,
        "failed": 0,
        "successful": 0
      }
    }
  ]
}

Once the snapshot finished, the FINISHED and DURATION columns show a correct result.

If this is an issue of ES, should vulcanizer check this before showing weird results?

allow setting username/password in yaml config

AWS Elasticsearch Service Support - PR's Welcome?

My colleague pointed me at this tool, and it looks great! I'd started implementing something similar before finding out about Vulcanizer.

We use AWS' managed service offering of ElasticSearch, AWS Elasticsearch Service. While it exposes the same API as standard ElasticSearch, requests to the AWS service must be signed using IAM credentials.

Is support for AWS offering something you'd consider supporting? I'd happily try to put together a PR that introduces request signing if the host ends with amazonaws.com - but I thought I'd check if that aligned with this project before I tried to implement anything.

Integrate linting to the project

Linting can catch a few things that the compiler won't and help with standardization. golangci-lint looks to be a good candidate: https://github.com/golangci/golangci-lint

Malformed http request

Hi!
Each command that I try looks like adding strange double slash symbols //

vulcanizer -f conf/vulcanizer.yaml -c local settings
Error getting settings: Bad HTTP Status from Elasticsearch: 400, No handler found for uri [///_cluster/settings] and method [GET]

My config:

conf/vulcanizer.yaml
local:
  host: "test-run-kproskurin-01"
  port: 9200

Am I doing something wrong?

Get Disk Allocation information for Nodes

I have a need to get disk allocation/usage information from the _cat/allocation API. Ideally it would combine with the information from _cat/nodes/GetNodes().

I have a working PR at #69 that implements this by adding the disk allocation information to the existing Node struct. It should probably be a flag or subcommand in the CLI for vulcanizer nodes. Flag would probably be tidier.

Integrate public CI server

Setup travis to build and run tests.

Allow setting cluster settings to null

First of all, thanks a ton for this library, makes my life so much easier managing Elasticsearch!!

My usecase for this is the following. I defer a RestoreClusterSettings() function so that I can reset whatever settings I've made changes to during operation after I'm done, or I cancel.
The originalValue, newValue := SetClusterSetting(...) pattern makes this really easy, however, if this setting was unset previously the originalValue is an empty string.

This leads to the following error when I restore cluster settings:

failed to restore cluster setting indices.recovery.max_bytes_per_sec: Bad HTTP Status from Elasticsearch: 400, {"error":{"root_cause":[{"type":"remote_transport_exception","reason":"[mynode][10.1337.1.1:9300][cluster:admin/settings/update]"}],"type":"illegal_argument_exception","reason":"failed to parse setting [indices.recovery.max_bytes_per_sec] with value [] as a size in bytes: unit is missing or unrecognized","caused_by":{"type":"parse_exception","reason":"failed to parse setting [indices.recovery.max_bytes_per_sec] with value [] as a size in bytes: unit is missing or unrecognized"}},"status":400}

It would be nice if either SetClusterSetting could handle this, or if we created a new function to set a setting to Null in Elasticsearch (to fall back to the default value).

I tried passing in the string "null" as well, but it expects an actual JSON null value rather than a string "null".

Resetting a value in 7.5: https://www.elastic.co/guide/en/elasticsearch/reference/7.5/cluster-update-settings.html
This procedure is the same in 5.x as well, so shouldn't be a breaking change: https://www.elastic.co/guide/en/elasticsearch/reference/5.0/cluster-update-settings.html

Diff mappings

I saw in the functionality diff mappings in the roadmap.
Have you already given any thought on how should this be implemented ? (calling git from the go code?)

The desired behaviour of this functionality using the CLI, I guess would be to print the diff git style on the console, but how should this functionality behave when vulcanizer is used as a library ?

A link in this repo's readme points to a folder called "vendor" which does not (or no longer?) exists in this repo. Gives Status code [404:NotFound]

The link in this repo's readme is:
Status code [404:NotFound] - Link: https://github.com/github/vulcanizer/tree/master/vendor

This is near the bottom in the License section.

However there is not file/folder in this repo (any more?) with this name.

Extra

This bad link was found using a tool I recently created that uses the GitHub API to find repos and then validate the links within the readme file:
https://github.com/MrCull/GitHub-Repo-ReadMe-Dead-Link-Finder

It can be used to re-check this repo via: http://githubreadmechecker.com/Home/Search?SingleRepoUri=https%3a%2f%2fgithub.com%2fgithub%2fvulcanizer

I'd love to head some feedback on this project/tool.
So please feel free to share any thoughts or comments.

Compatibility with Elasticsearch Domain hidden behind a Loadbalancer

Issue: When trying to access an Elasticsearch Domain behind a loadbalancer, a port is being forcibly attached to the request.

❯ vulcanizer settings -c prod
Error getting settings: dial tcp 0.0.0.0:0: connect: can't assign requested address
Get http://elasticsearchdomain.es.amazonaws.com:0/_cluster/settings: dial tcp 0.0.0.0:0: connect: can't assign requested address

Proposed Solution: If no port is specified in the .vulcanizer.yaml, no port should be attached to the request URL.

Panic during golangci-lint typecheck (cgo files)

When running script/test or script/integration-test my test cases always run fine, but I get this panic at the end:

INFO [lintersdb] Active 8 linters: [deadcode errcheck govet ineffassign megacheck structcheck typecheck varcheck] 
INFO [loader] Go packages loading at mode load deps types and syntax took 1.09808926s 
INFO [loader] SSA repr building took 9.743µs      
INFO [runner/skip dirs] sorted abs args: [/Users/jfudally/go/src/github.com/github/vulcanizer] 
INFO [runner] worker.10 took 4.856µs              
INFO [runner] worker.9 took 4.412µs               
INFO [runner] worker.11 took 5.524µs              
INFO [runner] worker.4 took 7.41µs                
INFO [runner] worker.6 took 5.302µs               
INFO [runner] worker.12 took 676.314µs with stages: errcheck: 668.225µs 
INFO [runner] worker.1 took 1.187459ms with stages: structcheck: 1.170657ms, typecheck: 2.98µs 
INFO [runner] worker.5 took 1.487666ms with stages: deadcode: 1.475917ms 
INFO [runner] worker.2 took 2.193127ms with stages: varcheck: 2.180181ms 
INFO [runner] worker.3 took 7.106743ms with stages: ineffassign: 7.092986ms 
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x163990c]

goroutine 1997 [running]:
github.com/golangci/golangci-lint/vendor/github.com/golangci/go-tools/lint.(*fnVisitor).Visit(0xc0119c0720, 0x1a28ee0, 0xc010abc300, 0x1009b98, 0x17bea40)
        /home/travis/gopath/src/github.com/golangci/golangci-lint/vendor/github.com/golangci/go-tools/lint/lint.go:837 +0x39c
github.com/golangci/golangci-lint/vendor/github.com/golangci/go-tools/lint.(*globalVisitor).Visit(0xc0119c0700, 0x1a28ee0, 0xc010abc300, 0x1a25420, 0xc0119c0700)
        /home/travis/gopath/src/github.com/golangci/golangci-lint/vendor/github.com/golangci/go-tools/lint/lint.go:820 +0xc2
go/ast.Walk(0x1a25420, 0xc0119c0700, 0x1a28ee0, 0xc010abc300)
        /home/travis/.gimme/versions/go1.11.1.linux.amd64/src/go/ast/walk.go:52 +0x66
go/ast.walkDeclList(0x1a25420, 0xc0119c0700, 0xc00bea3800, 0x37, 0x40)
        /home/travis/.gimme/versions/go1.11.1.linux.amd64/src/go/ast/walk.go:38 +0x9e
go/ast.Walk(0x1a25420, 0xc0119c0700, 0x1a28e60, 0xc001703000)
        /home/travis/.gimme/versions/go1.11.1.linux.amd64/src/go/ast/walk.go:353 +0x2656
github.com/golangci/golangci-lint/vendor/github.com/golangci/go-tools/lint.NodeFns.func1(0xc01042e740, 0xc010dd0f60, 0xc010bea9f0)
        /home/travis/gopath/src/github.com/golangci/golangci-lint/vendor/github.com/golangci/go-tools/lint/lint.go:787 +0x74
created by github.com/golangci/golangci-lint/vendor/github.com/golangci/go-tools/lint.NodeFns
        /home/travis/gopath/src/github.com/golangci/golangci-lint/vendor/github.com/golangci/go-tools/lint/lint.go:784 +0xe7

After some experimentation I noticed that after I upgraded to golangci-lint version 1.12 some warnings were revealed:

WARN [runner/megacheck] Can't run megacheck because of compilation errors in packages [github.com/github/vulcanizer/cmd/vulcanizer github.com/github/vulcanizer [github.com/github/vulcanizer.test]]: cmd/vulcanizer/allocation.go:1: /usr/local/Cellar/go/1.12.1/libexec/src/internal/bytealg/compare_amd64.s:5:1: illegal character U+0023 '#' and 65 more errors: run `golangci-lint run --no-config --disable-all -E typecheck` to see all errors 
INFO [runner] worker.6 took 819.498918ms with stages: megacheck: 819.457656ms 
INFO [runner] worker.1 took 836.366061ms with stages: typecheck: 836.351286ms 
INFO [runner] Workers idle times: #2: 836.34663ms, #3: 829.386557ms, #4: 836.298983ms, #5: 836.267343ms, #6: 16.983322ms, #7: 836.360184ms, #8: 836.311028ms, #9: 836.30117ms, #10: 836.286849ms, #11: 836.293199ms, #12: 836.317279ms 
INFO [runner] processing took 4.294101ms with stages: exclude: 1.599941ms, skip_dirs: 1.196091ms, cgo: 1.017974ms, source_code: 155.118µs, path_prettifier: 113.237µs, autogenerated_exclude: 101.29µs, nolint: 93.31µs, uniq_by_line: 8.445µs, max_same_issues: 2.85µs, path_shortener: 2.457µs, max_per_file_from_linter: 2.023µs, max_from_linter: 893ns, diff: 262ns, skip_files: 210ns 
cmd/vulcanizer/allocation.go:1: /usr/local/Cellar/go/1.12.1/libexec/src/reflect/asm_amd64.s:5:1: illegal character U+0023 '#' (typecheck)
package main
es.go:1: /usr/local/Cellar/go/1.12.1/libexec/src/reflect/asm_amd64.s:5:1: illegal character U+0023 '#' (typecheck)
package vulcanizer

Seems that it's trying to lint the cgo files as .go files, and I've found an issue with golangci-lint to validate: golangci/golangci-lint#433 .

Provide pre built releases for the major platforms

It would be very convenient if this project could provide pre built releases for the major platforms e.g. via GoReleaser. Installing it via go get -u or git clone and then use the custom build script is not optimal.

Passing authentication through Go API

Am I understanding your library correctly that currently there is no way to create a new client to interact with elasticsearch by just passing in the elasticsearch username, host, port and password to create a connection with a certain elasticsearch host?

I have credentials that I'd much rather just pass into a function to create a client rather than writing to a file like ~/.vulcanizer.yaml and then calling NewClient() on after writing to that file. Thanks

Get index settings for "private" indices not working

Note: when I say "private" I mean elasticsearch indices beginning with a . (i.e .security-* indices or .watches index etc..) couldn't find the official term.

Use case is if I want to check the index settings for a .security index (assume this exists in the cluster). Right now if I pass in .security into the GetIndexSettings function, the underlying function call handleErrWithBytes correctly retrieves the necessary info - however when calling get bytes, it will 'fail' and return an empty Result object

I think this is because the underlying Get function requires input strings to escape . characters but the handleErrWithBytes just takes the value as is. Thus, if I pass in a string with an escaped . then the handleErrWithBytes will return an empty body but if assuming I somehow got the body JSON response back - then the underlying Get function would have properly retrieved the string because I escaped the . and it would know how to interpret that path.

I have a current workaround which is just to pass in *security into the GetIndexSettings function - but I feel like the ideal would not have to do that. Let me know if you'd like some example code or need anymore clarification.

Release v0.5.2

There's quite a few changes since 0.5.1 and it would be nice to have a new release and not have to use forks or specific commits!

While running my own version v0.5.2 in a fork, I discovered an issue in the new release (v0.2.16) of gorequest.

There's been some changes to how it detects content-type which breaks Vulcanizer when talking to Elasticsearch 1.7. So be careful when you upgrade dependencies!
If you do want to upgrade gorequest, you will need to explicitly set the content-type in

vulcanizer/es.go

Line 352 in dc3a2fd

agent := gorequest.New().Set("Accept", "application/json")

by changing that line to:

agent := gorequest.New().Set("Accept", "application/json").Set("content-type","application/json")

This change works with ES 1.7, ES 5.6, and ES 7.7.

Integrate go-docs to build documentation

This project should generate documentation on the public API using godocs.

Add an ability to set multiple hosts per cluster

Hi!

As I understand right now it's possible to pass only a single host per cluster which is not very convenient since ES could easily survive a failer of a single node and continue to work.

I'd like to pass a list of hosts and make vulcanizer try to connect to them all in some kind of order with some kind of timeouts.

Does it make sense?

Request to add prompt for password

Currently we can password passwords only by vulcanizer.yaml configuration file or in cmdline.

Can/Should we allow password as prompt if not provided after --password flag? (or new flag all together)