Coder Social home page Coder Social logo

jsiebens / hashi-up Goto Github PK

View Code? Open in Web Editor NEW
670.0 13.0 54.0 191 KB

bootstrap HashiCorp Consul, Nomad, or Vault over SSH < 1 minute

License: MIT License

Go 76.09% Shell 23.91%
hashicorp nomad consul automation devops go raspberry-pi nomad-cluster vault consul-cluster

hashi-up's Introduction

hashi-up

hashi-up is a lightweight utility to install HashiCorp Consul, Nomad or Vault on any remote Linux host. All you need is ssh access and the binary hashi-up to build a Consul, Nomad or Vault cluster.

The tool is written in Go and is cross-compiled for Linux, Windows, MacOS and even on Raspberry Pi.

This project is heavily inspired on the work of Alex Ellis who created k3sup, a tool to to get from zero to KUBECONFIG with k3s

Go Report Card License: MIT GitHub All Releases

What's this for?

This tool uses ssh to install HashiCorp Consul, Nomad or Vault to a remote Linux host. You can also use it to join existing Linux hosts into a Consul, Nomad, Vault or Boundary cluster. First, Consul, Nomad or Vault is installed using a utility script, along with a minimal configuration to run the agent as server or client.

hashi-up was developed to automate what can be a very manual and confusing process for many developers, who are already short on time. Once you've provisioned a VM with your favourite tooling, hashi-up means you are only 60 seconds away from running nomad status on your own computer.

Download hashi-up

hashi-up is distributed as a static Go binary. You can use the installer on MacOS and Linux, or visit the Releases page to download the executable for Windows.

curl -sLS https://get.hashi-up.dev | sh
sudo install hashi-up /usr/local/bin/

hashi-up version

Usage

The hashi-up tool is a client application which you can run on your own computer. It uses SSH to connect to remote servers when installing HashiCorp Consul or Nomad. Binaries are provided for MacOS, Windows, and Linux (including ARM).

SSH credentials

By default, hashi-up talks to an SSH agent on your host via the SSH agent protocol. This saves you from typing a passphrase for an encrypted private key every time you connect to a server. The ssh-agent that comes with OpenSSH is commonly used, but other agents, like gpg-agent or yubikey-agent are supported by setting the SSH_AUTH_SOCK environment variable to the Unix domain socket of the agent.

The --ssh-target-key flag can be used when no agent is available or when a specific private key is preferred.

The --ssh-target-user and --ssh-target-password flags allow you to authenticate using a username and a password.

Guides

Resources

Deploying a highly-available Nomad cluster with hashi-up!

Building a Nomad cluster on Raspberry Pi running Ubuntu server

Installing HashiCorp Vault on DigitalOcean with hashi-up

hashi-up's People

Contributors

dependabot[bot] avatar drio avatar jsiebens avatar levex avatar mxab avatar tannevaled avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

hashi-up's Issues

Allow the location of the `get` actions to be configurable

At the moment, hashi-up <tool> get fetches always into ~/bin/. It would be wonderful to be able to control where this binary gets stored.

Hashi-up is an extremely valuable tool for working with disposable build agents. It is hoped that this feature will allow these build agents to be more "flexible".

cert create command is moved or broken

Womp, this does seem like an actual bug

In the blog post I referred to in the other issue, there is a line to setup the certs

hashi-up cert create --host 165.232.109.36 --host vault.example.com

This functionality seems to have been moved or deleted

Error: unknown command "cert" for "hashi-up"
Run 'hashi-up --help' for usage.
unknown command "cert" for "hashi-up"

Browsed through the CLI manual a bit but didn't see anything explicitly cert related.

Issue when installing caused by sudo requiring password.

hashi-up nomad install \ --ssh-target-addr $IP \ --ssh-target-user ubuntu \ --server [INFO] Uploading generated Nomad configuration ... [INFO] Installing Nomad ... sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper error received during installation: Process exited with status 1
I've generated an ssh key and added it to authorized keys on the remote server, but it seems to not be working. I tried to use a passwordless ssh key, but that didn't make a difference.

Not running 'apt update' causing: "E: Unable to locate package unzip"

Flag --client has been deprecated, use the new flag client-addr
[INFO] Uploading generated Consul configuration ...
[INFO] Installing Consul ...
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package unzip
error received during installation: Process exited with status 100

Logging in on the server and running apt update first solves the issue. :) (ah and also, the docs said to use --client, but I see it is now --client-addr.)

Error on nomad install on RASPI 4 with Ubuntu 20.04

I tried to install nomad with hashi-up as documented

export IP=192.168.0.31
hashi-up nomad install --ssh-target-addr $IP --ssh-target-user ubuntu --server

I got following output:
Uploading Nomad configuration and certificates...
Installing Nomad...
[INFO] Downloading and unpacking nomad_1.0_linux_arm64.zip
Error: error received during installation: Process exited with status 22

The consul installation worked without any issue

regards bigwasp

Consul config field deprecations

After starting Consul there are deprecation warnings in the log:

agent: The 'ca_file' field is deprecated. Use the 'tls.defaults.ca_file' field instead.
agent: The 'cert_file' field is deprecated. Use the 'tls.defaults.cert_file' field instead.
agent: The 'key_file' field is deprecated. Use the 'tls.defaults.key_file' field instead.
agent: The 'verify_incoming_rpc' field is deprecated. Use the 'tls.internal_rpc.verify_incoming' field instead.
agent: The 'verify_outgoing' field is deprecated. Use the 'tls.defaults.verify_outgoing' field instead.
agent: The 'verify_server_hostname' field is deprecated. Use the 'tls.internal_rpc.verify_server_hostname' field instead.
agent: The 'ui' field is deprecated. Use the 'ui_config.enabled' field instead.

It works right now, but probably need to update generated config before those config options are removed.

consul -v:

Consul v1.14.2
Revision 0ba7a401
Build Date 2022-11-30T19:54:31Z
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

https://developer.hashicorp.com/consul/docs/agent/config/config-files#tls_deprecated_options

terraform outputs in digitalocean/install_boundary.sh example cause host not found error

I recently discovered this great tool and ran into some trouble running the boundary_install script for digital ocean.

It seems that unless you pass the "-r" flag to the terraform IP address outputs the connection fails.

I don't enough about Go to make any educated guesses. I guess it could also be a shell issue, I am using ZSH on Ubuntu 22.04 for reference.

Thanks and great work!

-Alexi

how to only install (no configuration) of Consul/Nomad

Is it possible to only install the specified versions of Consul and Nomad on the machine without touching anything related to configurations?

The use case is a day-N operation where I would rather go around updating the binaries (and doing service restart) rather than roll out new AMIs. :)

hashi-up unable to ssh to fresh DigitalOcean droplet

When trying to run hash-up nomad install on a fresh digitalocean droplet, I'm met with the following error:

$ hashi-up nomad install --ssh-target-addr 67.207.94.29  --ssh-target-key ~/.ssh/id_rsa

There was an issue connecting to your target host.
This could happen when hashi-up can not reach the target host or when the private key authentication is invalid.

Reason: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

How to fix this?

- check if the target host is reachable and an SSH server is running
- check if the user and the private key are valid

What's odd is that I am able to ssh onto the host using root without a problem. Below I'll outline additional context along with my machine specs and stuff.

Context
I'm provisioning my droplet with terraform, using the following config...

resource "digitalocean_droplet" "nomad_server" {
  image    = "ubuntu-22-04-x64"
  name     = "nomad-server"
  region   = "nyc1"
  size     = "s-1vcpu-1gb"
  ssh_keys = [data.digitalocean_ssh_key.corey_mb_air.id]
}

The ssh key above is the public key on my M1 MacBook Air running MacOS 12.5.1. When running ssh [email protected], I am able to log onto the host without a problem. Passwordless login is also enabled.

What improvements can be made?

I am curious about what kind of improvements can be made to get this beyond proof of concept?(perhaps to alpha/beta phase) Could you possibly create some issues to indicate what work a contributor could help with? I am a fairly capable go lang developer so just curious where we can contribute some help.

Cert Documentation

is there any docs on using the secure by default options? setting up certs and what not? or any helpful nudges. I can write up my experiences once I get there!

3-node HA possible?

Great effort, thank you for this!

This is more of a question - is there a way to have 3 nodes in a HA config, with each node performing both server and agent duties? If yes, how would one set it up using hashi-up?

Is it possible to have both agent at server on the same hardware

I know it's strongly advised against.
But for small setups, and when you don't have a lot of hardware, it could be fine for a home-lab situation.
Having two or three raspberry pis, and have them all run servers and agents would enable for a little load balancing and redundancy.

In that scenario, having 1 server and two agents, is rather vulnerable.

Error while using `get` to fetch into current working directory

It would appear that the get action is unable to fetch binaries and store them in the current working directory...

$ hashi-up terraform get -d .
Downloading file https://releases.hashicorp.com/terraform/0.15.3/terraform_0.15.3_linux_amd64.zip 
31.23 MiB / 31.23 MiB [-------------------------------------------------------------------------] 100.00%
Extracting file: terraform to .
unable to install Terraform distribution: terraform: Illegal file path

Add installation instructions for Nix

I found myself using hashi-up in a Nix shell environment so I decided to contribute it to Nixpkgs: NixOS/nixpkgs#194288. The PR is under review but should land soon-ish. You're under no responsibility to add Nix installation docs, of course, but I wanted to make you aware that this is now an option for users ❤️

CNI Plugin

Hey,

I love it. Would be difficult to add CNI for nomad?

Non-optional yum update before installation

Is there a specific reason to always do a yum update before installing anything?

I would like that to be at least optional, or even not included and leave that up to the user.

Get feature should check if the file already exists before attempting to download overtop

At the moment, if you run hashi-up foo get it will download the file regardless of whether the binary already exists...

$ sudo hashi-up terraform get --dest /usr/local/bin
Password:
Downloading file https://releases.hashicorp.com/terraform/1.0.3/terraform_1.0.3_darwin_amd64.zip
31.22 MiB / 31.22 MiB [-----------------------------------------------------------------------------------------------------] 100.00%
Extracting file: terraform to /usr/local/bin
$ sudo hashi-up terraform get --dest /usr/local/bin
Downloading file https://releases.hashicorp.com/terraform/1.0.3/terraform_1.0.3_darwin_amd64.zip
31.22 MiB / 31.22 MiB [-----------------------------------------------------------------------------------------------------] 100.00%
Extracting file: terraform to /usr/local/bin
$ sudo hashi-up terraform get --dest /usr/local/bin
Downloading file https://releases.hashicorp.com/terraform/1.0.3/terraform_1.0.3_darwin_amd64.zip
31.22 MiB / 31.22 MiB [-----------------------------------------------------------------------------------------------------] 100.00%
Extracting file: terraform to /usr/local/bin
...

It would be fantastic if hashi-up would display the output of foo version if it finds the binary exists already and perhaps skips downloading if version is already latest (or else it would just halt right away without checking versions on releases.hashicorp.com).

Support for air-gaped environments using proxy

I really love hashi-up and looking forward to using it in our test environment at work, however its air gaped without internet access. It would be nice if i can provide cli flags to hashi-up so that it download binaries via a proxy. I'm also willing to work on that myself to contribute to this awesome tool.

Consul config error with TLS and Connect

Enabling both TLS and Connect generates config that Consul refuses to accept.

consul[10798]: ==> the `ports.grpc` listener no longer supports TLS. Use `ports.grpc_tls` instead. This message is appearing because GRPC is configured to use TLS, but `ports.grpc_tls` is not defined
systemd[1]: consul.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: consul.service: Failed with result 'exit-code'.
systemd[1]: consul.service: Service hold-off time over, scheduling restart.

--ssh-target-sudo-pass flag does not seem to work

I am passing the --ssh-target-sudo-pass flag and commands are failing. Command and output below. While this is the same error message as #12, I don't think it's the same root issue.

hashi-up consul install \
  --ssh-target-addr $SERVER_1_IP \
  --ssh-target-user atuser \
  --ssh-target-key ~/.ssh/nomad
  --ssh-target-sudo-pass {{redacted}}
  --server \
  --client-addr 0.0.0.0 \
  --bootstrap-expect 3 \
  --retry-join $SERVER_1_IP --retry-join $SERVER_2_IP --retry-join $SERVER_3_IP

Output:

[INFO] Uploading generated Consul configuration ...
[INFO] Installing Consul ...
sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper
sudo: a password is required
error received during installation: Process exited with status 1
 : command not found
 : command not found

Output with the -v flag:

panic: Invalid Semantic Version

goroutine 1 [running]:
github.com/Masterminds/semver.MustParse(...)
        /home/runner/go/pkg/mod/github.com/!masterminds/[email protected]/version.go:100
github.com/jsiebens/hashi-up/pkg/config.GetArmSuffix({0x927e5b?, 0xc00029c2a0?}, {0x7ffc233dc1fa, 0x2})
        /home/runner/work/hashi-up/hashi-up/pkg/config/version.go:79 +0xc8
github.com/jsiebens/hashi-up/cmd.InstallConsulCommand.func1.1({0xa05b00?, 0xc000140000})
        /home/runner/work/hashi-up/hashi-up/cmd/consul_install.go:157 +0xae8
github.com/jsiebens/hashi-up/pkg/operator.executeRemote({0x7ffc233dc20f, 0xe}, {0x7ffc233dc233, 0x6}, {0xa05590?, 0xc00007f580}, 0xc000193ca0)
        /home/runner/work/hashi-up/hashi-up/pkg/operator/operator.go:152 +0x309
github.com/jsiebens/hashi-up/pkg/operator.ExecuteRemote({0x7ffc233dc20f, 0xe}, {0x7ffc233dc233, 0x6}, {0x7ffc233dc24e?, 0x16?}, {0x0?, 0x0?}, 0x100?)
        /home/runner/work/hashi-up/hashi-up/pkg/operator/operator.go:93 +0x910
github.com/jsiebens/hashi-up/cmd.(*Target).execute(0xc000088360, 0xc000193ca0)
        /home/runner/work/hashi-up/hashi-up/cmd/target.go:42 +0xb1
github.com/jsiebens/hashi-up/cmd.InstallConsulCommand.func1(0xc0001eaf00?, {0x92773f?, 0xa?, 0xa?})
        /home/runner/work/hashi-up/hashi-up/cmd/consul_install.go:186 +0x2f1
github.com/muesli/coral.(*Command).execute(0xc0001eaf00, {0xc00020d400, 0xa, 0xa})
        /home/runner/go/pkg/mod/github.com/muesli/[email protected]/command.go:856 +0x67c
github.com/muesli/coral.(*Command).ExecuteC(0xc000005b80)
        /home/runner/go/pkg/mod/github.com/muesli/[email protected]/command.go:974 +0x3bd
github.com/muesli/coral.(*Command).Execute(...)
        /home/runner/go/pkg/mod/github.com/muesli/[email protected]/command.go:902
github.com/jsiebens/hashi-up/cmd.Execute()
        /home/runner/work/hashi-up/hashi-up/cmd/command.go:31 +0x3f4
main.main()
        /home/runner/work/hashi-up/hashi-up/main.go:12 +0x1d

I also tried running this without the flag and having provided sudo/no password permissions to the user in question

Unable to fetch newest version of terraform listed at releases.hashicorp.com

Using get causes hashi-up to automatically fetch the 0.15.3 version of terraform but https://releases.hashicorp.com/index.json or https://releases.hashicorp.com/index.html lists 0.15.4 as the latest version.

Do I need to reinstall hashi-up whenever there are new versions listed on the HashiCorp site or there some other method by which the newest versions are discovered by hashi-up?

$ hashi-up version
Version: v0.10.1
Git Commit: ec7f7281fc3536c3206ac64394517fb6850e13a1

Allow for host authentication

Hey, awesome tool, I had previously abandoned a vault installation because it was a headache lol. This looks like it resolves a lot of that pain.

Was running through your digital ocean demo for vault and came across a small little issue.

When a new droplet is created, (at least on Mac) you need to actually ssh to the box ahead of running hashi-up. The reason being, for new SSH connections, you need to approve adding the host.

Reproduce

To recreate, create a new droplet

doctl compute droplet create --image ubuntu-18-04-x64 --size s-1vcpu-2gb --region nyc1 vault --tag-names vault --wait --ssh-keys $SSH_KEY 

try to run hashi-up

hashi-up vault install --ssh-target-addr 111.111.111.11 --version 1.7.1 --ssh-target-key ~/my/key

you should(?) get the following error

Error: TargetConnectError

There was an issue connecting to your target host. 
This could happen when hashi-up can not reach the target host or when the private key authentication is invalid.

The resolution here is super simple, you should just be able to run

ssh -I /my/key [email protected]

and manually approve the host

The authenticity of host '111.111.111.11' can't be established.
ECDSA key fingerprint is {{omitted}}.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes

at this point you can go ahead with hashi-up.

Since this isn't really a bug with hashi-up, I wonder if the best path forward is to pass the host authentication through to the user, if possible. Alternatively, perhaps just a warning in the error output to make sure users have checked their host authentication.

https-only false ignored?

If I understood the cobra documentation correctly, I should be able to set the --https-only flag to false to enable HTTP even if HTTPS is configured. However, it seems that the flag is not evaluated.
Concretely, I have the following use case: I want to encrypt network traffic from Consul cluster, but allow locally the default address http://127.0.0.1:8500 to allow a Nomad agent to register automatically. For this I try the following command:

hashi-up consul install \
  --ssh-target-addr 10.3.5.20 \
  --ssh-target-user vagrant \
  --ssh-target-key ~/.ssh/insecure_private_key \
  --server \
  --connect \
  --encrypt tbWAuqbfUczBy6hWy3uNdjj1Q8W3XQWlodNKP61f0aU= \
  --ca-file consul-agent-ca.pem \
  --cert-file dc1-server-consul-0.pem \
  --key-file dc1-server-consul-0-key.pem \
  --client-addr 10.3.5.20 \
  --bind-addr 0.0.0.0 \
  --advertise-addr 10.3.5.20 \
  --https-addr 10.3.5.20 \
  --http-addr 127.0.0.1 \
  --https-only false \
  --bootstrap-expect 3 \
  --retry-join 10.3.5.20 --retry-join 10.3.5.30 --retry-join 10.3.5.40

This generates the following Consul configuration:

datacenter = "dc1"
data_dir = "/opt/consul"
bind_addr = "0.0.0.0"
advertise_addr = "10.3.5.20"
client_addr = "10.3.5.20"
retry_join = ["10.3.5.20", "10.3.5.30", "10.3.5.40"]
ports {
  grpc = 8502
  https = 8501
  http = -1
}
addresses {
  http = "127.0.0.1"
  https = "10.3.5.20"
}
ui = true
server = true
bootstrap_expect = 3
encrypt = "xxx"
ca_file = "/etc/consul.d/consul-agent-ca.pem"
cert_file = "/etc/consul.d/dc1-server-consul-0.pem"
key_file = "/etc/consul.d/dc1-server-consul-0-key.pem"
verify_incoming_rpc = true
verify_outgoing = true
verify_server_hostname = true
connect {
  enabled = true
}

I would expect the following configuration:

datacenter = "dc1"
data_dir = "/opt/consul"
bind_addr = "0.0.0.0"
advertise_addr = "10.3.5.20"
client_addr = "10.3.5.20"
retry_join = ["10.3.5.20", "10.3.5.30", "10.3.5.40"]
ports {
  grpc = 8502
  https = 8501
  http = 8500
}
addresses {
  http = "127.0.0.1"
  https = "10.3.5.20"
}
ui = true
server = true
bootstrap_expect = 3
encrypt = "xxx"
ca_file = "/etc/consul.d/consul-agent-ca.pem"
cert_file = "/etc/consul.d/dc1-server-consul-0.pem"
key_file = "/etc/consul.d/dc1-server-consul-0-key.pem"
verify_incoming_rpc = true
verify_outgoing = true
verify_server_hostname = true
connect {
  enabled = true
}

Am I doing something wrong or is this a bug?

Gateway timeout error

I have set up a cluster with nomad, consul and vault using hashi-up, but I receive gateway timeout errors when I connect to one nomad client ip but request a service running on a second nomad client (e.g. when trying one of nomad's load balancing tutorials). Since I am not sure whether this has to do with hashi-up, I asked for help on the nomad forum: https://discuss.hashicorp.com/t/internal-routing-problem/34201

This is how I deployed my stack: https://github.com/davosian/home-cluster-v2#nomad-consul-and-vault-installation
Am I missing something?

"Connection refused" on brand new Raspberry Pi 4 install

The setup:

From a windows machine, I ran the following commands:

.\hashi-up.exe consul install --ssh-target-addr 10.0.0.123 --ssh-target-user pi --ssh-target-password raspberry --server --advertise-addr "{{ GetInterfaceIP \"eth0\" }}" --bind-addr "{{ GetInterfaceIP \"eth0\" }}" --client-addr 0.0.0.0

.\hashi-up.exe nomad install --ssh-target-addr 10.0.0.123 --ssh-target-user pi --ssh-target-password raspberry --server --advertise "{{ GetInterfaceIP \"eth0\" }}" --bootstrap-expect 1

On the raspberry Pi, when I run the following command, I get an "connection refused" error
consul members -http-addr=http://10.0.0.123:8500
"Error retrieving members: Get "http://10.0.0.123:8500/v1/agent/members?segment=_all": dial tcp 10.0.0.123:8500: connect: connection refused"

I was able to install both consul and nomad on the pi.
Is there something I'm missing to do?

I've now tried doing the install with a ssh key instead of username/password
Same error

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.