picatz / terraform-google-nomad

📗 Terraform Module for Nomad clusters with Consul on GCP

Nomad Cluster

Terraform Module for Nomad clusters with Consul on GCP.

Module Features

• Includes HashiCorp's Consul service mesh
• Gossip encryption, mTLS, and ACLs enabled for Nomad and Consul
• Optional load balancer and DNS configuration
• Optional SSH bastion host
• Only the Docker task driver is enabled
• Installs the gVisor container runtime (runsc)
• Installs the Falco runtime security monitor

Cloud Shell Interactive Tutorial

For a full interactive tutorial to get started using this module:

Infrastructure Diagram

Logs

Logs are centralized using GCP's Cloud Logging. You can use the following filter to see all Nomad agent logs:

$ gcloud logging read 'resource.type="gce_instance" jsonPayload.ident="nomad"'
...

$ gcloud logging read 'resource.type="gce_instance" jsonPayload.ident="nomad" jsonPayload.host="server-0"' --format=json | jq -r '.[] | .jsonPayload.message' | less
...

Logs can also be collected within the cluster using Promtail and Loki, then visualized using Grafana (optionally exposed using a public load balancer and DNS name).

$ DNS_ENABLED=true PUBLIC_DOMAIN="nomad.your-domain.com" make terraform/apply
...
$ export CONSUL_HTTP_TOKEN=$(terraform output -json | jq -r .consul_master_token.value)
$ make consul/metrics/acls
...
🔑 Creating Consul ACL Token to Use for Prometheus Consul Service Discovery
AccessorID:       15b9a51d-7af4-e8d4-7c09-312c594a5907
SecretID:         2a1c7926-b6e3-566e-ddf5-b19279fa134e
Description:
Local:            false
Create Time:      2021-04-11 16:16:03.90231.6.1 +0000 UTC
Roles:
   6ae941.6.1c07-49a7-fa95-8ce14aa8a75e - metrics

$ consul_acl_token=2a1c7926-b6e3-566e-ddf5-b19279fa134e make nomad/metrics
$ make nomad/logs
$ make nomad/ingress
$ GRAFANA_PUBLIC_DOMAIN="grafana.your-domain.com" GRAFANA_LOAD_BALANCER_ENABLED=true DNS_ENABLED=true PUBLIC_DOMAIN="nomad.your-domain.com" make terraform/apply
$ open http://public.grafana.your-domain.com:3000/login

Bootstrap ACL Token

If the cluster is started with ACLs enabled, which is the default behavior of this module, you may see this:

$ export NOMAD_ADDR="https://$(terraform output -json | jq -r .load_balancer_ip.value):4646"
$ nomad status
Error querying jobs: Unexpected response code: 403 (Permission denied)

We can bootstrap ACLs to get the bootstrap management token like so:

$ nomad acl bootstrap
Accessor ID  = a1495889-37ce-6784-78f3-31.6.1984bca
Secret ID    = dc8c0349-c1fd-dc2c-299c-d513e5dd6df2
Name         = Bootstrap Token
Type         = management
Global       = true
Policies     = n/a
Create Time  = 2020-04-27 05:24:43.734587566 +0000 UTC
Create Index = 7
Modify Index = 7

Then we can use that token (Secret ID) to perform the rest of the ACL bootstrapping process:

$ export NOMAD_TOKEN="dc8c0349-c1fd-dc2c-299c-d513e5dd6df2"
$ nomad status
No running jobs
$ ...

Use ssh-mtls-terminating-proxy to access the Nomad UI

When using the SSH bastion, you can use the ssh-mtls-terminating-proxy.go helper script to tunnel a connection from localhost to the Nomad server API:

$ make ssh/proxy/mtls
2021/04/11.16.18:28 getting terraform output
2021/04/11.16.18:29 Bastion IP: "34.73.106.60"
2021/04/11.16.18:29 Server IP: "1.6.168.2.8"
2021/04/11.16.18:29 Setting up SSH agent
2021/04/11.16.18:29 connecting to the bastion
2021/04/11.16.18:29 connecting to the server through the bastion
2021/04/11.16.18:30 wrapping the server connection with SSH through the bastion
2021/04/11.16.18:30 tunneling a new connection for Consul to the server with SSH through the bastion
2021/04/11.16.18:30 loading Consul TLS data
2021/04/11.16.18:30 tunneling a new connection for somad to the server with ssh through the bastion
2021/04/11.16.18:30 loading Nomad TLS data
2021/04/11.16.18:30 starting Consul local listener on localhost:8500
2021/04/11.16.18:30 starting Nomad local listener on localhost:4646
...

Then open your browser at http://localhost:4646/ui/ to securely access the Nomad UI.

terraform-google-nomad's Issues

Use Ops Agent

GCP now recommends using the new Ops Agent for monitoring and logging.

• https://cloud.google.com/stackdriver/docs/solutions/agents/ops-agent
• https://cloud.google.com/stackdriver/docs/solutions/agents/ops-agent/installation#install-latest-version
• https://cloud.google.com/stackdriver/docs/solutions/agents/ops-agent/authorization

Install the Stackdriver monitoring agent

To start getting the builtin observability features working.

curl -sSO https://dl.google.com/cloudagents/add-monitoring-agent-repo.sh && sudo bash add-monitoring-agent-repo.sh --also-install && sudo service stackdriver-agent start

Likely need to add a new scope to the vm module:

https://www.googleapis.com/auth/monitoring.write

Minor typo, should be "sensitive"

terraform-google-nomad/ssh-mtls-terminating-proxy.go

Line 24 in b7ea9c7

Senstive bool `json:"senstive"`

Install Falco

Falco can be included to enable runtime security monitoring by default.

Investigate running Rootless Docker Daemon on Nomad clients

🤦 Following up on a1a3f29, when rootless and experimental were set to true, the Docker Daemon still seemed to be running as root. So, I seem to be missing something. Need to further investigate out that works, and then probably reimplement it.

Interpolation-only Expressions Are Deprecated

$ terraform validate .

Warning: Interpolation-only expressions are deprecated

  on modules/open-port/firewall.tf line 6, in resource "google_compute_firewall" "open_port":
   6:     protocol = "${var.protocol}"

Terraform 0.11 and earlier required all non-constant expressions to be
provided via interpolation syntax, but this pattern is now deprecated. To
silence this warning, remove the "${ sequence from the start and the }"
sequence from the end of this expression, leaving just the inner expression.

Template interpolation syntax is still used to construct strings from
expressions when the template includes multiple interpolation sequences or a
mixture of literal strings and interpolations. This deprecation applies only
to templates that consist entirely of a single interpolation sequence.

(and 4 more similar warnings elsewhere)

Does not work on `darwin_arm64`

$ terraform init 
╷
│ Error: Incompatible provider version
│ 
│ Provider registry.terraform.io/hashicorp/template v2.2.0 does not have a package available for your current platform, darwin_arm64.
│ 
│ Provider releases are separate from Terraform CLI releases, so not all providers are available for all platforms. Other versions of
│ this provider may have different platforms supported.
╵

The last release was two years ago.

Add an option to deploy Consul with Nomad

Nomad's integration with Consul is really great, and there should be an option to enable deploying a secure-by-default Consul cluster to make it easy to get started.

Prevent Plaintext Secrets in the Compute Metadata Service

As of v2.0.0/#18 , this module deploys a Consul cluster in tandem with the Nomad cluster. Moreover, it uses the metadata service to perform the majority of the dynamic server configuration, and exposes many secrets to malicious/compromised workloads on Nomad client instances.

All secrets should be removed from the metadata service, or at least not stored in plaintext.

Generate mTLS certificates dynamically with Terraform

Instead of generating the mTLS certificates with cfssl before using Packer during the creation of the images, it'd be great to just dynamically create those certificates when I spin up the cluster using Terraform's tls provider.

Permission error when setting up a server

I have this error when trying to setup the consul server, not sure how to add the required permissions?

Cannot discover address: cluster=LAN address="provider=gce project_name=<project>  tag_value=server" error="discover-gce: googleapi: Error 403: Required 'compute.zones.list' permission for 'projects/<project-name>'"

Stackdriver Logging agent error, VMs have insufficient authentication scopes

All nodes within the cluster are configured with the google-fluentd Stackdriver Logging agent. When looking at the logs from the agent, I'm getting the following errors:

$ cat /var/log/google-fluentd/google-fluentd.log 
2020-07-12 18:15:56 +0000 [warn]: Failed to extract log entry errors from the error details: "Request had insufficient authentication scopes.". error_class=JSON::ParserError error="String"
...

☝️ This also means the VM instance logs aren't available in the GCP console.

I think I need to expand the service_accounts.scopes for VMs to also include a logging scope to fix this error.

Dependabot can't parse your load_balancer.tf

Dependabot couldn't parse the load_balancer.tf found at /load_balancer.tf.

The error Dependabot encountered was:

unable to parse HCL: At 3:20: Unknown token: 3:20 IDENT var.region

View the update logs.

Dependabot can't parse your bucket.tf

Dependabot couldn't parse the bucket.tf found at /bucket.tf.

The error Dependabot encountered was:

unable to parse HCL: At 2:19: Unknown token: 2:19 IDENT format

View the update logs.

Run nomad as non root user

Hi!

I was referencing this repo for deploying a similar stack on AWS. I came across install_nomad.sh#L27 and I think this is a typo. Should it not be owned by nomad user instead of root user?

sudo chown --recursive nomad:nomad /nomad

Errors and other alerts when following the tutorial steps

In step 4, right after executing the command 'terraform plan...' I have the output below, please how can I solve this problem?

Terraform v0.12.28

• provider.google v3.65.0
• provider.local v2.1.0
• provider.random v3.1.0
• provider.template v2.2.0
• provider.tls v3.1.0

Warning: registry.terraform.io: This version of Terraform has an outdated GPG key and is unable to verify new provider releases. Please upgrade Terraform to at least 0.12.31 to receive new provider updates. For details see: https://discuss.hashicorp.com/t/hcsec-2021-12-codecov-security-event-and-hashicorp-gpg-key-exposure/23512

Error: Reference to undeclared output value

on outputs.tf line 4, in output "ca_cert":
4: value = module.nomad.ca_cert

An output value with the name "ca_cert" has not been declared in module.nomad.

Error: Reference to undeclared output value

on outputs.tf line 10, in output "cli_cert":
10: value = module.nomad.cli_cert

An output value with the name "cli_cert" has not been declared in
module.nomad.

Error: Reference to undeclared output value

on outputs.tf line 16, in output "cli_key":
16: value = module.nomad.cli_key

An output value with the name "cli_key" has not been declared in module.nomad.

Error: Reference to undeclared output value

on outputs.tf line 38, in output "nomad_server_ip":
38: value = module.nomad.nomad_server_ip

An output value with the name "nomad_server_ip" has not been declared in
module.nomad.

---

Terraform v0.12.31

• provider.google v3.65.0
• provider.local v2.1.0
• provider.random v3.1.0
• provider.template v2.2.0
• provider.tls v3.1.0

Error: Reference to undeclared output value

on outputs.tf line 4, in output "ca_cert":
4: value = module.nomad.ca_cert

An output value with the name "ca_cert" has not been declared in module.nomad.

Error: Reference to undeclared output value

on outputs.tf line 10, in output "cli_cert":
10: value = module.nomad.cli_cert

An output value with the name "cli_cert" has not been declared in
module.nomad.

Error: Reference to undeclared output value

on outputs.tf line 16, in output "cli_key":
16: value = module.nomad.cli_key

An output value with the name "cli_key" has not been declared in module.nomad.

Error: Reference to undeclared output value

on outputs.tf line 38, in output "nomad_server_ip":
38: value = module.nomad.nomad_server_ip

An output value with the name "nomad_server_ip" has not been declared in
module.nomad.

---

Terraform v1.1.9
on linux_amd64

• provider registry.terraform.io/hashicorp/google v4.20.0
• provider registry.terraform.io/hashicorp/local v2.2.2
• provider registry.terraform.io/hashicorp/random v3.1.3
• provider registry.terraform.io/hashicorp/template v2.2.0
• provider registry.terraform.io/hashicorp/tls v3.3.0

│ Warning: Argument is deprecated
│
│ with module.nomad.tls_self_signed_cert.consul-ca,
│ on .terraform/modules/nomad/consul_tls_ca.tf line 10, in resource "tls_self_signed_cert" "consul-ca":
│ 10: key_algorithm = tls_private_key.consul-ca.algorithm
│
│ This is now ignored, as the key algorithm is inferred from the private_key_pem.
│
│ (and 13 more similar warnings elsewhere)
╵
╷
│ Error: Unsupported attribute
│
│ on outputs.tf line 4, in output "ca_cert":
│ 4: value = module.nomad.ca_cert
│ ├────────────────
│ │ module.nomad is a object, known only after apply
│
│ This object does not have an attribute named "ca_cert".
╵
╷
│ Error: Unsupported attribute
│
│ on outputs.tf line 10, in output "cli_cert":
│ 10: value = module.nomad.cli_cert
│ ├────────────────
│ │ module.nomad is a object, known only after apply
│
│ This object does not have an attribute named "cli_cert".
╵
╷
│ Error: Unsupported attribute
│
│ on outputs.tf line 16, in output "cli_key":
│ 16: value = module.nomad.cli_key
│ ├────────────────
│ │ module.nomad is a object, known only after apply
│
│ This object does not have an attribute named "cli_key".
╵
╷
│ Error: Unsupported attribute
│
│ on outputs.tf line 38, in output "nomad_server_ip":
│ 38: value = module.nomad.nomad_server_ip
│ ├────────────────
│ │ module.nomad is a object, known only after apply
│
│ This object does not have an attribute named "nomad_server_ip".

Allow more configuration of Consul integration

Currently, there is no way to really tune the Consul integration outside of enabling/disabling Consul ACLs and the default policy.

terraform-google-nomad/vars.tf

Lines 138 to 148 in 8c61d66

	variable "consul_acls_enabled" {
	type = bool
	default = true
	description = "If ACLs should be enabled for the Consul cluster."
	}
	variable "consul_acls_default_policy" {
	type = string
	default = "deny"
	description = "The default policy to use for Consul ACLs (allow/deny)."
	}

But there are many options available. These should be exposed as Terraform variables with secure defaults.

terraform-google-nomad/packer/configs/nomad/server.hcl

Lines 45 to 53 in 8c61d66

	consul {
	ssl = true
	verify_ssl = true
	address = "127.0.0.1:8501"
	ca_file = "/consul/config/consul-ca.pem"
	cert_file = "/consul/config/server.pem"
	key_file = "/consul/config/server-key.pem"
	token = "{CONSUL-TOKEN}"
	}

Extra important ones to consider would be allow_unathenticated and share_ssl. Consider disabling these by default with adjustments to documentation and examples.

Prevent Access to the Metadata Service from Container Workloads using iptables

Related to #19: Nomad client nodes should have this iptables rule included to prevent access to the metadata service API:

$ iptables --insert FORWARD 1 --in-interface nomad --destination 169.254.169.254/32 --jump DROP

It should also be persisted across restarts.

Dependabot can't parse your acl_tokens.tf

Dependabot couldn't parse the acl_tokens.tf found at /acl_tokens.tf.

The error Dependabot encountered was:

unable to parse HCL: At 2:27: Unknown token: 2:27 IDENT uuid

View the update logs.

Why is the nomad server configuration using a temporary folder for data?

I see that this line configures the server to use a temporary path for data, is it intended, or should it also use /nomad/data?

Side question: if we run both a binary with the server config and another binary with the client config, on the same server, both pointing to the same data dir, does it cause issues?

Auto-update tutorial example module version upon release

Run a GitHub Actions workflow upon a new version release to keep the tutorial content up-to-date.

on:
  release:
    types: [published]

Hardcoded Project Name Usage in VM Module Image Var

Hardcoded project names were accidentally included included. The image variable should use the format function with the hardcoded packer image name, and the required project variable for this module.

Consul configuration defines unused options in server and client

Client agents do not need to enable connect, this is only used on servers:

Enabling Connect requires changing the configuration of only your Consul servers (not client agents).

terraform-google-nomad/packer/configs/consul/client.hcl

Lines 30 to 32 in 0db159e

	connect {
	enabled = true
	}

Servers do not need to enable the gRPC port, this is only used on clients:

terraform-google-nomad/packer/configs/consul/server.hcl

Line 17 in 0db159e

grpc = 8502

There might be others, but these should definitely be removed.

Make the Packer Template User-friendly

Right now the Packer template has a hard-coded project_id. To make this more friendly to use, the template needs to be updated to include a user variable, and the README documentation should be adjusted.

Use managed instance groups for server and client deployments

It would probably be a good idea to use managed instance groups for Nomad server and client deployments in the future with the compute_instance_group_manager Terraform resource. GoogleCloudPlatform/terraform-google-managed-instance-group looks like a good starting point.

Need to look into how that works, implement it, and then document how to trigger rolling updates for Nomad.

Investigate providing a Firecracker task driver option

It would be interesting to provide an option to enable running a Firecracker task driver since GCP supports nested virtualization.

Supporting Documentation

• https://github.com/firecracker-microvm/firecracker/blob/main/docs/dev-machine-setup.md#gcp
• https://cloud.google.com/compute/docs/instances/nested-virtualization/overview
• https://www.nomadproject.io/docs/drivers/external/firecracker-task-driver

Falco Installation Broken

$ make packer/build
2021-05-30T11:51:05-04:00: ==> client: E: Failed to fetch https://dl.bintray.com/falcosecurity/deb/dists/stable/InRelease  403  Forbidden [IP: 54.185.54.139 443]
2021-05-30T11:51:05-04:00: ==> client: E: The repository 'https://dl.bintray.com/falcosecurity/deb stable InRelease' is not signed.

Add a Vault integration option

Support an option to deploy Vault with github.com/picatz/terraform-google-vault

Dependabot can't parse your locals.tf

Dependabot couldn't parse the locals.tf found at /locals.tf.

The error Dependabot encountered was:

unable to parse HCL: At 2:18: Unknown token: 2:18 IDENT base64encode

View the update logs.