jsiebens / nomad-droplets-autoscaler

DigitalOcean Droplets target plugin for HashiCorp Nomad Autoscaler

Nomad DigitalOcean Droplets Autoscaler

The do-droplets target plugin allows for the scaling of the Nomad cluster clients via creating and destroying DigitalOcean Droplets.

Requirements

• nomad autoscaler 0.3.0+
• DigitalOcean account

Documentation

Agent Configuration Options

To use the do-droplets target plugin, the agent configuration needs to be populated with the appropriate target block. Currently, Personal Access Token (PAT) is the only method of authenticating with the API. You can manage your tokens at the DigitalOcean Control Panel Applications Page.

target "do-droplets" {
  driver = "do-droplets"
  config = {
    token = "local/token"
  }
}

• token (string: "") - a DigitalOcean API token or a path to a file containing a token. Alternatively, this can also be specified using environment variables ordered by precedence:
  • DIGITALOCEAN_TOKEN
  • DIGITALOCEAN_ACCESS_TOKEN

Policy Configuration Options

check "hashistack-allocated-cpu" {
  # ...
  target "do-droplets" {
    name                = "hashi-worker"
    region              = "nyc1"
    size                = "s-1vcpu-1gb"
    snapshot_id         = 84589509
    user_data           = "local/hashi-worker-user-data.sh"
    tags                = "hashi-stack"
    node_class          = "hashistack"
    node_drain_deadline = "5m"
    node_purge          = "true"
  }
  # ...
}

• name (string: <required>) - A logical name of a Droplet "group". Every managed Droplet will be tagged with this value and its name is this value with a random suffix

• region (string: <required>) - The region to start in.

• vpc_uuid (string: <required>) - The ID of the VPC where the Droplet will be located.

• size (string: <required>) - The unique slug that indentifies the type of Droplet. You can find a list of available slugs on DigitalOcean API documentation.

• snapshot_id (string: <required>) - The Droplet image ID.

• user_data (string: "") - A string of the desired User Data for the Droplet or a path to a file containing the User Data

• ssh_keys (string: "") - A comma-separated list of SSH fingerprints to enable

• tags (string: "") - A comma-separated list of additional tags to be applied to the Droplets.

• datacenter (string: "") - The Nomad client datacenter identifier used to group nodes into a pool of resource. Conflicts with node_class.

• node_class (string: "") - The Nomad client node class identifier used to group nodes into a pool of resource. Conflicts with datacenter.

• node_drain_deadline (duration: "15m") The Nomad drain deadline to use when performing node draining actions. Note that the default value for this setting differs from Nomad's default of 1h.

• node_drain_ignore_system_jobs (bool: "false") A boolean flag used to control if system jobs should be stopped when performing node draining actions.

• node_purge (bool: "false") A boolean flag to determine whether Nomad clients should be purged when performing scale in actions.

• node_selector_strategy (string: "least_busy") The strategy to use when selecting nodes for termination. Refer to the node selector strategy documentation for more information.

nomad-droplets-autoscaler's Issues

Plugin is trying to check the cluster status after draining the nodes

Hi @jsiebens

First of all, thank you so much for creating this plugin.

We are testing this plugin to scale droplets in DigitalOcean. The droplet is getting scaled as per the defined policy but after scaling down and draining the node the plugin starts giving the below warning continuously. The plugin is actually trying to monitor the health of the drained droplet and also the droplet is not getting removed from DigitalOcean.

2022-09-30T13:55:08.537Z [WARN] external_plugin.do-droplets: node pool status readiness check failed: error="node c92595ac-08f2-7686-75fc-5fb90e29f478 is ineligible" timestamp=2022-09-30T13:55:08.537Z

Scaling policy:

scaling "app-redis-slave" {
  enabled = true
  min     = 1
  max     = 2

  policy {
    cooldown            = "1m"
    evaluation_interval = "10s"

    check "redis_jobs_in_progess" {
      source = "prometheus"
      query  = "sum(nomad_nomad_job_summary_queued{exported_job=~\"app\",task_group=\"redis-slave\"} + nomad_nomad_job_summary_running{exported_job=~\"app\",task_group=\"redis-slave\"}) OR on() vector(0)"

      strategy "pass-through" {}
    }

    target "do-droplets" {
      name = "app-redis-slave"
      region = "xxx1"
      size = "s-1vcpu-1gb"
      snapshot_id = "XXXXXX"
      user_data = "local/do-user_data.sh"
      tags = "hashi-stack,job:app,group:redis-slave"

      datacenter             = "do"
      node_drain_deadline    = "15m"
      node_drain_ignore_system_jobs = true
      node_selector_strategy = "empty_ignore_system"
    }
  }
}