splunk / terraform-provider-victorops

Manage VictorOps teams, users, escalation policies and routing keys using Terraform

Terraform VictorOps Provider

• Website: https://www.terraform.io
• 
• Mailing list: Google Groups

Requirements

• A VictorOps account that you want to manage alongwith API key and token.
• Terraform 0.10.x or higher
• Go 1.14 (to build the provider plugin)

Building The Provider

Clone repository to: $GOPATH/src/github.com/splunk/terraform-provider-victorops

$ git clone [email protected]:splunk/terraform-provider-victorops.git $GOPATH/src/github.com/splunk/terraform-provider-victorops

Enter the provider directory and build the provider

$ cd $GOPATH/src/github.com/splunk/terraform-provider-victorops

# MacOC
$ go build

# Ubuntu
$ make build
$ cp $GOPATH/bin/terraform-provider-victorops .

Features

Using this VictorOps Terraform provider, you can manage the following VictorOps resources.

1. User
2. Team
3. User-Team assignment
4. Escalation Policy
5. Routing Key

Usage

terraform {
	required_providers {
		victorops = {
			source = "splunk/victorops"
			version = "0.1.1"
		}
	}
}

provider "victorops" {
  api_id  = "6d700de8"   // An API id tied to an admin user
  api_key = "<REDACTED>" // An API key tied to an admin user
}

// Define the first tf-configured user, 'John Dane'
resource "victorops_user" "jdane_tf" {
  first_name       = "John"
  last_name        = "Dane"
  user_name        = "jdane"
  email            = "[email protected]"
  is_admin         = false // deprecated - We no longer support creating admin users through TF/public APIs. The value in this field is ignored.
  replacement_user = "myDefaultVOUser" // optional
  // Specify this with the default username to replace all users when deleting users using TF
}

// Create a new team
resource "victorops_team" "team_vikings" {
  name = "VO-Vikings"
}

// Assigning an existing user to a team
resource "victorops_team_membership" "jdane_membership" {
  team_id          = victorops_team.team_vikings.id
  user_name        = victorops_user.jdane_tf.user_name
}

// Create escalation policies for existing VO rotation (created using portal)
// Note: You need to fetch an existing Rotation Group Slug for using the Escalation Policy
resource "victorops_escalation_policy" "high_severity" {
  name    = "High Severity"
  team_id = victorops_team.team_vikings.id
  step {
    timeout = 60
    entries = [
      {
        type = "rotationGroup"
        slug = "rtg-wvvhXshpvaRdn7jM"
      }
    ]
  } 
}

// Create routing keys to push alerts to our escalation policies
resource "victorops_routing_key" "viking_high_severity" {
  name = "viking-high-severity"
  targets = [victorops_escalation_policy.high_severity.id]
}

Developing the Provider

If you wish to work on the provider, you'll first need Go installed on your machine (version 1.14+ is required). You'll also need to correctly setup a GOPATH, as well as adding $GOPATH/bin to your $PATH.

To compile the provider, build the provider as mentioned above. In order to test the provider, you can simply run make test.

$ make test

In order to run the full suite of Acceptance tests, run make testacc.

Acceptance tests require the following environment variables to be set.

• VO_API_ID
• VO_API_KEY
• VO_BASE_URL
• VO_REPLACEMENT_USERNAME
  • the default username to replace all users when removed

$ make testacc

terraform-provider-victorops's Issues

Error: Missing Authentication Token

Testing with

export VO_API_ID="rhfgh"
export VO_API_KEY="fghgfh"
export VO_BASE_URL="https://alert-mapping.victorops.com/integrations/azure/v1/xxxxxxxxxx"
# VO_REPLACEMENT_USERNAME 

terraform apply --auto-approve

Failing with a simple test:

resource "victorops_routing_key" "routing_key" {
  name = "damien-test"
  targets = []
}

$ ./test.sh
victorops_routing_key.routing_key: Creating...

Error: failed to create routing key (403): {"message":"Missing Authentication Token"}

  on main.tf line 15, in resource "victorops_routing_key" "routing_key":
  15: resource "victorops_routing_key" "routing_key" {

Updating escalation policies does not work

Updating an escalation policy, specifically hit this when modifying ignore_custom_paging_policies, the routing key resource is forced replacement because the policy itself is forced to be replaced. I'm not sure this is necessary and just done cleanly when doing it through the UI, but in this case we hit:

module.victorops.victorops_routing_key.my_routing_key: Destroying... [id=My-thing]
Error: deleting routing keys not yet implemented in the API, please delete in the UI

Would be nice to have a heads up about this in the documentation, or to remove the limitation in the provider. I've hit similar issues when trying to update a user email address, in that case the resource isn't marked for replacement but we still hit an issue and have to perform the update in the UI. Any guidance on reporting these issues individually would be nice as we're considering if we should keep using the provider or reverting back to UI-only.

Reading & Writing too quickly - {"message":"Too Many Requests"}

The Problem

Terraform can't run, as it is calling too many routing keys at the same time. Sometimes it works, other times it will fail. I've noticed this effects the plan as well as the apply. Sometimes it can't refresh the state, other times it skips creation of new routing keys. Sometimes the plan is not accurate, as it missed reading the correct routing keys etc.. This was not a problem when we had fewer routing keys and escalation policies (less than 25) however this has slowly became a bigger and bigger issue as we add more policies and routing keys. (We are well over 100 now)

This is the exact error:

module.victorops_REDACTED_team.victorops_routing_key.s1_routing_key: Refreshing state... [id=REDACTED_s1]
module.victorops_REDACTED_team.victorops_routing_key.s2_routing_key: Refreshing state... [id=REDACTED_s2]
module.victorops_REDACTED_team.victorops_routing_key.s3_routing_key: Refreshing state... [id=REDACTED_s3]
module.victorops_REDACTED_team.victorops_routing_key.s4_routing_key: Refreshing state... [id=REDACTED_s4]
module.victorops_REDACTED_team.victorops_routing_key.s1_routing_key: Refreshing state... [id=REDACTED_s1]
module.victorops_REDACTED_team.victorops_routing_key.s2_routing_key: Refreshing state... [id=REDACTED_s2]
module.victorops_REDACTED_team.victorops_routing_key.s3_routing_key: Refreshing state... [id=REDACTED_s3]
module.victorops_REDACTED_team.victorops_routing_key.s4_routing_key: Refreshing state... [id=REDACTED_s4]

│ Error: failed to get escaltion policy (429): {"message":"Too Many Requests"}

My suspect reason to why

This provider looks to be calling the read for each routing key with no sort of throttle/limit, however the API docs for routing keys says it has a limit of 2 call per second.

Retrieves a list of routing keys and associated teams. This API may be called a maximum of 2 times per second

It fails when trying to refresh many routing key(s) at roughly the same time.

Provider does Not Support Import

The provider does not allow state import to terraform.

Error: resource victorops_team doesn't support import

Any chance this could be implemented?

Provider "victorops" not available for installation.

I can't get the provider to simply install with no resources being called. (I've also tested with resources and not had any luck).

I've copied the base template from both the github repo as well as VictorOps.

Do I have to install this manually first? I got this working on my personal computer running a later version of Terraform v0.14.2. Trying to get it working in the work environment v0.12.24 with no luck. Could this be the issue? or is there something else I can try/missing?

terraform {
	required_providers {
		victorops = {
			source = "splunk/victorops"
			version = "0.1.1"
		}
	}
}

provider "victorops" {
  api_id  = "6d700de8"   // This is a global Enviroment Varialbe defined in gitlab 
  api_key = "apikeytest" // This is a global Enviroment Varialbe defined in gitlab 
}


// Create a new team
resource "victorops_team" "team_vikings" {
  name = "VO-Vikings"
}

Implement Datasources

It would be good to have datasources defined for the available resources so we can automate addition of users to teams etc.
The Pager Duty provider has this sort of functionality.

Apple Silicon not supported

Unable to run this on Apple Silicon.

│ Error: Incompatible provider version
│ Provider registry.terraform.io/splunk/victorops v0.1.1 does not have a package available for your current platform, darwin_arm64.
│ Provider releases are separate from Terraform CLI releases, so not all providers are available for all platforms. Other versions of this provider may have different platforms supported.

Error: Failed to query available provider packages

I'm attempting to use modules to create resources. When referencing the module, it doesn't seem to be able to check the version option and gives me an error.

This is the error I'm getting.

Error: Failed to query available provider packages
Could not retrieve the list of available versions for provider
hashicorp/victorops: provider registry registry.terraform.io does not have a
provider named registry.terraform.io/hashicorp/victorops
Cleaning up file based variables

This is the main.tf file

# Install VictorOps Terraform Provider  
terraform {
 required_version = ">= 0.14.2"
}


terraform {
 required_providers {
 victorops = {
  source = "splunk/victorops"
  version = "~> 0.1.1" 
 }
 }
}
 provider "victorops" { 
    api_id  = "<REDACTED>"
  #  api_key = "api_key" Defined in gitlab
 }

If I add the terraform and provider info within the module itself, it seems to be able to work just fine.

I've tested.

• Removing the version completely, and attempted adding ~> or the specific version etc...
• Adding the team directly within the code works fine (just can't do it through a module)

If I add the the terraform section within the module itself and it seems to work just fine. Not sure if this is suppose to be like this, but I feel its kind of weird duplicating the code in both the main.tf file as well as the modules maint.tf.

Provider documentation is out of date

Documentation for resource victorops_escalation_policy is out of date.
Current documentation (in GitHub and on Terraform) only shows 2 possible values for victorops_escalation_policy.step.entries[].type (rotationalGroup, targetPolicy):

step {
       timeout = [time-out duration in seconds]
       entries = [
           {
               type = [ rotationalGroup | targetPolicy ]
               slug = [ rotatioGroup slug | next escalation policy ID ]
           },
       ]
   }

Code in github indicates that there are 7 values accepted here: "user", "email", "rotationGroup", "rotationGroupNext", "rotationGroupPrevious", "webhook", "targetPolicy".

Adding multiple users/teams and mapping them

Is there any way to create multiple users/teams and then add specific users to specific team/s?

I tried to use different methods like for_each but couldn't achieve this.

Support Rotation policy resource for seamless terraform workflow

Describe the problem

This provider exists to make managing Splunk easier, which is great! We are using this to create teams in victorops. Although the docs say:

// Create a new team
resource "victorops_team" "team_vikings" { ...

// Create escalation policies for existing VO rotation (created using portal)
// Note: You need to fetch an existing Rotation Group Slug for using the Escalation Policy
resource "victorops_escalation_policy" "high_severity" {
    ...
    entries = [
      {
        type = "rotationGroup"
        slug = "rtg-wvvhXshpvaRdn7jM"
      }
    ]
  } 
}

// Create routing keys to push alerts to our escalation policies
resource "victorops_routing_key" "viking_high_severity" { ...

This theoretically looks fine. But in reality, one cannot create rotation group without creating the team first.

So the actual workflow becomes like this:

• First commit terraform code with just the team
• Then go to the team, create a rotation, note it's rotation ID
• Then come back to terraform, commit new code for the routing key with the rotation ID

If we're trying to manage 10 teams in VictorOps or keeping them in-sync with a team directory, the workflow becomes a constant back and forth of broken jobs and partial commits, and having to coordinate across multiple teams whether or not they have finished creating their groups and so on.

What would be good is just to have the basic flow working without breaks in the middle, i.e. have a routing key and escalation policy that we can start using everywhere, and let the teams to configure their shifts in the rotation at their own pace.

Given

1. Given that the objective of this terraform provider is to have a good automation workflow, unfortunately this middle-of-the-road human blocker for rotation groups is making the automation extremely fragile. Instead of messing around with 2 or 3 commits, it's just more straight forward to do all the clicks in VictorOps itself, undermining the usefulness of this effort.

2. "Creating a Rotation Policy" is a harmless operation. It does not hurt on-call. Rotation groups are always created with empty members. VictorOps tells this clearly.

During creation, the Rotation policy is just pure metadata (shift type, etc). It does not need any user for creation. Users can be selected later after creation.

Request

• To have ability to create just an empty Rotation policy – with no users assigned. This is the real workflow in the UI as well.
• This unblocks and gives us a seamless terraform workflow. We can create escalation policies and routing keys, and we can setup the whole chain. Teams will only have to populate their shift in the UI offline, independent of terraform. That too, it's fully optional, if they don't, VictorOps does not hurt or change any on-call. The empty rotation policy unblocks the rest of the terraform resources in one shot. We can start plugging in the routing keys into Grafana and emails.
• This also won't hurt any larger organizational requirements. Since escalation policies are chainable, we can always chain the terraform-created policy with a higher level "default" escalation policies.

TL;DR:

Given the objective of this plugin is to make automation smooth for Terraform, having such a cumbersome, manual failure point on creating Rotation groups makes the workflow very flaky, with hard rollbacks, multi-commits, copying bits and pieces here and there, constantly having to intervene. In practical real life, it's easier to just do everything in the UI, instead of having to go back and forth.

Being able to create an empty Rotation Policy with no members will make the terraform workflow very smooth and unblock the entire terraform experience.

Thank you for your patience to read this through.

Provider produced inconsistent result after apply

Hello,

I am using the available resources - https://registry.terraform.io/providers/splunk/victorops/latest/docs

I was just hoping to get some assistance with the error below. Thank you.

Error:
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to victorops_team_membership.membership["team_name.username"], provider
│ "provider["registry.terraform.io/splunk/victorops"]" produced an
│ unexpected new value: Root resource was present, but now absent.
│
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker

Error: Provider produced inconsistent final plan

Hi Team,

I am using all the available resources - https://registry.terraform.io/providers/splunk/victorops/latest/docs

Look like there is bug with team_membership: https://registry.terraform.io/providers/splunk/victorops/latest/docs/resources/team_membership

Error:

│ Error: Provider produced inconsistent final plan

│ When expanding the plan for victorops_team_membership.team_membership[0] to include new values learned so far during apply, provider "registry.terraform.io/splunk/victorops" produced an invalid new value for .user_name: was cty.StringVal("xxx.yyy"), but now cty.StringVal("xxx.yyy").

│ This is a bug in the provider, which should be reported in the provider's own issue tracker.