hashicorp / terraform-google-terraform-enterprise

A Terraform module for deploying Terraform Enterprise on GCP.

Terraform Enterprise GCP Module

IMPORTANT: You are viewing a beta version of the official module to install Terraform Enterprise. This new version is incompatible with earlier versions, and it is not currently meant for production use. Please contact your Customer Success Manager for details before using.

This is a Terraform module for provisioning a Terraform Enterprise Cluster on GCP. Terraform Enterprise is our self-hosted distribution of Terraform Cloud. It offers enterprises a private instance of the Terraform Cloud application, with no resource limits and with additional enterprise-grade architectural features like audit logging and SAML single sign-on.

About This Module

This module will install Terraform Enterprise on GCP according to the HashiCorp Reference Architecture. This module is intended to be used by practitioners seeking a Terraform Enterprise installation which requires minimal configuration in the GCP cloud.

As the goal for this main module is to provide a drop-in solution for installing Terraform Enterprise via the Golden Path, it leverages GCP native solutions such as Cloud DNS and runs a basic Ubuntu 20.04 image on Cloud Compute Engine. We have provided guidance and limited examples for other use cases.

Pre-requisites

This module is intended to run in a GCP project with minimal preparation, however it does have the following pre-requisites:

1. Create a Cloud DNS zone for the DNS name you wish to use.
2. Create a managed SSL Certificate in Network Services -> Load Balancing to serve as the certificate for the FQDN.

Optionally a pre-existing service account may be specified with the existing_service_account_id variable to circumvent the need to manage IAM permissions within the module.

Provisioning the SSL certificate

The SSL certificate for the TFE load balancer is a pre-requisite for this module.

The certificate can be provisioned in GCP here either by creating a managed GCP certificate or by uploading an existing certificate.

For more information on provisioning certificates in GCP, read the documentation here.

Examples of creating a self-signed certificate for use with internal load balancers can be found within the examples.

Certificate Advice

• If you are deploying an instance with a private load balancer, the certificate needs to be a regional certificate. If you have a working certificate, you can upload this to GCP using the gcloud CLI command using the example below.

gcloud compute ssl-certificates create my-cert --certificate=fullchain.pem --private-key=privkey.pem --region=us-central1

where fullchain.pem and privkey.pem are paths to local files containing the relevant certificate material. Specify the region where you intend to deploy Terraform Enterprise. Note that these certificates will not be listed in the GCP UI, but are visible if you run

gcloud compute ssl-certificates list

• If you are deploying a public instance of TFE, the certificate should be a global certificate. Global certificates can be uploaded through the GCP UI (Network services > Load balancing > select advanced menu > Certificates > CREATE SSL CERTIFICATE) and will be visible both in the UI and also via the use of the above gcloud command.

Project Service APIs

The following sections list the APIs that will be enabled in general and for specific operation modes.

General

• Cloud Logging API
• Identity and Access Management API
• Service Networking API (unless an existing network is used)

External Services Mode

• Cloud SQL Admin API

Active/Active

• Memorystore for Redis API

Permissions

If you are using a service account to authenticate calls to Google Cloud APIs, the following roles are required:

• Basic Editor role
• Service Networking Admin
• Project IAM Admin
• Secret Manager Admin

How to Use This Module

• Ensure account meets module pre-requisites from above.

• Create a Terraform configuration that pulls in this module and specifies values of the required variables:

module "tfe_node" {
  source                = "[email protected]:hashicorp/terraform-google-terraform-enterprise.git"
  namespace             = "<Namespace to uniquely identify resources>"
  node_count            = "<Number of TFE nodes to provision>"
  tfe_license_secret_id = "<Secret Manager secret comprising license>
  fqdn                  = "<Fully qualified domain name>"
  ssl_certificate_name  = "<Name of the SSL certificate provisioned in GCP>"
  dns_zone_name         = "<Name of the DNS zone in which a record set will be created>"
}

• Run terraform init and terraform apply

Notes:

• The google-beta provider is required to create the subnetwork that is reserved for Internal HTTP(S) Load Balancing.
• If you are managing DNS outside of Cloud DNS:
  • Module will output resulting load balancer IP address as lb_address
  • You must configure an external DNS record using the lb_address output

Required inputs

is_replicated_deployment - If TFE will be installed using a Replicated license and deployment method

namespace - Namespace to uniquely identify resources. Used in name prefixes

dns_zone_name - Name of the DNS zone in which a record set will be created

fqdn - Fully qualified domain name

ssl_certificate_name - Name of the SSL certificate provisioned in GCP

node_count - Number of TFE nodes to provision. A number greater than 1 will enable Active/Active

Replicated Deployment

tfe_license_secret_id - The Secret Manager secret which comprises the Base64 encoded Replicated license file. The Terraform provider calls this value the secret_id and the GCP UI calls it the name.

Flexible Deployment Options - Docker

hc_license - The raw TFE license that is validated on application startup

registry_username - The username for the docker registry from which to source the terraform_enterprise container images

registry_password - The password for the docker registry from which to source the terraform_enterprise container images

Module Manifest

This module will create all infrastructure resources required to install Terraform Enterprise in a Standalone or Active/Active configuration (depending on how many nodes you specify) on GCP in the designated region according to the Reference Architecture. The default base machine image used is Ubuntu 20.04 LTS.

The resources created are:

• VPC with public and private subnets
• PostgreSQL instance
• Redis cache
• Google Storage Bucket
• Load Balancer
• Service Account for fetching storage objects
• Instances and Instance group manager

Examples

• Standalone
• Active/Active
• Existing Network
• Outbound Proxy
• SharedVPC

Note: If you destroy and recreate the infrastructure, you will need to update the A record in Cloud DNS with the new Load Balancer address.

License

This code is released under the Mozilla Public License 2.0. Please see LICENSE for more details.

terraform-google-terraform-enterprise's Issues

CI testing bug

Expected Behavior

This repo was created to have ephemeral TFC workspaces for testing.

Actual Behavior

At some point this broke so that the workspaces are cleaned up even if a destroy was not done.

Steps to Reproduce

Use the slash command to run /test <scenario> destroy=false. The workspace will delete before you have a chance to destroy your resources, leaving you with orphaned GCP resources to clean up.

Allow using non-us object storage buckets

terraform-google-terraform-enterprise/modules/object_storage/main.tf

Line 7 in 565a77e

location = "us"

Adding additional permission to the `service_account` that gets attached to the vm.

How do I add additional permission to the service account attached to the VM?

Custom configurations to exclude LB & DNS

The module should be flexible to allow the user to setup TFE with custom LB URL and not rely on Google managed services.

Module does not meet TFE Network Requirements - MTU

This issue is in direct response to a ticket filed with support.

The TFE network requirements docs details the appropriate MTU values for GCP: https://www.terraform.io/enterprise/requirements/network#other-configuration

For GCP only: Configure Docker to use an MTU (maximum transmission unit) of 1460, as required by Google (GCP Cloud VPN Documentation: MTU Considerations).
To configure Docker's MTU, create an /etc/docker/daemon.json file with the following content:

{
 "mtu": 1460
}

It also details:

Note: Beginning in version v202004-1, non-default Docker networks named tfe_services and tfe_terraform_isolation were added for the Terraform Enterprise component Docker containers as part of a network segmentation update. Custom configuration may be required for MTU settings.

This links to a support knowledgebase article with the following key commands: https://support.hashicorp.com/hc/en-us/articles/4405507244691

docker network create tfe_terraform_isolation --subnet=172.20.0.0/16 -o com.docker.network.driver.mtu=1460
docker network create tfe_services --subnet=172.19.0.1/16 -o com.docker.network.driver.mtu=1460

Unfortunately, this module does not account for this requirement and causes intermittent 503s and other odd issues on customer infrastructure.

This issue based on an earlier revision of this project, where this consideration was also forgotten and caused customer support tickets to be filed: https://github.com/hashicorp/is-terraform-google-tfe-standalone/pull/6

Replicated settings are not applied

While performing cluster setup, the installation completes but doesnt apply the settings from replicated.conf. The script prompts to continue installation through UI where we need to pass the settings again.

Airgap File Version: v201911-1

Final Output:
To access the cluster with kubectl, reload your shell:

bash -l

To continue the installation, visit the following URL in your browser:

http://10.156.0.74:8800

TEST - GCP

Expected Behavior

Actual Behavior

Steps to Reproduce

1. terraform apply

Important Factoids

References

• #0000

GitHub Actions - deprecated warnings found - action required!

Workflow Name: Create Jira Issue
Branch: main
Run URL: https://github.com/hashicorp/terraform-google-terraform-enterprise/actions/runs/4855843469

save-state deprecation warnings: 0
set-output deprecation warnings: 1
node12 deprecation warnings: 0

Please review these deprecation warnings as soon as possible and merge in the necessary updates.

GitHub will be removing support for these commands and plan to fully disable them on 31st May 2023. At this time, any workflow that still utilizes these commands will fail. See https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/.

GitHub have not finalized a date for deprecating node12 yet but have indicated that this will be summer 2023. So it is advised to switch to node16 asap. See https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/.

If you need any help, please reach out to us in #team-rel-eng.

Parameters in metadata exposes secrets

As the GCP instance reads the metadata on the fly for parameters, the credentials and keys are exposed in console if we use this registry.

setup is not running

The "main" branch has version which is creating infrastructure, but doesn't setup the service.

Encryption password must be set or install fails on RHEL

Issue

The default value for encryption_password is set to null. If no value is passed, the install will error with startup-script: INFO startup-script: cat: /etc/ptfe/encpasswd: No such file or directory startup-script: INFO startup-script: Return code 1. startup-script: INFO Finished running startup scripts.

Workaround

Pass a value for encryption_password to avoid this issue on RHEL.

Effects

RHEL on GCP with no value set for encryption_password.

This issue does not exist on Ubuntu.

Metrics support (gcp)

What problem would you like to address?

Currently there is no configuration for the load balancers or security groups for the metrics ports to be open.

Who is the audience for this request?

all

What would the ideal solution look like?

Here is the Azure solution.
hashicorp/terraform-azurerm-terraform-enterprise#214

What alternatives have you considered?

n/a

How does this impact you?

This is an important customer feature that we need to not manually configuring.

Are there any deadlines?

n/a

Is there any additional context?

Implement check for authorized users of slash command dispatcher

What problem would you like to address?

Currently for someone to run the slash commands (i.e. /test all), the user must be admin of the repo. This security posture is not preferred.

Who is the audience for this request?

TFE engineers / maintainers of this repo

What would the ideal solution look like?

Implement a conditional that is a check of authorized users for the slash command dispatcher action, like done here

What alternatives have you considered?

What we're doing now

How does this impact you?

Currently we have to add everyone as a user and remember to remove admins when they're offboarded.

Are there any deadlines?

Is there any additional context?

create variable to increase autohealing timeout for release candidate instances

What problem would you like to address?

When we created the release candidate instance, it failed during an upgrade, however, we weren't able to gather logs before the instance group rolled the unhealthy instance. (slack)

Who is the audience for this request?

everyone who wants more granular control over their instance group

What would the ideal solution look like?

The settings here would need to be fine tuned to better account for this need. At the very least, the 600 initial_delay_sec can be raised to 3600 (which is the max)

What alternatives have you considered?

How does this impact you?

release testing is compromised

Are there any deadlines?

by the next release would be nice

Is there any additional context?

broken examples links in readme

4 of the links under https://github.com/hashicorp/terraform-google-terraform-enterprise#examples are broken

[Standalone](https://github.com/hashicorp/terraform-google-terraform-enterprise/blob/main/examples/standalone)
[Active/Active](https://github.com/hashicorp/terraform-google-terraform-enterprise/blob/main/examples/active-active)
[Existing Network](https://github.com/hashicorp/terraform-google-terraform-enterprise/blob/main/examples/existing-network)
[Outbound Proxy](https://github.com/hashicorp/terraform-google-terraform-enterprise/blob/main/examples/proxy)
[SharedVPC](https://github.com/hashicorp/terraform-google-terraform-enterprise/blob/main/examples/shared-vpc)

the examples folder contains 6 examples not 5 https://github.com/hashicorp/terraform-google-terraform-enterprise/tree/main/examples

it's not immediately obvious which link should now go to which link to provide a quick pull request.

Support for passing provider for `dns_zone_name`

Our DNS zone lives in a different gcp project then where we would like to deploy tfe. This means we need a different provider when querying for the dns zone.

Is it possible to add support for this?

Unable to use module when CloudDNS not in the same project

terraform-google-terraform-enterprise/primary.tf

Line 53 in 26f0457

data "google_dns_managed_zone" "dnszone" {

This won't work if the DNS zone is not part of the credentials default project. Generally this will always be the case in something like a SharedVPC model.