googlecloudplatform / cloud-foundation-fabric

End-to-end modular samples and landing zones toolkit for Terraform on GCP.

Terraform Examples and Modules for Google Cloud

This repository provides end-to-end blueprints and a suite of Terraform modules for Google Cloud, which support different use cases:

• organization-wide landing zone blueprint used to bootstrap real-world cloud foundations
• reference blueprints used to deep dive into network patterns or product features
• a comprehensive source of lean modules that lend themselves well to changes

The whole repository is meant to be cloned as a single unit, and then forked into separate owned repositories to seed production usage, or used as-is and periodically updated as a complete toolkit for prototyping. You can read more on this approach in our contributing guide, and a comparison against similar toolkits here.

Organization blueprint (Fabric FAST)

Setting up a production-ready GCP organization is often a time-consuming process. Fabric FAST aims to speed up this process via two complementary goals. On the one hand, FAST provides a design of a GCP organization that includes the typical elements required by enterprise customers. Secondly, we provide a reference implementation of the FAST design using Terraform.

Modules

The suite of modules in this repository is designed for rapid composition and reuse, and to be reasonably simple and readable so that they can be forked and changed where the use of third-party code and sources is not allowed.

All modules share a similar interface where each module tries to stay close to the underlying provider resources, support IAM together with resource creation and modification, offer the option of creating multiple resources where it makes sense (eg not for projects), and be completely free of side-effects (eg no external commands).

The current list of modules supports most of the core foundational and networking components used to design end-to-end infrastructure, with more modules in active development for specialized compute, security, and data scenarios.

Currently available modules:

• foundational - billing account, Cloud Identity group, folder, service accounts, logging bucket, organization, project, projects-data-source
• process factories - project factory
• networking - DNS, DNS Response Policy, Cloud Endpoints, address reservation, NAT, VLAN Attachment, External Application LB, External Passthrough Network LB, External Regional Application Load Balancer, Firewall policy, Internal Application LB, Cross-region Internal Application LB, Internal Passthrough Network LB, Internal Proxy Network LB, IPSec over Interconnect, VPC, VPC firewall, VPC peering, VPN dynamic, HA VPN, VPN static, Service Directory, Secure Web Proxy
• compute - VM/VM group, MIG, COS container (coredns, mysql, onprem, squid), GKE cluster, GKE hub, GKE nodepool, GCVE private cloud
• data - Analytics Hub, BigQuery dataset, Bigtable instance, Dataplex, Dataplex DataScan, Cloud SQL instance, Data Catalog Policy Tag, Data Catalog Tag, Data Catalog Tag Template, Datafusion, Dataproc, GCS, Pub/Sub, Dataform Repository
• development - API Gateway, Apigee, Artifact Registry, Container Registry, Cloud Source Repository, Workstation cluster
• security - Binauthz, KMS, SecretManager, VPC Service Control
• serverless - Cloud Function v1, Cloud Function v2, Cloud Run, Cloud Run v2

For more information and usage examples see each module's README file.

End-to-end blueprints

The blueprints in this repository are split into several main sections: networking blueprints that implement core patterns or features, data solutions blueprints that demonstrate how to integrate data services in complete scenarios, cloud operations blueprints that leverage specific products to meet specific operational needs, and factories that implement resource factories for the repetitive creation of specific resources, and finally GKE, serverless, and third-party solutions design blueprints.

cloud-foundation-fabric's Issues

Add tiny proxy to shared-vpc-gke example

Installing tiny proxy on the bastion host allows to user kubectl on the developer machine and access GKE masters through the bastion host, leveraging IAP.

Extend the gke-cluster module to support shared VPC configuration

Currently, the gke-cluster module does not appear to allow specifying a host project ID and shared VPC network.

 module "gke-dev-cluster" {
  source                    = "github.com/terraform-google-modules/cloud-foundation-fabric/modules/gke-cluster"
  name                      = "gke-dev-cluster"
  project_id                = "gke-dev-3291"
  location                  = local.location
  network                   = local.network
  subnetwork                = local.gke_subnetwork
  secondary_range_pods      = "pods"
  secondary_range_services  = "services"
  default_max_pods_per_node = 32

  peering_config = {
     import_routes = true
     export_routes = false
  }
  
  ....

}

As you can see in the configuration above, when using the peering_config, one can only specify a single project_id and network, which in turn is used by both the GKE cluster and google_compute_network_peering_routes_config as per this code:

modules/gke-cluster/main.tf

resource "google_container_cluster" "cluster" {
  ...
  project                     = var.project_id
   ...
  network                     = var.network
  ...
}
....
resource "google_compute_network_peering_routes_config" "gke_master" {
  ...
  project              = var.project_id
  ...
  network              = element(reverse(split("/", var.network)), 0)
  ...
}

In a Shared VPC scenario where the network is shared by host project, one would need to specify host project id and the shared vpc so the peering is setup on that host project.

To make this possible, the following proposes adding configuration block (peering_config_shared_vpc) like the following, which will allow specifying a host project id and a shared vpc:

module "gke-dev-cluster" {
  source                    = "github.com/terraform-google-modules/cloud-foundation-fabric/modules/gke-cluster"
  name                      = "gke-dev-cluster"
  project_id                = "gke-dev-3291"
  location                  = local.location
  network                   = local.network
  subnetwork                = local.gke_subnetwork
  secondary_range_pods      = "pods"
  secondary_range_services  = "services"
  default_max_pods_per_node = 32

  peering_config = {
     import_routes = true
     export_routes = false
  }

  peering_config_shared_vpc = {
     network    = "gke-dev-shared-vpc"
     project_id = "gke-dev-shared"
  }
  ...
}

I'm planning to submit a PR with the proposed solution. Of course, all open to discussion and changes. :)

New OKD upstream

oc patch clusterversion/version --patch '{"spec":{"upstream":"https://amd64.origin.releases.ci.openshift.org/graph"}}' --type=merge

Support injecting labels in OCP machinesets

 spec:
      metadata: {}
      providerSpec:
        value:
          machineType: e2-standard-2
          gcpMetadata:
            - key: acronimo-from-machineset
              value: CONC0
          userDataSecret:
            name: worker-user-data
          deletionProtection: false
          networkInterfaces:
            - network: vpc-1
              subnetwork: subnet-eu-west-3
          credentialsSecret:
            name: gcp-cloud-credentials
          zone: europe-west3-b
          canIPForward: false
          metadata:
            creationTimestamp: null
          projectID: ocp-isolated
          region: europe-west3
          kind: GCPMachineProviderSpec
          disks:
            - autoDelete: true
              boot: true
              image: >-
                projects/rhcos-cloud/global/images/rhcos-47-83-202102090044-0-gcp-x86-64
              labels:
                acronimo: pippo
              sizeGb: 128
              type: pd-ssd
          tags:
            - gcpwepoc002l-pxlkq-worker
          apiVersion: gcpprovider.openshift.io/v1beta1
          labels:
            acronimo: pippo

Call to function "list" in ../../modules/gke-cluster/main.tf failed

I was trying out the example in .../networking/hub-and-spoke-peering using Terraform (v0.15.3) from within Cloud Shell.

The plan command returned the following error:

Error: Error in function call
│
│   on ../../modules/gke-cluster/main.tf line 105, in resource "google_container_cluster" "cluster":
│  105:     for_each = length(var.master_authorized_ranges) == 0 ? [] : list(var.master_authorized_ranges)
│     ├────────────────
│     │ var.master_authorized_ranges is map of string with 3 elements
│
│ Call to function "list" failed: the "list" function was deprecated in Terraform v0.12 and is no longer available; use tolist([ ... ]) syntax to write a literal list.

Going by the last line in the error message, it appears that the error occurred because I'm using a version that's higher than 0.12. But, interestingly, the same example worked fine when I tested from an off-cloud host that has Terraform v0.14.11.

Add support for tags

• google_tags_tag_binding
• google_tags_tag_key
• google_tags_tag_key_iam
• google_tags_tag_value
• google_tags_tag_value_iam

Copyright headers are outdated (still report 2019)

Module gke-cluster misses master_global_access_config option

The module cke-cluster misses the option private_cluster_config.master_global_access_config which allows to reach private master endpoints from other regions from where the masters live

Additive IAM for folder

On #181 we introduced the ability to fully authoritative define the IAM policy of an organization. It would be nice to do the same for folders.

Fix static access identity in GCS to BQ end to end example

The BQ dataset access identity has a static reference to caggioland.com, can we replace this with a variable, or remove entirely if it's not mandatory to have it?

I'ts here

https://github.com/terraform-google-modules/cloud-foundation-fabric/blob/5ae489f50d29126772ad056727763717f299fb24/data-solutions/gcs-to-bq-with-dataflow/main.tf#L304

example infrastructure/shared-vpc: failing to provision

I tried to launch without success the example infratructure/shared-vpc

First issue is 3 errors like this one:

Error: "enable_flow_logs": [REMOVED] This field is being removed in favor of log_config. If log_config is present, flow logs are enabled. Please remove this field

  on .terraform/modules/net-vpc-host/terraform-google-modules-terraform-google-network-e273b5e/main.tf line 51, in resource "google_compute_subnetwork" "subnetwork":
  51: resource "google_compute_subnetwork" "subnetwork" {

Passed by forcing the Google provider to pre-3.0

provider "google" {
  version = "~> 2.0"
}
provider "google-beta" {
  version = "~> 2.0"
}

Now bumping on the following error:

Error: Error setting billing account "xxxxxx-xxxxxx-xxxxxx" for project "projects/testvpc3-gce": googleapi: Error 400: Precondition check failed., failedPrecondition

  on .terraform/modules/project-service-gce/terraform-google-modules-terraform-google-project-factory-46e55ae/modules/fabric-project/main.tf line 30, in resource "google_project" "project":
  30: resource "google_project" "project" {

It's the same issue I couldn't solve for the tests of terraform-google-bootstrap.

Gave the "billing account user" role to the terraform service account, but didn't solve it.

Do you think I could have missed something or is it by "chance" a known issue? :)

Convert log sinks module to single resource management

Support for random project id?

Under the foundations>environments without specifying a random suffix in the project name there is a chance that the project will not be created if it is not unique. Can we add the random_project_id variable or am I just missing something?

Add examples to log sink module documentation

Use the log sink configuration in the foundation examples as a starting point.

project creation fails if skipDefaultNetworkCreation org policy is enforced

Might be related to hashicorp/terraform-provider-google#3582

https://github.com/terraform-google-modules/cloud-foundation-fabric/blob/479be4d47d27440a2de6066acfbbc2300aa1a7cf/modules/project/main.tf#L42 #

Clarify vpc output key formats

The current vpc output documentation doesn't specify the format of the key to pull data out of the subnet_regions, subnet_secondary_ranges and subnet_self_links maps.
It would be good for users to know the exact key.

delete

Yow! It's some people inside the wall! This is better than mopping!

I'm interested in this also ... bootstrapping a new GCP account coming from AWS with a fairly complex org. The examples in [https://github.com/terraform-google-modules/cloud-foundation-fabric] are great for organisational structure but they bootstrap terraform without using this module. This module seems preferable for getting a clean start to an account and observing good practices with regard to iam roles and groups etc.

I'm interested in this also ... bootstrapping a new GCP account coming from AWS with a fairly complex org. The examples in [https://github.com/terraform-google-modules/cloud-foundation-fabric] are great for organisational structure but they bootstrap terraform without using this module. This module seems preferable for getting a clean start to an account and observing good practices with regard to iam roles and groups etc.
I'm particularly looking at business unit example in the fabric repo.

Originally posted by @gilesw-vion in terraform-google-modules/terraform-google-bootstrap#11 (comment)

iam-service-account IAM folders dependency

I am trying to bind IAM roles to a folder which is created dynamically but it looks like it's failing due a dependency even if I explicitly add depends_on on my code. Here is an example:

module "my-sa" {
  source            = "../modules/iam-service-account"
  project_id      = "abc"
  name              = "my-sa"
  generate_key      = false

  iam_folder_roles = {
    (module.my_folder.id) = var.tf_sa_roles
  }

module.my_folder is basically creating a folder (with no IAM bindings). When I run this, I get the error:

│ Error: Invalid for_each argument
│ 
│   on ../modules/iam-service-account/main.tf line 107, in resource "google_folder_iam_member" "folder-roles":
│  107:   for_each = {
│  108:     for pair in local.iam_folder_pairs :
│  109:     "${pair.entity}-${pair.role}" => pair
│  110:   }
│     ├────────────────
│     │ local.iam_folder_pairs will be known only after apply
│ 
│ The "for_each" value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many instances will be created. To work around this, use the -target argument to first apply only
│ the resources that the for_each depends on.

I've tried to add depends_on but no luck. Am I missing anything?

depends_on = [
   module.my_folder
]

Can we have a project with the name different from the project_id ?

I couldn't create a project with a name différent from the project_id. Is there a reason ?

i'd have for example :

id: prj-c-logging-1c78
name: prj-c-logging

this module allow it:
https://github.com/terraform-google-modules/terraform-google-project-factory

Add support for auto-creating service account in GKE nodepool module

The GKE nodepool module should offer the option to auto-create the service account used for nodes, similarly to what the Compute VM module is doing. It should of course adopt the same interface for variables.

Fix for next hop instance routes in net-vpc

Routes with next hop instance need the next_hop_instance_zone attribute or the route will be dropped and recreated each time, despite what the docs specify.

Also change to avoid constant plan changes

• network so the name instead of the self link is passed
• strip https://www.googleapis.com/compute/v1/ from instance self link
• tags to [] instead of null

Incorrect argument name in /modules/net-vpc/main.tf for resource "google_compute_route" "ilb"

The argument name "next_hop_ilb" is causing an issue while creating a shared vpc for a project.
Although the resource shouldn't be ideally called while making a shared VPC without custom routes, terraform plan blurts out an error.

Disabling snat using Terraform - GKE

Hi. I faced an issue with the gke-cluster module in this repo relating to disabling SNAT. Have posted the complete details of the issue here: hashicorp/terraform-provider-google#6537

Any help? Not sure how to fix the problem. Thanks.

Add PSC endpoint support to net-address module

Use a single variable for bigquery access roles

Conform bigquery access roles to the new iam variables design.

Add support for description property

Any chance for "description" support?

I love the modules for their ease of use and so far this is the only property missing that we are asked to use.

Improvements to VPC-SC module

I noticed a couple of improvements we can do to the VPC-SC module:

• The org_id output has a depends_on on resources that don't exist in the module
• The variable and output org_id are not in line with the naming conventions used in other places. A better name would be organization_id
• Now that we're supporting Terraform 0.13, some of the variables can be validated

Add support for global access to OCP apps router

https://docs.openshift.com/container-platform/4.8/networking/ingress-operator.html#nw-ingress-controller-configuration-gcp-global-access_configuring-ingress

Org policies exception

Hey @ludoo @juliocc, I've run into a scenario where I had to apply an org policy at the org level but make exceptions for certain projects. I've modified the organization module locally and added two variables in order to be able to do that.

Vars declaration example:

exception_policy_boolean = {
    "iam.disableServiceAccountCreation" = ["project1","project5"],
    "iam.disableServiceAccountKeyCreation" = ["project5"]
}
exception_policy_list = {
    "constraints/gcp.resourceLocations" = ["project20"]
}

Basically, a map having constraint (boolean or list) as a key and a list with 1 or N projects you'd like to make an exception.
Please let me know if that's something interesting we could add to the module so I can prepare a PR.

Thanks!

CORS support for gcs module

Any reason why we don't support a dynamic cors block in the GCS module?

Add support for hierarchical firewalls

Probably in the org/folder modules.

https://www.terraform.io/docs/providers/google/r/compute_organization_security_policy_association.html https://www.terraform.io/docs/providers/google/r/compute_organization_security_policy.html https://www.terraform.io/docs/providers/google/r/compute_organization_security_policy_rule.html

GCS label fail

Hey all,

As per GCS bucket names convention, it allow us to create bucket names with dashes, underscores and dots. However, in the GCS module we are merging the name of the bucket in the labels field which will fail in case there are dots in the bucket name.

For example, if I create a bucket name: "my-gcs-bucket.com"

This part of the code will merge the name of the bucket as labels

labels = merge(var.labels, {
    location      = lower(var.location)
    name          = lower(var.name)
    storage_class = lower(var.storage_class)
  })

and we get a 400 from googleapis once run terraform apply:

│ Error: googleapi: Error 400: Invalid argument., invalid
│
│ with module.gcs.google_storage_bucket.bucket,
│ on ../../modules/gcs/main.tf line 25, in resource "google_storage_bucket" "bucket":
│ 25: resource "google_storage_bucket" "bucket" {

I can open a PR to fix it if you all are ok making this change.

Thank you!

views in the bigquery-dataset module requires a depends_on attribute

The google_bigquery_table.views resource directly depends on the creation of table so it is evident that the tables should be created and only then the views should start spinning up. Even running the code provided in Readme.md is throwing up an error when the view needs to be created.

An argument named "mtu" is not expected here.

Hello Team,
Thanks for fabulous minimal modules. I'm trying to create a minimal VPC with following script:

module "gke_vpc" {
  source     = "github.com/terraform-google-modules/cloud-foundation-fabric/modules/net-vpc"
  project_id = var.project_id
  name       = "gke-vpc"
  subnets = [
    {
      ip_cidr_range       = "10.0.0.0/24"
      name                = "gke-sub"
      region              = var.region
      secondary_ip_range  = {
        pods     = "10.1.0.0/20"
        services = "10.2.0.0/24"
      }
    }
  ]
}

Getting following error:

# terraform plan -var-file=dev-env.tfvars
Error: Unsupported argument
  on .terraform/modules/gke_vpc/modules/net-vpc/main.tf line 91, in resource "google_compute_network" "network":
  91:   mtu                             = var.mtu
An argument named "mtu" is not expected here.

Checked .terraform modules directory. It has mtu variable declared variable.tf.
Any help what's wrong here?

Add support for google_compute_machine_image

Added to version 3.53 of the beta provider:

https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_machine_image

Adding short names for the GCS nomenclature instead of the complete region

I have observed that in the case of regional buckets, the complete region is included in the name of the bucket
(prefix-us-central1-suffix), this is due to the default logic that if the bucket is multi-regional then the regional prefix is simply us or eu which makes sense. But in the case of regional buckets, can we use short names of the regions?
This can certainly help:
module "utils" {
source = "terraform-google-modules/utils/google"
version = "~> 0.3"
region = "us-central1"
}
Anything that's already stopping us from doing so?

Add support for stateful MIGs

Implement support for stateful policy in the compute-mig module via compute_per_instance_config resource and stateful_disk block in instance manager.

/modules/bigquery-dataset : resource "google_bigquery_dataset_access" has a provider issue

The resource "google_bigquery_dataset_access" under /modules/bigquery-dataset has a provider issue. The provider "google" does not support this resource anymore and thus the provider has to be explicitly mentioned in the resource block in main.tf. I cloned this repository and tried to implement this solution which resolved the issue.

Although my code never really mentioned any reference to the dataset access, this error popped up which is quite surprising.
Attaching a screenshot of the same with this issue.

Allow using gVNIC in compute-vm

https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_instance#nic_type

Hierarchy types

Would be nice to have some other commonly used hierarchies for GCP folders.

We could have the following hierarchy types:

• [exists] Environment folder --> Business Unit folder --> Team folder
• [exists] Business Unit folder --> Environment folder --> Team folder
• Business Unit folder --> Team folder --> Environment folder
• Business Unit folder --> Team folder --> Flat - Projects with -dev, -prod suffix

We could also have the Netflix's tribe model which adds a notion of floating / non-permanent groups of people than can be moved easily, but not sure if it really would be in scope here.

I've seen all of those, but there might be others to consider.

Add support for regional PDs

Support regional PD in compute-vm module.

cloud-config-container modules not working

Looks like COS VMs launched using cloud-config-container cloud init are no longer working. Didn't dig too much into it but looks like there are some changes to COS that breaks it.

member and role exchanged in v4.1.0 for terraform project

When trying to apply configuration using module terraform-google-modules/cloud-foundation-fabric.git//foundations/environments?ref=v4.1.0 I am getting badRequest error:

module.tf-project.google_project_iam_member.additive["roles/[email protected]"]: Creating...

Error: Request "Create IAM Members roles/owner [email protected] for \"project \\\"xxx-terraform\\\"\"" returned error: Error applying IAM policy for project "xxx-terraform": Error setting IAM policy for project "xxx-terraform": googleapi: Error 400: Policy members must be of the form "<type>:<value>"., badRequest

After looking into plan I found that probably member and role values are exchanged:

Terraform will perform the following actions:

  # module.tf-project.google_project_iam_member.additive["[email protected]/owner"] will be created
  + resource "google_project_iam_member" "additive" {
      + etag    = (known after apply)
      + id      = (known after apply)
      + member  = "roles/owner"
      + project = "xxx-terraform"
      + role    = "[email protected]"
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Remove unused IAM policies from shared-vpc-gke example module

Switch subnets variable in `net-vpc` module from map to list

The current variable makes it impossible to use the same subnet names in different regions.

Invalid for_each argument when using folder_unit

Hi. I am trying to use the folder_unit module within foundation and I get this error:

Not sure, but I guess the service account email is determined at runtime. Also did refer this: https://discuss.hashicorp.com/t/for-each-value-depends-on-resource-attributes-that-cannot-be-determined-until-apply/6061

May I know how I can solve this?

Running TF 0.13.2

CC: @ludoo

Update deprecated "bucket_policy_only" attribute in folders-unit module

While using the folders-unit module, the following messsage is returned: Warning: "bucket_policy_only": [DEPRECATED] Please use the uniform_bucket_level_access as this field has been renamed by Google.

Error when changing or removing environment in environments list variable

On using the Environment-based organizational sample, with 4 environments first, all went smoothly.

When I changed environments list variable (see below - removed "nonprod"),

variable "environments" {
  description = "Environment short names."
  type        = list(string)
  default     = ["prod","nonprod","test","dev"]
  # default     = ["prod","nonprod","test","sandpit"]
} 

I received this error.

Error: Error in function call

  on .terraform/modules/service-accounts-tf-environments/terraform-google-modules-terraform-google-service-accounts-2db8e39/outputs.tf line 44, in output "emails":
  44:   value       = zipmap(var.names, google_service_account.service_accounts[*].email)
    |----------------
    | google_service_account.service_accounts is tuple with 4 elements
    | var.names is list of string with 3 elements

Call to function "zipmap" failed: number of keys (3) does not match number of
values (4).

Force OCP example to use google provider 3.65.0

The latest google provider seems to have some significant issues bringing up consistently the OCP environment.
I'd suggest to pin for now the sample code to version 3.65.0.

Sending a PR right in few mins...