pulumi / pulumi-google-native

Native Google Cloud Pulumi Provider (developer preview)

The native Google Cloud Provider for Pulumi lets you provision Google Cloud resources in your cloud programs.

This provider uses the Google Cloud REST API directly and therefore provides full access to Google Cloud.

Warning: Google Cloud Native is available in developer preview. While upstream changes continue to be delivered, active development is paused. Breaking changes may be introduced in minor version releases.

To use this package, install the Pulumi CLI.

Installing

This package is available in many languages in the standard packaging formats.

Node.js (JavaScript/TypeScript)

To use from JavaScript or TypeScript in Node.js, install using either npm:

npm install @pulumi/google-native

or yarn:

yarn add @pulumi/google-native

Python

To use from Python, install using pip:

pip install pulumi_google_native

Go

To use from Go, use go get to grab the latest version of the library

go get github.com/pulumi/pulumi-google-native/sdk

.NET

To use from .NET, install using dotnet add package:

dotnet add package Pulumi.GoogleNative

Concepts

The native Google Cloud package provides a strongly-typed means to build cloud applications that create and interact closely with Google Cloud resources. Resources are exposed for the entire Google Cloud surface area, including (but not limited to) 'compute', 'container', 'run', 'storage', and more.

The native Google Cloud provider works directly with the Google Cloud API instead of depending on a handwritten layer as with the Google Cloud Classic Provider. This approach ensures higher quality and higher fidelity with Google Cloud.

Configuring credentials

To learn how to configure credentials refer to the Google Cloud configuration options.

When developing locally, we recommend that you install the Google Cloud SDK and then authorize access with a user account. Other configuration settings should be set either via pulumi config set google-native:<KEY> <VALUE> or pass options to the constructor of a new google-native Provider.

Building

Dependencies

• Go 1.15
• NodeJS 10.X.X or later
• Python 3.6 or later
• .NET Core 3.1

Please refer to Contributing to Pulumi for installation guidance.

Building locally

Run the following commands to install Go modules, generate all SDKs, and build the provider:

make ensure
make build

Add the bin folder to your $PATH or copy the bin/pulumi-resource-google-native file to another location in your $PATH.

Running an example

Navigate to one of the examples and run Pulumi:

cd ./exampes/simple
yarn link @pulumi/google-native
pulumi up

pulumi-google-native's Issues

Handle collisions between parent properties and flattened equivalents

Study “magic modules” - see if they are useful for us and can be mixed with discovery documents.

Set Pulumi user agent

Set the user agent here similar to azure-native. Do we have a partner ID to use?

• Follow the example format below.
• A user should be able to opt-out or modify the user agent.

Suggested format:

<prod_name>/ver (GPN:<company name>; <other comments>) <other prod>/ver

Example:

Product_A/1.1 (GPN:Partner_ABC; staging) Product_B/1.3 Google­HTTP­Java­Client/1.20.0

*User Agent compliance with RFC (http://tools.ietf.org/html/rfc7231#section­5.5.3)

Auto-project and auto-region

Automatically populate the project and the region for all resources based on the provider value. Should they even be in the SDK arguments?

[build] enable publishing binaries + sdks from master

Typescript definitions for enum types should use string unions or enums

For cases where the Google Cloud discovery document provides an enum of possible values for a string field or parameter, the types generated by Pulumi should generate a string union/enum of these values and use that, instead of just a plain string. This would allow values to be autocompleted by IDEs (and therefore saving users from having to check allowed values in GCP's web docs), and would prevent invalid values from ever even reaching Pulumi since they'd be caught by the Typescript compiler first.

serviceusage API access

Currently there is no representation for the serviceusage API. It appears in the discovery JSON, but there's no generated code to interact with it.

No delete method for some resources

Quite a few resources don't have a DELETE method. Currently, we ignore this and do nothing on deletion (which is probably confusing to say the least but the classic provider sometimes does that too). There could be multiple cases there:

• DELETE missing in the discovery
• Resource can't be deleted
• A different method should be used instead of DELETE (e.g., "cancel", "disable", etc.)

We should categorize those cases and create separate issues with a plan for each resource.

Robust authentication with all methods supported by API/TF provider

Get CI/CD/releases working for GCP-Native

Error handling, retries, timeouts

Our current HTTP implementation is somewhere naive:

• Handle errors from the HTTP client
• If they are retryable, run exponential retries
• Handle timeouts according to the provider settings

[build] hook in automated tests from examples repo

Identify gaps between resources that are easily automated and resources that customers need -- build a plan for resources that need to be implemented manually.

Support resources that are created with a PUT

We require the creation operation to be a POST right now, but some resources (e.g. in Pubsub) have a PUT as a create operation. We should store this in metadata and send the right HTTP request. Also, add a test and make sure that this works.

Update of pubsub.v1.Subscription fails

Repro code:

import * as pulumi from "@pulumi/pulumi";
import * as google from "@pulumi/google-native";
import * as random from "@pulumi/random"

const config = new pulumi.Config("google-native");
const project = config.require("project");
const region = config.require("region");

const randomString = new random.RandomString("name", {
    upper: false,
    number: false,
    special: false,
    length: 8,
});

const topic = new google.pubsub.v1.Topic("topic", {
    project,
    topicId: randomString.result,
});

const sub = new google.pubsub.v1.Subscription("sub", {
    project,
    topic: topic.name,
    subscriptionId: randomString.result,
    labels: {
        test: "test",
    }
});

Steps:

1. Run pulumi up.
2. Change the label in the subscription definition.
3. Run pulumi up again and get the error:

error sending request: googleapi: Error 400: The update_mask in the UpdateSubscriptionRequest must be set, and 
must contain a non-empty paths list.: "https://pubsub.googleapis.com/v1/projects/development/subscriptions/uawesqfy" 
map[subscription:map[ackDeadlineSeconds:<nil> deadLetterPolicy:<nil> detached:<nil> enableMessageOrdering:<nil> 
expirationPolicy:<nil> filter:<nil> labels:map[test:test] messageRetentionDuration:<nil> name:<nil> pushConfig:<nil> 
retainAckedMessages:<nil> retryPolicy:<nil> topic:projects/development/topics/uawesqfy]]

Incorrect documentation: 'Id' should be 'ID' with Go

The documentation here:

https://www.pulumi.com/docs/reference/pkg/google-native/artifactregistry/v1beta2/repository/#outputs

Says that the output Id is available:

And yet it appears that it is actually called ID:

error: reg.Id undefined (type *v1beta2.Repository has no field or method Id)

Using reg.ID works.

Nix that, reg.ID is the standard ID() function on resources. Hmm, is this colliding or something?

Rename the repo to pulumi-gcp-native

Rename the provider to gcp-native

Drift detection is not sensitive about bucket configuration

I am trying to deploy a Google Cloud storage bucket with Pulumi 3.0 with the native provider. Using the example code below:

import * as storage from "@pulumi/google-native/storage/v1";
import * as pulumi from "@pulumi/pulumi";

const config = new pulumi.Config("google-native");
const project = config.require("project");
const bucketName = "pulumi-goog-native-ts-01";

// Create a Google Cloud resource (Storage Bucket)
const bucket = new storage.Bucket("my-bucket", {
    name: bucketName,
    bucket: bucketName,
    project: project,
    labels: {
        "bla": "123",
        "blla2": "234"
    }
});

Here is my configuration on the console:

Expected behavior

pulumi up --refresh should detect drift and correct the relevant configuration values - here labels.

Current behavior

pulumi up --refresh does not detect drift, and leaves the labels unchanged.

Output:

Previewing update (gcs-refresh-2):
     Type                                Name      
     pulumi:pulumi:Stack                 gcp-native
     └─ google-native:storage/v1:Bucket  my-buc
 
Resources:
    2 unchanged

Updating (gcs-refresh-2):
     Type                                Name      
     pulumi:pulumi:Stack                 gcp-native
     └─ google-native:storage/v1:Bucket  my-buc
 
Outputs:
    bucketSelfLink: "https://www.googleapis.com/storage/v1/b/pulumi-goog-native-ts-01"

Resources:
    2 unchanged

Duration: 2s

Steps to reproduce

1. Run pulumi new google-native-typescript in a new directory to create a new project
2. Replace the contents of index.ts with the example code above.
3. Run pulumi up --refresh (new bucket should be deployed)
4. Add or remove labels in the configuration screen of your new Storage bucket.
5. Run pulumi up --refresh

Context (Environment)

We're trying to evaluate different infrastructure-as-code options and drift detection is a very important feature for us to be able confidently deploy infrastructure.

Affected feature

https://www.pulumi.com/docs/reference/cli/pulumi_refresh/

No create method for some resources

We log all the resources that have no "create" method, here is the list:

OrganizationApiproductAttribute (discovery/apigee_v1.json)
OrganizationApiKeyvaluemap (discovery/apigee_v1.json)
OrganizationApiRevision (discovery/apigee_v1.json)
OrganizationDeveloperAppAttribute (discovery/apigee_v1.json)
OrganizationDeveloperAppKeyApiproduct (discovery/apigee_v1.json)
OrganizationDeveloperAttribute (discovery/apigee_v1.json)
OrganizationEnvironmentKeyvaluemap (discovery/apigee_v1.json)
OrganizationKeyvaluemap (discovery/apigee_v1.json)
OrganizationSharedflowRevision (discovery/apigee_v1.json)
AppService (discovery/appengine_v1.json)
AppServiceVersionInstance (discovery/appengine_v1.json)
AppService (discovery/appengine_v1beta.json)
AppServiceVersionInstance (discovery/appengine_v1beta.json)
RepositoryPackageVersion (discovery/artifactregistry_v1beta1.json)
RepositoryPackageVersion (discovery/artifactregistry_v1beta2.json)
Model (discovery/bigquery_v2.json)
DebuggerDebuggeeBreakpoint (discovery/clouddebugger_v2.json)
Group (discovery/clouderrorreporting_v1beta1.json)
DeviceDeviceUser (discovery/cloudidentity_v1.json)
DeviceDeviceUser (discovery/cloudidentity_v1beta1.json)
Profile (discovery/cloudprofiler_v2.json)
Organization (discovery/cloudresourcemanager_v1beta1.json)
IndexingDatasourceItem (discovery/cloudsearch_v1.json)
Snapshot (discovery/compute_v1.json)
ScanConfig (discovery/containeranalysis_v1alpha1.json)
ScanConfig (discovery/containeranalysis_v1beta1.json)
TagTemplateField (discovery/datacatalog_v1beta1.json)
Snapshot (discovery/dataflow_v1b3.json)
AnswerRecord (discovery/dialogflow_v2.json)
AnswerRecord (discovery/dialogflow_v2.json)
AnswerRecord (discovery/dialogflow_v2beta1.json)
AnswerRecord (discovery/dialogflow_v2beta1.json)
Project (discovery/firebase_v1beta1.json)
DatabaseDocument (discovery/firestore_v1.json)
DatabaseDocument (discovery/firestore_v1beta1.json)
DatasetDicomStoreStudy (discovery/healthcare_v1.json)
DatasetDicomStoreStudy (discovery/healthcare_v1beta1.json)
BillingAccountLog (discovery/logging_v2.json)
FolderLog (discovery/logging_v2.json)
Log (discovery/logging_v2.json)
OrganizationLog (discovery/logging_v2.json)
Log (discovery/logging_v2.json)
CertificateAuthorityCertificateRevocationList (discovery/privateca_v1beta1.json)
NamespaceRevision (discovery/run_v1.json)
Revision (discovery/run_v1.json)
FolderSourceFinding (discovery/securitycenter_v1.json)
SourceFinding (discovery/securitycenter_v1.json)

Why is that? Would any of them be useful as Pulumi resources?

Universal way to track all discovery documents

Currently, we download all discovery documents from https://discovery.googleapis.com/discovery/v1/apis?parameters

There are several issues to this:

1. The endpoint returns some documents unrelated to Google Cloud, and there's no obvious way to filter them out. We have setup a somewhat arbitrary criteria here but we may be missing something or taking too much, or may start doing so in the future.
2. Some relevant discovery documents are missing in that list. We found one example so far, are there more? Why is vpcaccess not present in the registry?
3. There's no way to track the history of changes in the discovery documents at the source. We keep them in our source control now but we lack context for any changes made.

Is there a better way to discover discovery documents?

Study “Cloud Deployment Manager” - see if it’s useful for us as a deployment method and/or metadata source.

Functions (invokes)

Define the plan for which invokes we need to support. Implement them.

Pubsub Schema creation fails with "Invalid resource name given"

When attempting to create a pubsub schema with python, I'm getting an error "Invalid resource name given (name=projects/my-gcp-project/schemas/)" even though I'm specifying the resource_name and name params with the pattern: projects/my-gcp-project/schemas/someuniquename

Expected behavior

The schema should be created without error.

Current behavior

I'm getting the following error: google-native:pubsub/v1:Schema (someuniquename):
error: error sending request: googleapi: Error 400: Invalid resource name given (name=projects/my-gcp-project/schemas/). Refer to https://cloud.google.com/pubsub/docs/admin#resource_names for more information.:

Steps to reproduce

1. Add this code: schema = pubsub_native.v1.Schema(resource_name=schemas_id, projects_id=GCP_PROJECT_ID,
   schemas_id=schemas_id, definition=json.dumps(sch1), name=schema_name, type='avro')
2. pulumi up
3. yes

Refine the representation of IamPolicy resources

At the moment, the resource IDs get :getIamPolicy suffix and we hard-coded its replacement to :setIamPolicy in the update/delete of the provider. Would it make sense to set a separate UpdatePath field in the metadata to encode exceptions such as these where the IdPath can't be used for updates? Should the ID not have the :xxxIamPolicy suffix at all?

Bucket is missing read-only annotations

Storage Bucket type is missing read-only annotations for some obviously read-only properties like timeCreated and updated:

pulumi-google-native/discovery/storage_v1.json

Lines 3551 to 3560 in 968254d

	"timeCreated": {
	"description": "The creation time of the bucket in RFC 3339 format.",
	"format": "date-time",
	"type": "string"
	},
	"updated": {
	"description": "The modification time of the bucket in RFC 3339 format.",
	"format": "date-time",
	"type": "string"
	},

Required/read-only annotations are missing in discovery documents

Both required and readOnly properties are used in multiple discovery documents. However, some of the documents do not use this annotations but "mark" properties as required/optional or readonly in description text:

• Read-only: diagflow, monitoring
• Required: iamcredentials - lots of those.

There are other variations like "output only", or "read-only" somewhere further in the text, which are hard to catch, unreliable, and hard to detect for breaking changes.

Deprecation is also signaled in descriptions with no formal attribute for it.

Confirm that common scenarios are possible with GCP-Native

e.g. our examples, Google's examples

Missing serviceusage.googleapis.com:enable

Affected feature

It seems like the services.enable method of serviceusage.googleapis.com is not yet included in the google-native provider. Is there an other way to enable services in a google project with this new provider? I am experimenting with migrating our current pulumi 2.x setup to 3.x. Enabling services is one of the first thing we do in our stacks. This would be the equivalent of this 2.x resource: https://www.pulumi.com/docs/reference/pkg/gcp/projects/service/

API documentation for the underlying serviceusage.googleapis.com is at https://cloud.google.com/service-usage/docs/reference/rest/v1/services/enable

Allow for shorthand property values instead of duplicating project, zone, etc.

For some resource properties, the Google Cloud API expects a fully-qualified value based on the project and zone values, rather than accepting a shorthand value.

One example of this is the machineType argument on google.compute.v1.Instance. The Google Cloud API expects this to be projects/${project}/zones/${zone}/machineTypes/f1-micro instead of f1-micro. This isn't immediately obvious to the end user and the error message returned isn't very helpful either.

Affected feature

It would be great if the provider would allow the end user to provide the shorthand value (e.g. machineType: "f1-micro") and automatically expand this into the fully-qualified value since the project and zone properties will already be known by the provider (default or explicit).

Code to reproduce

This code is the format that the Google Cloud API expects. Changing the value to f1-micro will result in the error below.

pulumi-google-native/examples/webserver-ts/index.ts

Line 49 in 458dda1

machineType: `projects/${project}/zones/${zone}/machineTypes/f1-micro`,

Error

  google-native:compute/v1:Instance (main):
    error: error sending request: googleapi: Error 400: Invalid value for field 'resource.machineType': 'n1-standard-1'. The URL is malformed., invalid: "https://compute.googleapis.com/compute/v1/projects/pulumi-ce-team/zones/us-west2-a/instances" map[machineType:n1-standard-1 name:cameron]

[build] Add environment variables for authentication with GCP during tests

Update templates to add gcp-native

ManagedZoneRrset: Invalid value for 'entity.rrset.name': ''

Declaring a resource of type 'ManagedZoneRrset' always throws an error sending the request to the googleapi and I am not able to create it. It seems that the name of the Rrset (entity.rrset.name) is being send with an empty value to the API, though it is declared.

Expected behavior

The resource is created without any error.

Current behavior

After providing pulumi up I get the following error message: error: error sending request: googleapi: Error 400: Invalid value for 'entity.rrset.name': '', invalid: "https://dns.googleapis.com/dns/v1beta2/projects/gcp-project-xxx/managedZones/xxx-xxx/rrsets" map[rrdatas:[x.x.x.x] ttl:300] error: update failed

Steps to reproduce

1. Create the resource:

const dnsRecord = new google.dns.v1beta2.ManagedZoneRrset('dns-record', {
  managedZone: managedZone.name,
  name: 'test',
  project: project,
  ttl: 300,
  rrdatas: [ip.address],
  type: 'A',
});

2. pulumi up --diff
3. Verifying the diff:

google-native:dns/v1beta2:ManagedZoneRrset: (create)
    [urn=urn:pulumi:my-stack::my-project::google-native:dns/v1beta2:ManagedZoneRrset::dns-record]
    [provider=urn:pulumi:my-stack::my-project::pulumi:providers:google-native::default_0_1_0::fae4c8b6-80d8-4328-b011-362febee7abc]
    managedZone: "my-managed-zone"
    name       : "test"
    project    : "my-project"
    rrdatas    : [
        [0]: "x.x.x.x"
    ]
    ttl        : 300
    type       : "A"

Context (Environment)

I am using the version 0.1.0 and getting the same error if I using the v1 API instead of v1beta2. When I change the GCP provider to @pulumi/gcp and create the resource with exactly the same parameters and values I succeeded with creating the resource.

Affected feature

dns.v1beta2.ManagedZoneRrset
dns.v1.ManagedZoneRrset

Description missing for path parameters

We build SDK properties from the flatPath property of an API method. Currently, those properties lack description in SDKs and docs. Find a way to populate them.

Example for Redis:

• flatPath containing instancesId
• SDK property
• instanceId which we could reify with instancesId to get the description

Is there any source of description text for parameters projectsId, locationsId, and instancesId that constitute the path?

Auto-naming

Automatically apply a default value to a resource name, whatever the property name it uses. Use the logical name and append a random suffix to it.

Incorrect args for project creation

ProjectArgs imported from @pulumi/gcp-native/cloudresourcemanager/v3 has both projectId and projectsId as fields, with the former being optional and the latter being required

However, according to the discovery document and also the API docs, projectsId should not exist and projectId should be required

Implement graceful cancellation

In the Cancel method

Provider configuration and first-class providers

Support a set of configuration values similar to the classic provider

Unique method names

The majority of methods in discovery documents use standard names for CRUD operations: "create", "get", "update", and "delete". However, there are some snowflakes:

• dropDatabase in Spanner
• replaceService in Cloud Run
• insert in Storage

Are those name differences significant? How do we discover all of them?

Handle 409 responses as retryable

Some APIs like sql db have tight constraints on concurrent operations, e.g. creating two DBs on the same DB instance will cause a 409 on the second:

Error 409: Operation failed because another operation was already in progress., operationInProgress: "https://sqladmin.googleapis.com/sql/v1beta4/projects/pulumi-310722/instances/pgdb-oyflyrsr/databases" map[name:app_development]

We should consider handling such 409s as retryable (with some constraints around giving up).

Import resources

Enable support of import operations in the Read method

Harden resource await logic

We have resource operation await login in waitForResourceOpCompletion which is a bit ad-hoc and mixes two styles of awaiting.

1. Split the two styles up and document them well.
2. Based on the classic provider, there seem to be 2-3 more styles of awaiting. Explore them and cover as needed.
3. Potentially, augment resource metadata to explicitly opt in the right await style for each resource (or any resource that deviates from default).

Add enum support

Currently pulumi schema doesn't capture enum information.

Breaking change policy for discovery document

What is the policy for breaking changes in discovery documents?

We observed one example so far - in v1beta version. Are these limited to pre-release versions? We installed the machinery to detect new breaking changes as they appear, so we'll extend this issue as we go.

We should find an official source.

Handle resources where no GET endpoint exists but LIST exists

Currently we require the existence of a GET endpoint for resource detection. However, a number of resources have LIST endpoints which return all children of the resource's parent, we currently print a warning during the generation phase: https://github.com/pulumi/pulumi-gcp-native/blob/master/provider/pkg/gen/schema.go#L236. We should explore the possibility of using the LIST + a filter in lieu of a GET.

Examples:

SshKey (discovery/baremetalsolution_v2.json)
Assignment (discovery/bigqueryreservation_v1.json)
TagBinding (discovery/cloudresourcemanager_v3.json)
TagHold (discovery/cloudresourcemanager_v3.json)
Comment (discovery/cloudsupport_v2beta.json)
Tag (discovery/datacatalog_v1.json)
DnsPeering (discovery/datafusion_v1.json)
Sha (discovery/firebase_v1beta1.json)
Release (discovery/firebasehosting_v1beta1.json)
TimeSeries (discovery/monitoring_v3.json)
PredictionApiKeyRegistration (discovery/recommendationengine_v1beta1.json)
Model (discovery/retail_v2alpha.json)
Finding (discovery/securitycenter_v1.json)
TenancyUnit (discovery/serviceconsumermanagement_v1.json)
ProducerOverride (discovery/serviceconsumermanagement_v1beta1.json)
Connection (discovery/servicenetworking_v1.json)
PeeredDnsDomain (discovery/servicenetworking_v1.json)
AdminOverride (discovery/serviceusage_v1beta1.json)
ConsumerOverride (discovery/serviceusage_v1beta1.json)

Multiple paths to the same resource

Discovery documents contain a hierarchy of resources. Two resources with the same name can be in multiple places of the tree: e.g. directly under projects and then again under projects/location. We pick a ~random one currently and build up the names of resources using parent names (except project/location/etc). Is that a valid strategy? Are those endpoints always equivalent? Any preference which one to use?

BucketObject implementation is incomplete

We hard-coded the basic upload functionality for BucketObject but it misses a bunch of properties. We should either get rid of the special case and handle uploads based on discovery documents (preferred), or create a full custom resource and make sure it works 100%.

Generate API docs for the provider

Evaluate automated example generation for docs

Evaluate deployment manager examples, e.g. here to generate resource examples in docs.

Cloud Run Service outputs are missing after creation

The URL is missing in the outputs, and refresh sometimes brings changes: https://github.com/pulumi/pulumi-gcp-native/runs/2297186491?check_suite_focus=true#step:20:126