hashicorp / terraform-plugin-go

A low-level Go binding for the Terraform protocol for integrations to be built on top of.

terraform-plugin-go

terraform-plugin-go provides low-level Go bindings for the Terraform plugin protocol, for integrations to be built upon. It strives to be a minimal possible abstraction on top of the protocol, only hiding the implementation details of the protocol while leaving its semantics unchanged.

Status

terraform-plugin-go is a Go module versioned using semantic versioning.

The module is currently on a v0 major version, indicating our lack of confidence in the stability of its exported API. Developers depending on it should do so with an explicit understanding that the API may change and shift until we hit v1.0.0, as we learn more about the needs and expectations of developers working with the module.

We are confident in the correctness of the code and it is safe to build on so long as the developer understands that the API may change in backwards incompatible ways and they are expected to be tracking these changes.

Terraform CLI Compatibility

Providers built on terraform-plugin-go will only be usable with Terraform v0.12.0 and later. Developing providers for versions of Terraform below 0.12.0 is unsupported by the Terraform Plugin SDK team.

Go Compatibility

This project follows the support policy of Go as its support policy. The two latest major releases of Go are supported by the project.

Currently, that means Go 1.21 or later must be used when including this project as a dependency.

Getting Started

terraform-plugin-go is targeted towards experienced Terraform developers. Familiarity with the Resource Instance Change Lifecycle is required, and it is the provider developer's responsibility to ensure that Terraform's requirements and invariants for responses are honored.

Provider developers are expected to create a type that implements the tfprotov5.ProviderServer interface. This type should be passed to tfprotov5server.Serve along with the name (like "hashicorp/time").

Resources and data sources can be handled in resource-specific or data source-specific functions by using implementations of the tfprotov5.ResourceServer and tfprotov5.DataSourceServer interfaces, using the provider-level implementations of the interfaces to route to the correct resource or data source level implementations using req.TypeName.

When handling requests, tfprotov5.DynamicValue types should always have their Unmarshal methods called; their properties should not be inspected directly. The tftypes.Value returned from Unmarshal can be inspected to check whether it is known and subsequently converted to a plain Go type using its As method. As will return an error if the Value is not known.

Testing

The Terraform Plugin SDK's helper/resource package can be used to test providers written using terraform-plugin-go. While we are working on a testing framework for terraform-plugin-go providers that is independent of the Plugin SDK, this may take some time, so we recommend writing tests in the meantime using the plugin SDK, which will not be a runtime dependency of your provider.

You must supply a factory for your provider server by setting ProtoV5ProviderFactories on each TestCase. For example:

package myprovider

import (
	"regexp"
	"testing"

	"github.com/hashicorp/terraform-plugin-go/tfprotov5"
	"github.com/hashicorp/terraform-plugin-sdk/v2/helper/resource"
)

func TestAccDataSourceFoo(t *testing.T) {
	resource.UnitTest(t, resource.TestCase{
		ProtoV5ProviderFactories: map[string]func() (tfprotov5.ProviderServer, error){
			"myprovider": func() (tfprotov5.ProviderServer, error) {
				return Server(), nil
			},
		},
		Steps: []resource.TestStep{
			{
				Config: `"data" "myprovider_foo" "bar" {}`,
				Check: resource.ComposeTestCheckFunc(
					resource.TestMatchResourceAttr("data.myprovider_foo.bar", "current", regexp.MustCompile(`[0-9]+`)),
				),
			},
		},
	})
}

Debugging

Provider servers can be instrumented with debugging tooling, such as delve, by using the WithManagedDebug() and WithDebug() ServeOpt. In this mode, Terraform CLI no longer manages the server lifecycle and instead connects to the running provider server via a reattach configuration supplied by the TF_REATTACH_PROVIDERS environment variable. The WithDebug() implementation is meant for advanced use cases which require manually handling the reattach configuration, such as managing providers with terraform-exec, while the WithManagedDebug() implementation is suitable for provider main() functions. For example:

func main() {
	debugFlag := flag.Bool("debug", false, "Start provider in debug mode.")
	flag.Parse()

	opts := []tf6server.ServeOpt{}

	if *debugFlag {
		opts = append(opts, tf6server.WithManagedDebug())
	}

	tf6server.Serve("registry.terraform.io/namespace/example", /* Provider function */, opts...)
}

Protocol Data

To write raw protocol MessagePack or JSON data to disk, set the TF_LOG_SDK_PROTO_DATA_DIR environment variable. During Terraform execution, this directory will get populated with {TIME}_{RPC}_{MESSAGE}_{FIELD}.{EXTENSION} named files. Tooling such as jq can be used to inspect the JSON data. Tooling such as fq or msgpack2json can be used to inspect the MessagePack data.

Documentation

Documentation is a work in progress. The GoDoc for packages, types, functions, and methods should have complete information, but we're working to add a section to terraform.io with more information about the module, its common uses, and patterns developers may wish to take advantage of.

Please bear with us as we work to get this information published, and please open issues with requests for the kind of documentation you would find useful.

Scope

This module is intentionally limited in its scope. It serves as the foundation for the provider ecosystem, so major breaking changes are incredibly expensive. By limiting the scope of the project, we're limiting the choices it needs to make, making it less likely that breaking changes will be required once we've hit version 1.0.0.

To that end, terraform-plugin-go's scope is limited to providing a common gRPC server interface and an implementation of Terraform's type system. It specifically is not trying to be a framework, nor is it attempting to provide any utility or helper functions that only a subset of provider developers will rely on. Its litmus test for whether something should be included is "will every Terraform provider need this functionality or can this functionality only be added if it's in this module?" All other functionality should be considered out of scope and should live in a separate module.

Contributing

Please see .github/CONTRIBUTING.md.

Unit Testing

Run go test ./... or make test after any changes.

Linting

Ensure the following tooling is installed:

• `golangci-lint: Aggregate Go linting tool.

Run golangci-lint run ./... or make lint after any changes.

Protocol Updates

Ensure the following tooling is installed:

• protoc: Protocol Buffers compiler. This isn't Go specific tooling, so follow this installation guide
• protoc-gen-go: Go plugin for Protocol Buffers compiler. Install by running make tools
• protoc-gen-go-grpc: Go gRPC plugin for Protocol Buffers compiler. Install by running make tools

The Protocol Buffers definitions can be found in tfprotov5/internal/tfplugin5 and tfprotov6/internal/tfplugin6.

Run make protoc to recompile the Protocol Buffers files after any changes.

License

This module is licensed under the Mozilla Public License v2.0.

terraform-plugin-go's Issues

Allow surfacing AttributePaths from attributePathError

the tftypes.attributePathError could use some love. We should have a way to surface the AttributePath it's wrapping, and it should probably support the Unwrap interface.

Log SDK Version Information

Version

v0.5.0

Use-cases

Providers can be running a variety of SDK packages and versions. Similar to the Terraform CLI:

2021-12-23T16:20:04.565-0500 [INFO]  Terraform version: 1.1.2

It would be great to include SDK version information in the logging.

Attempted Solutions

Asking folks after the fact which SDK version is involved, which might require spelunking which provider version was being used then which SDK version was being used with that provider version.

Proposal

Create a version or meta package which contains the release version information. Ensure release process appropriately updates this information.

Once available, use that version information to output an informational log:

tfsdklog.Info(ctx, "terraform-plugin-go version: "+meta.SDKVersionString()))

References

• hashicorp/terraform-plugin-sdk#839
• hashicorp/terraform-plugin-framework#238
• https://pkg.go.dev/github.com/hashicorp/terraform-plugin-sdk/v2/meta#SDKVersionString

Standardise stored values in tftypes.NewValue

We rely on all the Go types stored in the interface{} for tftypes.Value to be of one of a few types. This internal consistency is a nice property. But our tftypes.NewValue function allows users to construct tftypes.Values out of any Go type, and stores whatever the user provides in that interface{}. We should do some validation, and maybe even conversion, here.

Possible steps:

• Ensure only the supported types (string, *big.Float, bool, map[string]Value, []Value) are stored in that interface{}
• As an enhancement, accept pointers of these types, and maybe map[string]{whatever} and []{whatever} of these types, where {whatever} is any of our supported primitive types/interface{}`, or pointers to our primitive types.
• Use the tftypes.ValueCreator interface to let users define their own conversions for their types.
• Allow common variants (e.g. int instead of *big.Float) and convert them in tftypes.NewValue before setting them

Consider Unit Testable Schema Validation

terraform-plugin-go version

v0.5.0

Use cases

Terraform Plugin SDK has long included the (*helper/schema.Provider).InternalValidate() method, which:

... should be called in a unit test for any provider to verify before release that a provider is properly configured for use with this library.

Terraform CLI version 1.1.0 and later includes some additional schema validation upfront, instead of potentially erroring out later in workflows: hashicorp/terraform#28124

While providers implementing this library are generally encouraged to use the available acceptance testing framework (currently in Terraform Plugin SDK helper/resource), that type of testing is not guaranteed to be used, nor is it guaranteed to trigger some of the Terraform CLI-based validation.

Having a schema validation functionality within terraform-plugin-go should allow provider developers to receive feedback on schema issues more quickly and potentially without the extra burden of acceptance testing (which itself may require credentials, etc). Theoretically, downstream SDKs can also benefit by being able to call into this functionality.

Attempted solutions

Manually implementing schema validation unit tests.

Acceptance testing using Terraform Plugin SDK helper/resource.

Proposals

Schema Method

The tfprotov5.Schema and tfprotov6.Schema types can implement a new method, e.g.

func (s Schema) Validate(ctx context.Context) error { /* ... */ }

Which calls into further Validate methods on SchemaBlock and SchemaAttribute.

terraform-plugin-go providers could then implement a unit test for each schema based on this method:

func TestExampleSchema(t *testing.T) {
  t.Parallel()

  err := exampleSchema.Validate(context.Background())

  if err != nil {
    t.Error("unable to validate schema: %s", err)
  }
}

Providers could also introduce this check prior to returning GetProviderSchemaResponse. Downstream SDKs could either implement wrapping unit testing or similar checks during GetProviderSchemaResponse handling or tfprotov5/tfprotov6 type conversion

GetProviderSchemaResponse Method

Expanding the above, provide a method on top of the tfprotov5.GetProviderSchemaResponse and tfprotov6.GetProviderSchemaResponse types. e.g.

func (r GetProviderSchemaResponse) Validate(ctx context.Context) error {
  err := r.Provider.Validate(ctx)

  if err != nil {
    return fmt.Errorf("unable to validate provider schema: %w", err)
  }

  for name, schema := range r.DataSourceSchemas {
    err := schema.Validate(ctx)

    if err != nil {
      return fmt.Errorf("unable to validate data source %s schema: %w", name, err)
    }
  }

  for name, schema := range r.ResourceSchemas {
    err := schema.Validate(ctx)

    if err != nil {
      return fmt.Errorf("unable to validate resource %s schema: %w", name, err)
    }
  }
}

This could also be introduced later.

ProviderServer Method

Expanding the above, provide a method on top of the tfprotov5.ProviderServer and tfprotov6.ProviderServer types, e.g.

func (s ProviderServer) Validate(ctx context.Context) error {
  resp, err := s.GetProviderSchema(ctx, &GetProviderSchemaRequest{})

  if err != nil {
    return fmt.Errorf("unable to get schema: %w", err)
  }

  err = resp.Validate(ctx)

  if err != nil {
    return fmt.Errorf("unable to validate schema: %w", err)
  }
}

This, along with additional provider server validations, could also be introduced later. This may be the most intuitive place for provider developers wanting to implement unit testing similar to Terraform Plugin SDK, albeit it would be the most broad. Since the RPC request objects are not customizable in this proposal without modifying the function signature, it may not be future proof.

Additional Context

Terraform CLI 1.1.0 implements the following rules:

• Attribute
  • Must be non-nil
  • Must set Optional, Required or Computed
  • Cannot set both Optional and Required
  • Cannot set both Computed and Required
  • Must set Type or NestedType
  • Cannot set both Type and NestedType
  • NestingSet NestedType may not contain attributes of cty.DynamicPseudoType
• Block
  • Must be non-nil
  • Must not have overlapping attribute and nested block names
  • Must have valid name (^[a-z0-9_]+$)
  • MinItems and MaxItems must both be greater than zero
  • MinItems and MaxItems must match in NestingSingle mode
  • MinItems and MaxItems must be set to either 0 or 1 in NestingSingle mode
  • MinItems and MaxItems cannot be used in NestingGroup mode
  • MinItems must be less than or equal to MaxItems in NestingList/NestingSet mode
  • NestingSet blocks may not contain attributes of cty.DynamicPseudoType
  • MinItems and MaxItems must both be 0 in NestingMap mode

References

• https://pkg.go.dev/github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema#Provider.InternalValidate
• hashicorp/terraform#28124
• https://github.com/hashicorp/terraform/blob/6530055d188d363bf64ff99b56bb00b350de2cfb/internal/configs/configschema/internal_validate.go
• hashicorp/terraform-plugin-framework#113

Investigate diagnostics helper package

Constructing diagnostics by hand is a bit annoying at the moment. Is it worth shipping a helper package for diagnostics with terraform-plugin-go, or should we keep that in terraform-plugin-sdk where it's easier for us to version it and make breaking changes?

tftypes: Type should implement AttributePathStepper

terraform-plugin-go version

v0.4.0

Use cases

Given this code in the downstream terraform-plugin-framework:

https://github.com/hashicorp/terraform-plugin-framework/blob/2cedf45c2bf94bfb82ff65b1402b3fd75a806aa4/tfsdk/plan.go#L199-L284

It would not correctly be able handle cases where the schema (framework's) type is DynamicPseudoType, but the value itself is concrete.

Attempted solutions

Looking up types via the wrapping type system, however this will only work as long as the framework's type system is 1:1 with Terraform's concrete type system. In the case of DynamicPseudoType constraints, this assumption can no longer work.

Proposal

Allow Type to be walked by attribute path by implementing AttributePathStepper. This would allow downstream logic to use tftypes.WalkAttributePath(/* tftypes.Type */, /* *tftypes.AttributePath */) to correctly determine a type at a given path.

References

• hashicorp/terraform-plugin-framework#165

tftypes: Consider Additional Errors for (Value).ApplyTerraform5AttributePathStep()

terraform-plugin-go version

v0.4.0

Use cases

Downstream consumers using tftypes.WalkAttributePath() against a tftypes.Value tend to alway receive the same tftypes.ErrInvalidStep error, regardless of the cause, which may include:

• Stepping into a null value
• Stepping into an unknown value
• Stepping into a type incorrectly (e.g. attempting WithAttributeName() on a List value)

The ability to distinguish between these situations means that these consumers can further tailor their intended behaviors, without the need to then determine the underlying cause.

Attempted solutions

After receiving ErrInvalidStep, walk backwards in the path (via recursive (AttributePath).WithoutLastStep() until no ErrInvalidStep is received) to detect a parent value with IsUnknown().

Proposal

Create two new error values (one for stepping into a null value, one for stepping into an unknown value) and convert this logic to use them:

	if !val.IsKnown() || val.IsNull() {
		return nil, ErrInvalidStep
	}

e.g.

	if val.IsNull() {
		return nil, ErrInvalidStepNullValue
	}

	if !val.IsKnown() {
		return nil, ErrInvalidStepUnknownValue
	}

In the error implementation, they should still be appropriately detectable via errors.Is(tftypes.ErrInvalidStep) or this enhancement becomes a breaking change.

References

• hashicorp/terraform-plugin-framework#186

Initial Terraform Plugin Protocol Version 6 (tfprotov6) Implementation

terraform-plugin-go version

0.2.1

Use cases

As part of the upcoming Terraform CLI version 0.15 release, a newer version 6 of the Terraform Plugin Protocol is now available. This protocol bump includes goodies such as NestedType that will benefit the future of plugin development.

Proposal

Begin porting over the existing tfprotov5 directory into a new tfprotov6 directory and perform the protocol additions. If there is any decision making that needs to be made regarding the implementation details, coalesce them onto this issue.

(I don't mind starting this work at some point in the near future as this directly impacts a project I am working on.)

References

• hashicorp/terraform#27699
• hashicorp/terraform#27826

Create terraform.io/plugin Section for terraform-plugin-go Documentation

terraform-plugin-go version

v0.5.0

Use cases

While the Go documentation for this project available at https://pkg.go.dev/github.com/hashicorp/terraform-plugin-go is fairly comprehensive for describing low level details, it can be challenging to tease out higher level concepts or functionality which having website documentation similar to SDKv2 and the framework would be very helpful for provider developers.

For example:

• Terraform Plugin Protocol at a glance and how to inspect it (since this is important when working at this layer)
• Creating gRPC servers
• Environment variables used to control logging, contextual log key descriptions
• Using tftypes.UnknownValue to indicate computed values will change, but also that it cannot be used during apply
• (Needs more consideration, maybe its own section) Using terraform-plugin-mux to merge providers
• When to use terraform-plugin-go, versus the other frameworks
• Acceptance testing (beyond just the README)

Proposal

Create new section on the Terraform documentation website, e.g. https://www.terraform.io/plugin/go

• Overview
• Protocol
• Provider Servers
• Types and Values
• Logging
• Testing
  • Unit
  • Acceptance

Update https://www.terraform.io/plugin/which-sdk with appropriate details about when to use terraform-plugin-go.

References

• #84
• #93 (comment)
• #107
• #117
• https://github.com/hashicorp/terraform-plugin-go#getting-started

Convert json.Number to big.Float when unmarshaling JSON

When unmarshaling JSON right now, we return any numbers as a json.Number. For consistency's sake, we should return them as a big.Float, which is how we talk about numbers elsewhere.

Consider Splitting tftypes.Type into Value and Constraint/Schema Types

terraform-plugin-go version

v0.5.0

Use cases

Terraform has two conceptually different meanings of a "type" within its type system today:

• Value types, which concisely describe the structure and data kind for a specific value.
• Type constraints or schema types, which may contain additional metadata that should be sent across the protocol, but should not necessarily be usable as a value type.

One place where this distinction is important are object types, which may define OptionalAttributes. An object value must specify all attributes (e.g. with nils), regardless of that additional information. Another odd case is DynamicPseudoType, which technically can be used in both contexts.

The tftypes package is currently written with a flattened Type definition that serves both purposes. Functions like NewValue() accept tftypes.Object with OptionalAttributes, however this should not be valid in all scenarios and can cause provider development issues as it can be difficult to reason between the intended usage of functions. Since there is a single Go type to work with, it is slightly more difficult to enforce correctness. If Terraform expands usage of type constraints in the future, this issue will be exasperated.

Proposal

Split tftypes.Type into multiple Go types, one with the sole purpose of value handling and the other for constraint/metadata/schema handling. e.g. (naming up for discussion, it just didn't feel appropriate to still call one Type)

// TypeConstraint is an interface representing a Terraform type with additional
// characteristics. It is only meant to be implemented by the tftypes package.
// TypeConstraints define metadata on top of ValueTypes.
type TypeConstraint interface {
	// Equal performs deep type equality checks, including attribute/element
	// types and whether attributes are optional or not.
	Equal(TypeConstraint) bool

	// String returns a string representation of the TypeConstraint's name.
	String() string

	// ValueType returns the associated ValueType.
	ValueType() ValueType

	// private is meant to keep this interface from being implemented by
	// types from other packages.
	private()
}

// ValueType is an interface representing a Terraform type. It is only meant to be
// implemented by the tftypes package. ValueTypes define the shape and
// characteristics of data coming from or being sent to Terraform.
type ValueType interface {
	// Equal performs deep type equality checks, including attribute/element
	// types.
	Equal(ValueType) bool

	// Is performs shallow type equality checks, in that the root type is
	// compared, but underlying attribute/element types are not.
	Is(ValueType) bool

	// String returns a string representation of the ValueType's name.
	String() string

	// UsableAs performs type conformance checks. This primarily checks if the
	// target implements DynamicPsuedoType in a compatible manner.
	UsableAs(TypeConstraint) bool

	// private is meant to keep this interface from being implemented by
	// types from other packages.
	private()
}

The most impactful changes for implementors will then be the updated function signatures to align with these new types:

// package tfprotov5/tfprotov6
type SchemaAttribute struct {
	// ...

	// TypeConstraint indicates the type of data the attribute expects. See the
	// documentation for the tftypes package for information on what types
	// are supported and their behaviors.
	Type tftypes.TypeConstraint
}

func NewDynamicValue(t tftypes.ValueType, v tftypes.Value) (DynamicValue, error)

func (d DynamicValue) Unmarshal(typ tftypes.ValueType) (tftypes.Value, error)

func (s RawState) Unmarshal(typ tftypes.ValueType) (tftypes.Value, error)

// package tftypes
func NewValue(t ValueType, val interface{}) Value

func TypeFromElements(elements []Value) (ValueType, error)

func ValidateValue(t ValueType, val interface{}) error

func (v Value) Type() ValueType

The value types that are implemented within tftypes should then support both ValueType and TypeConstraint.

References

• #97
• #83
• #82
• #133

How to you indicate a computed attribute will change?

Does this documenation exist?

• This is new documentation
• This is an enhancement to existing documentation

Where would you expect to find this documentation?

• On terraform.io
• In the GoDoc for this module
• In this repo as a markdown file
• Somewhere else

Details

Feel like this can go everywhere.

Description

I have got everything working as I expected to using this plugin with the exception of Computed attributes. There is no documentation about how to indicate a computed attribute will change in a Plan or Apply response? It would be super-helpful if there was even the most basic a tutorial on this in the docs.

I actually think even the simplest examples of reconciling plans and applies would be helpful, including how to make use of PriorState, PlannedState and Config in the PlanResourceRequest, as well as what further steps an Apply should take. I am unsure whether I need to do anything in Plan as it is quite duplicative of the work you need to do in Apply.

tftypes: Disallow msgpack unmarshalling to call NewValue(DynamicPseudoType, nil)

terraform-plugin-go version

main branch after #94 is reviewed/merged, but should be handled before next release

Relevant Code

terraform-plugin-go/tftypes/value_msgpack.go

Lines 321 to 326 in 7943864

	switch {
	case length == -1:
	return NewValue(DynamicPseudoType, nil), nil
	case length != 2:
	return Value{}, path.NewErrorf("expected %d elements in DynamicPseudoType array, got %d", 2, length)
	}

Expected Behavior

Code should return error instead of calling NewValue() in manner that will panic.

Actual Behavior

If condition is hit after #94 is merged, code will panic.

Steps to Reproduce

Needs unit test added.

References

• #94

QoL: Make argument to AttributePath.WithElementKeyInt an int (not int64)

terraform-plugin-go

0.2.1

Use cases

When iterating over slices in Go with:

for i, v := range mySlice { ... }

The type of i tends to be int.
Since this seems likely to be the most common source for the value passed into WithElementKeyInt() it'd be nice to not have to always cast it into int64 to make the compiler happy.

Purely from a developer quality of life perspective, I would suggest to make the argument an int.
That is:

func (a *AttributePath) WithElementKeyInt(key int) { ... }

JSON and MsgPack don't handle OptionalAttributes correctly

terraform-plugin-go version

v0.3.1

Expected Behavior

Complex types that have object elements or attributes that contain optional attributes (a map of objects with optional attributes, an object with an object attribute that has optional attributes, etc.) should be surfaced as a value without optional attributes when surfaced from msgpack or JSON.

Actual Behavior

The values retain their type constraints as types, instead of the actual type. Meaning they'd still have optional attributes as part of their type, instead of their concrete type.

References

• #94 discovered this problem but for DynamicPseudoType, which is the other place type constraints come up

Document using helper/resource to test

We should include information in the README on how to test providers written on terraform-plugin-go with the helper/resource testing framework in SDKv2.

Update README to account for removal of ResourceRouter and DataSourceRouter

Does this documentation exist?

• This is new documentation
• This is an enhancement to existing documentation

Where would you expect to find this documentation?

• On terraform.io
• In the GoDoc for this module
• In this repo as a markdown file
• Somewhere else

Details

The README currently instructs developers to use the ResourceRouter and DataSourceRouter types, but these types no longer exist after #24

Implement Unit Tests

We need some unit tests on this functionality, especially around the encoding and decoding stuff.

CircleCI Legacy Convenience Image Deprecation

Description

CircleCI is deprecating certain Docker images, which will no longer receive updates. This repository's configuration (.circleci/config.yml) uses circleci/golang, which should be updated to cimg/go before the end of the year. Migrating to GitHub Actions can be handled at a later time.

References

• https://discuss.circleci.com/t/legacy-convenience-image-deprecation/41034
• https://circleci.com/blog/announcing-our-next-generation-convenience-images-smaller-faster-more-deterministic/

Let's consider getting rid of ResourceRouter and DataSourceRouter

ResourceRouter and DataSourceRouter make it easy to split requests out to be handled by resource or data source specific types. This is helpful, for example, for something like this:

type resource struct {}

func (r resource) ApplyResourceChange(ctx context, req *tfprotov5.ApplyResourceChangeRequest) (*tfprotov5.ApplyResourceChangeResponse, error) {
  // only requests for foo_resource will be handled here
  // other resources will get routed to different functions
}

router := tfprotov5.ResourceRouter{
  "foo_resource": resource{},
}

This makes it so provider developers don't need to handle all resources in the same function, or write their own routing code based on req.TypeName.

But it has a fundamental problem: the provider configuration has no way to get into that method. Or if there is a way, it's convoluted and not intuitive.

We could solve this by having the ResourceRouter change:

type ResourceRouter map[string]func(tfprotov5.ProviderServer) (tfprotov5.ResourceServer, error)

Then provider developers could do something like this:

// myProvider holds provider configuration and fills the tfprotov5.ProviderServer interface
type myProvider struct{}

type resource struct {
  provider myProvider
}

func (r resource) ApplyResourceChange(ctx context, req *tfprotov5.ApplyResourceChangeRequest) (*tfprotov5.ApplyResourceChangeResponse, error) {
  // only requests for foo_resource will be handled here
  // other resources will get routed to different functions
}

// assume the other ResourceServer methods are defined on resource

func newResource(provider tfprotov5.ProviderServer) (tfprotov5.ResourceServer, error) {
  prov, ok := provider.(myProvider)
  if !ok {
    return nil, errors.New("provider is of wrong type, something went wrong")
  }
  return resource{provider: prov}, nil
}

router := tfprotov5.ResourceRouter{
  "foo_resource": newResource,
}

ResourceRouter would then instantiate the instance of the resource, passing in the provider, completing the chain.

But this begs the question: how sure are we that this is the right abstraction? Does this need to live in this module? Could we move it to plugin-sdk or another module that's easier to bump the major version on without losing much, or even anything?

Should we bother reworking it, or just remove it outright and put it somewhere else?

Regardless, I think we're going to want to make the provider available, as it's going to be what has the Terraform version and any information passed in the ConfigureProvider RPC.

QoL: Make tfprotov5.Diagnostic.Attribute a value instead of a pointer

terraform-plugin-go version

v0.2.1

Use cases

Now that AttributePath.With* methods return values and can be chained, it's become a bit awkward to construct
tfprotov4.Diagnostic with it because the Attibute in Diagnostic is a pointer. This requires a additional intermediate variable off of which to collect the pointer.
Here's a real-life example (Kubernetes alpha provider):

for key, present := range validKeys {
    if !present {
        kp := att.WithAttributeName(key)
        resp.Diagnostics = append(resp.Diagnostics, &tfprotov5.Diagnostic{
            Severity:  tfprotov5.DiagnosticSeverityError,
            Summary:   `Required attribute key missing from "manifest" value`,
            Detail:    fmt.Sprintf("'%s' attribute key is missing from manifest configuration", key),
            Attribute: &kp,
        })
    }
}

Proposal

Making tfprotov5.Diagnostic.Attribute be of type tftypes.AttributePath would allow an expression like att.WithAttributeName(key) to be passed directly in the Diagnostic literal, like this:

tfprotov5.Diagnostic{
            Severity:  tfprotov5.DiagnosticSeverityError,
            Summary:   `Required attribute key missing from "manifest" value`,
            Detail:    fmt.Sprintf("'%s' attribute key is missing from manifest configuration", key),
            Attribute: att.WithAttributeName(key),
        }

References

Evaluate well-known errors for compatibility posture

Our well-known errors, like ErrUnsupportedDataSource, are currently set as error values, not types. We may want to evaluate making them less well-known, or returning fmt.Errorf wrapped versions, to force people to use errors.As instead of doing equality comparisons. This would give us a more flexible posture for future changes than we currently have.

DynamicPseudoType nulls aren't handled correctly

terraform-plugin-go version

post merge of #94

Expected Behavior

DynamicPseudoType values should be able to be null or unknown.

Actual Behavior

In the wake of #94, DynamicPseudoType values are only able to be unknown.

OptionalAttributes are not optional

terraform-plugin-go version

v0.3.0

Relevant provider source code

package main

import (
	"github.com/hashicorp/terraform-plugin-go/tfprotov5"
	"github.com/hashicorp/terraform-plugin-go/tftypes"
)

var (
	objType = tftypes.Object{
		AttributeTypes: map[string]tftypes.Type{
			"id":   tftypes.Number,
			"name": tftypes.String,
		},
		OptionalAttributes: map[string]struct{}{
			"name": struct{}{},
		},
	}
)

func main() {
	value := tftypes.NewValue(objType, map[string]tftypes.Value{
		"id":   tftypes.NewValue(tftypes.Number, 1),
	})

	_, err := tfprotov5.NewDynamicValue(value.Type(), value)
	if err != nil {
		panic(err)
	}
}

Terraform Configuration Files

Unrelated

Expected Behavior

I am able to create a DyanmicValue without error.

Actual Behavior

panic: AttributeName("name"): no value set

Steps to Reproduce

Run the provided script.

References

• Possibly related to #82

Panic when transforming an empty tftypes.Value

terraform-plugin-go version

v0.3.1

Relevant provider source code

Confidential and under NDA, unfortunately.

Terraform Configuration Files

Confidential and under NDA, unfortunately.

Expected Behavior

An error should be returned, or the tftypes.Value should be built out to have just the transformed data in it.

Actual Behavior

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x119e65e]
goroutine 25 [running]:
github.com/hashicorp/terraform-plugin-go/tftypes.transform(0xc000341848, 0x0, 0x0, 0x0, 0x0, 0xc0005bec98, 0x0, 0x1d91a28, 0xc000341830, 0x0, ...)
	/[snip]/github.com/hashicorp/[email protected]/tftypes/walk.go:179 +0x37e
github.com/hashicorp/terraform-plugin-go/tftypes.Transform(...)
	/[snip]/github.com/hashicorp/[email protected]/tftypes/walk.go:112

Better error messages when passing an incompatible type to NewDynamicValue

SDK version

v0.1.0

Use cases

It is easy to accidentally pass a tftypes.Type to tftypes.NewDynamicValue that is incompatible with the tftypes.Type of the tftypes.Value that is passed. This can be the result of drift, typos, or just plain user error. When this happens, the user gets a confusing message about an invalid type for the value, which doesn't really indicate where things went wrong. They could've gone wrong when constructing the tftypes.Value, they could've gone wrong when marshaling it, etc.

Attempted solutions

Proposal

A validation check in tfprotov5.NewDynamicValue or tftypes.Value.MarshalMsgPack could pinpoint this specific user error and indicate how to fix it, offering a more friendly debugging experience.

References

Refactor JSON/MsgPack encoding/decoding into tftypes

We currently have a bunch of logic about how to encode and decode tftypes.Values from and to JSON and MsgPack duplicated across tfprotov5 and tfprotov6. We should refactor them into the tftypes package, where both protocol versions can share them.

Consider Adding Type Alias for tftypes.Transform Functions

terraform-plugin-go version

v0.3.1

Use cases

Large downstream implementations of tftypes transformations tend to be wrapped into helper functions.

Attempted solutions

Declaring helper functions with whole function in returns, e.g.

func (s State) setAttributeTransformFunc(ctx context.Context, path *tftypes.AttributePath, tfVal tftypes.Value, diags diag.Diagnostics) (func(*tftypes.AttributePath, tftypes.Value) (tftypes.Value, error), diag.Diagnostics) {

Proposal

It would be great to have a type alias to simplify these declarations, e.g.

type TransformFunc func(*AttributePath, Value) (Value, error)

So implementations can then be:

func (s State) setAttributeTransformFunc(ctx context.Context, path *tftypes.AttributePath, tfVal tftypes.Value, diags diag.Diagnostics) (tftypes.TransformFunc, diag.Diagnostics) {

References

• hashicorp/terraform-plugin-framework#165

MsgPack doesn't handle DynamicPseudoType in complex types correctly

terraform-plugin-go version

v0.3.1

Expected Behavior

Complex types that have DynamicPseudoType elements or attributes (a map of DynamicPseudoTypes, an object with a DynamicPseudoType attribute, etc.) should be surfaced as a value with concrete, known types when unmarshaled from msgpack.

Actual Behavior

The values retain their type constraints as types, instead of the actual type. Meaning they'd still have DynamicPseudoType as their type, instead of their concrete type.

References

• #94 discovered this problem and fixed it for JSON

Handle bool <-> string conversion when unmarshalling JSON

I'm not going to lie, I'm not entirely clear on why we would handle this, but cty does, and compatibility is important, so....

DynamicValue.Unmarshal panics with a null value in DynamicPseudoType from version 0.4.0

terraform-plugin-go version

github.com/hashicorp/terraform-plugin-go v0.4.0

Relevant provider source code

Previously in version 0.3.1 of plugin-go you could unmarshal a MsgPack for a DynamicPseudoType and handle whether a null value was set by the user and return appropriate errors. Now the provider will panic and die.

These two playgrounds show the behaviour for versions 0.3.1 and 0.4.0 with an example MsgPack i took from a unit-test I have.

https://play.golang.org/p/UXllmjPZSHi
Produces:

panic: AttributeName("description"): invalid value tftypes.DynamicPseudoType<null> for tftypes.DynamicPseudoType

goroutine 1 [running]:
github.com/hashicorp/terraform-plugin-go/tftypes.NewValue(...)
	/tmp/gopath2210971451/pkg/mod/github.com/hashicorp/[email protected]/tftypes/value.go:278
github.com/hashicorp/terraform-plugin-go/tftypes.msgpackUnmarshalObject(0xc00010c990, 0xc00010c930, 0xc00011a078)
	/tmp/gopath2210971451/pkg/mod/github.com/hashicorp/[email protected]/tftypes/value_msgpack.go:331 +0x6d4
github.com/hashicorp/terraform-plugin-go/tftypes.msgpackUnmarshal(0xc0001000c0, {0x54d630, 0xc00010c990}, 0x506c20)
	/tmp/gopath2210971451/pkg/mod/github.com/hashicorp/[email protected]/tftypes/value_msgpack.go:129 +0x111b
github.com/hashicorp/terraform-plugin-go/tftypes.msgpackUnmarshalDynamic(0xc00010c3c0, 0x54d720)
	/tmp/gopath2210971451/pkg/mod/github.com/hashicorp/[email protected]/tftypes/value_msgpack.go:357 +0x1d7
github.com/hashicorp/terraform-plugin-go/tftypes.msgpackUnmarshal(0x509800, {0x54d720, 0xc00010c3c0}, 0xd)
	/tmp/gopath2210971451/pkg/mod/github.com/hashicorp/[email protected]/tftypes/value_msgpack.go:50 +0x11c5
github.com/hashicorp/terraform-plugin-go/tftypes.msgpackUnmarshalObject(0xc00010c3f0, 0xc00010c300, 0xc00011a048)
	/tmp/gopath2210971451/pkg/mod/github.com/hashicorp/[email protected]/tftypes/value_msgpack.go:323 +0x446
github.com/hashicorp/terraform-plugin-go/tftypes.msgpackUnmarshal(0xc000106d70, {0x54d630, 0xc00010c3f0}, 0x120)
	/tmp/gopath2210971451/pkg/mod/github.com/hashicorp/[email protected]/tftypes/value_msgpack.go:129 +0x111b
github.com/hashicorp/terraform-plugin-go/tftypes.ValueFromMsgPack({0xc000130000, 0xf3, 0xf3}, {0x54d630, 0xc00010c3f0})
	/tmp/gopath2210971451/pkg/mod/github.com/hashicorp/[email protected]/tftypes/value_msgpack.go:33 +0x149
github.com/hashicorp/terraform-plugin-go/tfprotov5.DynamicValue.Unmarshal({{0xc000130000, 0xf3, 0xf3}, {0x0, 0x0, 0x0}}, {0x54d630, 0xc00010c3f0})
	/tmp/gopath2210971451/pkg/mod/github.com/hashicorp/[email protected]/tfprotov5/dynamic_value.go:84 +0x77
main.main()
	/tmp/sandbox3240024673/prog.go:15 +0x4ca

https://play.golang.org/p/lAKSRaU25YO
Produces

tftypes.Object["class_name":tftypes.String, "configuration":tftypes.DynamicPseudoType, "id":tftypes.String, "name":tftypes.String]<"class_name":tftypes.String<"com.pingidentity.pa.accesstokenvalidators.JwksEndpoint">, "configuration":tftypes.Object["description":tftypes.DynamicPseudoType, "path":tftypes.String, "subjectAttributeName":tftypes.String]<"description":tftypes.DynamicPseudoType<null>, "path":tftypes.String<"/bar/foo">, "subjectAttributeName":tftypes.String<"foo">>, "id":tftypes.String<null>, "name":tftypes.String<"acctest_bar">> <nil>

Terraform code

Example of current behaviour

terraform {
  required_providers {
    pingaccess = {
      source = "iwarapter/pingaccess"
      version = "0.8.0"
    }
  }
}

# this is an example using a dynamic type in terraform allowing for the json
# configuration structure to be written in HCL
resource "pingaccess_access_token_validator" "dynamic" {
  class_name = "com.pingidentity.pa.accesstokenvalidators.JwksEndpoint"
  name       = "demo_two"

  configuration = {
    "description"          = null
    "path"                 = "/bar"
    "subjectAttributeName" = "demo"
  }
}

╷
│ Error: Configuration Validation Failure
│ 
│   with pingaccess_access_token_validator.dynamic,
│   on main.tf line 14, in resource "pingaccess_access_token_validator" "dynamic":
│   14: resource "pingaccess_access_token_validator" "dynamic" {
│ 
│ configuration fields cannot be null, remove 'description' or set a non-null value
╵

With a local dev override using plugin-go 0.4.0

terraform plan
╷
│ Warning: Provider development overrides are in effect
│ 
│ The following provider development overrides are set in the CLI configuration:
│  - iwarapter/pingaccess in /tmp/terraform-provider-pingaccess
│ 
│ The behavior may therefore not match any released version of the provider and applying changes may cause the state to become incompatible with published releases.
╵

Stack trace from the terraform-provider-pingaccess plugin:

panic: AttributeName("description"): invalid value tftypes.DynamicPseudoType<null> for tftypes.DynamicPseudoType

goroutine 42 [running]:
github.com/hashicorp/terraform-plugin-go/tftypes.NewValue(...)
        github.com/hashicorp/[email protected]/tftypes/value.go:278
github.com/hashicorp/terraform-plugin-go/tftypes.msgpackUnmarshalObject(0xc0004e5670, 0xc0007b4bd0, 0xc00000e870, 0xc00081c101, 0xc0001af838, 0x0, 0x0, 0x18b8000, 0xc0007b4a80)
        github.com/hashicorp/[email protected]/tftypes/value_msgpack.go:331 +0xa6c
github.com/hashicorp/terraform-plugin-go/tftypes.msgpackUnmarshal(0xc0004e5670, 0x1ad4268, 0xc00081c030, 0xc00000e870, 0xc0001143b0, 0x0, 0x0, 0x0, 0x0, 0x0)
        github.com/hashicorp/[email protected]/tftypes/value_msgpack.go:129 +0x1d29
github.com/hashicorp/terraform-plugin-go/tftypes.msgpackUnmarshalDynamic(0xc0004e5670, 0xc00000e870, 0xc0007b4a20, 0xc0007b4a01, 0x1ad4358, 0xc0007b49f0, 0x187bcc0, 0xc000114380)
        github.com/hashicorp/[email protected]/tftypes/value_msgpack.go:357 +0x365
github.com/hashicorp/terraform-plugin-go/tftypes.msgpackUnmarshal(0xc0007cf670, 0x1ad4358, 0xc0007b4660, 0xc00000e870, 0x1, 0xc0001143a0, 0x0, 0x1, 0x0, 0x0)
        github.com/hashicorp/[email protected]/tftypes/value_msgpack.go:50 +0x1e91
github.com/hashicorp/terraform-plugin-go/tftypes.msgpackUnmarshalObject(0xc0004e5670, 0xc0007b45a0, 0xc00000e840, 0xc0007b4801, 0x0, 0xed, 0xc0007d2000, 0x0, 0xf0)
        github.com/hashicorp/[email protected]/tftypes/value_msgpack.go:323 +0x471
github.com/hashicorp/terraform-plugin-go/tftypes.msgpackUnmarshal(0xc0007cf670, 0x1ad4268, 0xc0007b4690, 0xc00000e840, 0x194ff80, 0x8, 0x1adc108, 0x187b340, 0xffffffffffffffff, 0x0)
        github.com/hashicorp/[email protected]/tftypes/value_msgpack.go:129 +0x1d29
github.com/hashicorp/terraform-plugin-go/tftypes.ValueFromMsgPack(0xc0007d2000, 0xed, 0xf0, 0x1ad4268, 0xc0007b4690, 0x1ac60d8, 0x203000, 0x116, 0x8, 0x203000, ...)
        github.com/hashicorp/[email protected]/tftypes/value_msgpack.go:33 +0x151
github.com/hashicorp/terraform-plugin-go/tfprotov5.DynamicValue.Unmarshal(0xc0007d2000, 0xed, 0xf0, 0x0, 0x0, 0x0, 0x1ad4268, 0xc0007b4690, 0x30, 0x28, ...)
        github.com/hashicorp/[email protected]/tfprotov5/dynamic_value.go:84 +0x97
github.com/iwarapter/terraform-provider-pingaccess/internal/protocolprovider.valuesFromTypeConfigRequest(0xc00000e828, 0x1ad4268, 0xc0007b4690, 0xc0007b4690, 0xc000370538)
        github.com/iwarapter/terraform-provider-pingaccess/internal/protocolprovider/helper.go:67 +0xa5
github.com/iwarapter/terraform-provider-pingaccess/internal/protocolprovider.genericPluginResource.ValidateResourceTypeConfig(0x0, 0x1acd918, 0xc000120200, 0xc00000e828, 0xc000114301, 0xb800000000000010, 0x10)
        github.com/iwarapter/terraform-provider-pingaccess/internal/protocolprovider/genericPluginResource.go:33 +0x2ee
github.com/iwarapter/terraform-provider-pingaccess/internal/protocolprovider.(*provider).ValidateResourceTypeConfig(0xc00004b980, 0x1acd918, 0xc000120200, 0xc00000e828, 0xc000314528, 0x100f801, 0x10)
        github.com/iwarapter/terraform-provider-pingaccess/internal/protocolprovider/resource_router.go:23 +0x132
github.com/hashicorp/terraform-plugin-mux.SchemaServer.ValidateResourceTypeConfig(0xc00037e3f0, 0xc00037e420, 0xc0001978e0, 0x2, 0x2, 0xc00019c700, 0x1, 0x1acd918, 0xc000120200, 0xc00000e828, ...)
        github.com/hashicorp/[email protected]/schema_server.go:207 +0x98
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ValidateResourceTypeConfig(0xc0001979a0, 0x1acd9c0, 0xc000120200, 0xc0001201c0, 0xc0001979a0, 0xc0007b4180, 0xc000528ba0)
        github.com/hashicorp/[email protected]/tfprotov5/tf5server/server.go:264 +0xcc
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ValidateResourceTypeConfig_Handler(0x19790c0, 0xc0001979a0, 0x1acd9c0, 0xc0007b4180, 0xc000194720, 0x0, 0x1acd9c0, 0xc0007b4180, 0xc0007c6240, 0x116)
        github.com/hashicorp/[email protected]/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:272 +0x214
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000272a80, 0x1ad5c18, 0xc0000ff800, 0xc000370000, 0xc00037e4b0, 0x1f385d0, 0x0, 0x0, 0x0)
        google.golang.org/[email protected]/server.go:1297 +0x52b
google.golang.org/grpc.(*Server).handleStream(0xc000272a80, 0x1ad5c18, 0xc0000ff800, 0xc000370000, 0x0)
        google.golang.org/[email protected]/server.go:1626 +0xd0c
google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc000388150, 0xc000272a80, 0x1ad5c18, 0xc0000ff800, 0xc000370000)
        google.golang.org/[email protected]/server.go:941 +0xab
created by google.golang.org/grpc.(*Server).serveStreams.func1
        google.golang.org/[email protected]/server.go:939 +0x1fd

Error: The terraform-provider-pingaccess plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.
╷
│ Error: Plugin did not respond
│ 
│   with pingaccess_access_token_validator.dynamic,
│   on main.tf line 11, in resource "pingaccess_access_token_validator" "dynamic":
│   11: resource "pingaccess_access_token_validator" "dynamic" {
│ 
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ValidateResourceConfig call. The plugin logs may contain more details.
╵

Allow configuration of gRPC maximum message sizes

terraform-plugin-go version

v0.3.0

Use cases

The Kubernetes manifest provider has to deal with some potentially huge resource configurations. Particularly, the CRD objects of the Strimzi project (Apache Kafka on K8s) are as big as 16K (10^3) lines of HCL.

Because of this, it can run into the gRPC message limits quite easily. This is most likely to happen in the UpgradeResourceState call first, because the previous state payload is transported in raw JSON for this call which is more verbose than the msgpack payload of other protocol calls.

An instance of this happening would look like this to the user:

│ Error: Plugin error
│
│   with kubernetes_manifest.customresourcedefinition_kafkas_kafka_strimzi_io,
│   on crds.tf line 1, in resource "kubernetes_manifest" "customresourcedefinition_kafkas_kafka_strimzi_io":
│    1: resource "kubernetes_manifest" "customresourcedefinition_kafkas_kafka_strimzi_io" {
│
│ The plugin returned an unexpected error from plugin.(*GRPCProvider).UpgradeResourceState: rpc error: code = ResourceExhausted desc = grpc: received message larger than max
│ (4528743 vs. 4194304)

Attempted solutions

I have successfully worked around this issue by maxxing out the message size limits on the gRPC server that terraform-plugin-go instantiates. However, this would require us to consume terraform-plugin-go from a fork and constantly rebase on new releases. This would not be ideal.

The changes I had to make are available here: alexsomesan@6f9312a

Proposal

We would prefer for this mechanism be exposed to consumers of terraform-plugin-go so that we can configure practical limits as they are needed.

References

Make directory names and package names agree

The case I ran in to was server vs tf5server

Possible bug: panic after Ctrl-C during apply

The following issue was raised against Core: hashicorp/terraform#26885

It appears to be an AWS provider panic on receiving Ctrl-C during apply, but further reproduction effort is needed.

Versions

Terraform v0.13.5
terraform-provider-aws must be v3.14.0 or v3.14.1 due to the terraform-plugin-go dependency added in SDK v2.2.0

Nice to have: Generic way to assert if value is a primitive type

When implementing recursive traversal functions, I find myself repeatedly doing checks like:

var val tftypes.Value
...
if val.Is(tftypes.String) || val.Is(tftypes.Number) || val.Is(tftypes.Bool) || val.Is(tftypes.DynamicPseudoType) {
  // no more recursion - do stuff with value and return
}
// Implement recursion for complex types

It would be great to have a more concise way to check if the value is a primitive.
I guess I'm suggesting something like:

func (val *Value ) IsPrimitve() bool {
  return val.Is(tftypes.String) || val.Is(tftypes.Number) || val.Is(tftypes.Bool) || val.Is(tftypes.DynamicPseudoType)
}

Consider Subsystem Logger with Protocol Request and Response Data

terraform-plugin-go version

v0.4.0 (not yet released)

Use cases

Provider and Terraform developers may wish to see the request and response data being passed over gRPC during troubleshooting.

Attempted solutions

Manually replacing the terraform-plugin-go dependency and injecting logging statements.

Proposal

Create a new logging subsystem specifically for protocol request and response data at TRACE log level. This subsystem should default to OFF and should be explicitly enabled by the executor, rather infer the log level like other logging subsystems, to prevent leaking sensitive information. The details of this logging subsystem should be documented along side other logging details for this project.

References

• #93 (comment)

Enhance error messages in marshalMsgPack

SDK version

I am working in a fork of terraform-plugin-go; currently synced with upstream commit 958fd96 (today)

Use cases

I'm working on a provider and find this error message from MarshalMsgPack insufficiently informative:

no value set

Attempted solutions

Proposal

I've only looked at marshalMsgPackObject so far, but I think adding the object key to the error message is a good step forward and something like this can be applied to the other marshal functions:

Current

terraform-plugin-go/tfprotov5/tftypes/value_msgpack.go

Lines 228 to 230 in 958fd96

	if !ok {
	return p.NewErrorf("no value set")
	}

Proposed diff

                if !ok {
-                       return p.NewErrorf("no value set")
+                       return p.NewErrorf("no value set for %s", k)
                }

I will open a PR if this sounds like the right path (and also if you have a different suggestion but are ok with me implementing it).

References

Make type comparisons clearer

We currently have a very overloaded Is method on types that's used for shallow and deep comparisons. With #79 raising the need for deep comparisons that allow concrete types to be substituted for DynamicPseudoTypes, it has become obvious we need a clearer interface for this.

I'm proposing:

• Type.Is(Type) is a shallow check, that root types are the same type
• Type.Equals(Type) is a deep check, that root types and all elements and attributes are the same type
• Type.ConformsTo(Type) is a deep check, that root types and all elements and attributes conform (are equivalent, or a concrete type is being substituted for DynamicPseudoType)

This would lead to less confusion and fewer gotchas around Type.Is(Type), I believe, and more clarity about what code is actually doing.

Enable dependabot

Description

Noticed while submitting #118 that appears dependabot is not enabled for this repository for dependency management. It should be, similar to the other plugin repositories.

Add Walk and Transform to tftypes

Use cases

Walk:

• IsFullyKnown is a special case of Walk as an example
• Validation

Transform:

• Dynamic returns things that may possibly be lists as tuples, especially if there is a field mismatch due to limited info in the dynamic, and in some providers they need to be simplified to lists (kubernetes-alpha for example)
• Applying defaults to an object from schema

Proposal

(similar to cty)

Since these implementations are not opinionated and leverage existing type/path info in the tftypes package, hopefully something that makes more sense in this repo as opposed to an outside one.

tftypes: Consider (AttributePath).LastStep() Method

terraform-plugin-go version

v0.4.0

Use cases

When creating tftypes.Transform() functions, it can be necessary to know the definition of the last attribute path, to determine the methodology for upserting values into the parent value to replace it.

Attempted solutions

This code works, although is verbose:

var childValueDiags diag.Diagnostics
childStep := path.Steps()[len(path.Steps())-1] // <-- this here
parentValue, childValueDiags = upsertChildValue(ctx, parentPath, parentValue, childStep, tfVal)
diags.Append(childValueDiags...)

Proposal

Create a method, opposite of (AttributePath).WithoutLastStep() that returns the last step, e.g.

func (p *AttributePath) LastStep() *AttributePath

References

• https://github.com/hashicorp/terraform-plugin-framework/blob/2cedf45c2bf94bfb82ff65b1402b3fd75a806aa4/tfsdk/plan.go#L266-L269
• https://github.com/hashicorp/terraform-plugin-framework/blob/2cedf45c2bf94bfb82ff65b1402b3fd75a806aa4/tfsdk/state.go#L275-L278
• hashicorp/terraform-plugin-framework#176 (comment)

Panic in AWS acceptance test

I haven't investigated this at all yet - dumping this here to check after full AWS run is kicked off.

$ make testacc TESTARGS='-run=TestAccAWSAcmCertificate'
...
=== CONT  TestAccAWSAcmCertificate_PrivateKey_Tags
panic: Length on non-tuple Type

goroutine 413 [running]:
github.com/hashicorp/go-cty/cty.Type.Length(...)
	/home/katy/dev/go/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/go-cty/cty/tuple_type.go:95
github.com/hashicorp/terraform-plugin-sdk/v2/internal/plugin/convert.tftypeFromCtyType(0x7e0dc20, 0xc000710e00, 0x7e0dc20, 0xc000710e00, 0x6df4400, 0xb5b1bc0)
	/home/katy/dev/go/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform-plugin-sdk/v2/internal/plugin/convert/schema.go:62 +0x652
github.com/hashicorp/terraform-plugin-sdk/v2/internal/plugin/convert.tftypeFromCtyType(0x7e0dba0, 0xc00167e990, 0x6fccdac, 0x9, 0xc001145ab0, 0x0)
	/home/katy/dev/go/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform-plugin-sdk/v2/internal/plugin/convert/schema.go:34 +0x25d
github.com/hashicorp/terraform-plugin-sdk/v2/internal/plugin/convert.ConfigSchemaToProto(0xc000dacb40, 0xc000dacb40)
	/home/katy/dev/go/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform-plugin-sdk/v2/internal/plugin/convert/schema.go:153 +0x1f1
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).GetProviderSchema(0xc00106c2a0, 0x7e0caa0, 0xc0019fce40, 0xb5e20f0, 0xc0019fce40, 0xc000de7d00, 0x7e41040)
	/home/katy/dev/go/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema/grpc_provider.go:82 +0x213
github.com/hashicorp/terraform-plugin-go/tfprotov5/server.(*server).GetSchema(0xc00106c400, 0x7e0cb60, 0xc00082bec0, 0xc00082bef0, 0xc00106c400, 0xc00082bec0, 0xc001dd2ba0)
	/home/katy/dev/go/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform-plugin-go/tfprotov5/server/server.go:147 +0x7c
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_GetSchema_Handler(0x6df1480, 0xc00106c400, 0x7e0cb60, 0xc00082bec0, 0xc000de7d40, 0x0, 0x7e0cb60, 0xc00082bec0, 0x0, 0x0)
	/home/katy/dev/go/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:236 +0x214
google.golang.org/grpc.(*Server).processUnaryRPC(0xc001dca380, 0x7e351e0, 0xc000a7b080, 0xc0013e2600, 0xc001b07cb0, 0xb584b00, 0x0, 0x0, 0x0)
	/home/katy/dev/go/src/github.com/terraform-providers/terraform-provider-aws/vendor/google.golang.org/grpc/server.go:1194 +0x522
google.golang.org/grpc.(*Server).handleStream(0xc001dca380, 0x7e351e0, 0xc000a7b080, 0xc0013e2600, 0x0)
	/home/katy/dev/go/src/github.com/terraform-providers/terraform-provider-aws/vendor/google.golang.org/grpc/server.go:1517 +0xd05
google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc0017d8840, 0xc001dca380, 0x7e351e0, 0xc000a7b080, 0xc0013e2600)
	/home/katy/dev/go/src/github.com/terraform-providers/terraform-provider-aws/vendor/google.golang.org/grpc/server.go:859 +0xa5
created by google.golang.org/grpc.(*Server).serveStreams.func1
	/home/katy/dev/go/src/github.com/terraform-providers/terraform-provider-aws/vendor/google.golang.org/grpc/server.go:857 +0x1fd
FAIL	github.com/terraform-providers/terraform-provider-aws/aws	5.876s
FAIL
make: *** [GNUmakefile:28: testacc] Error 1

Implement comparison of tftypes.Value in the sense of go-cmp

I've repeatedly run into an issue while writing unit tests for code that makes use of tftypes.Number values.

It turns out big.Float values which are backing the tftypes.Number values are pretty tricky to compare correctly. Depending on how the value has been constructed, certain private attributes in big.Float (like precision) end up being set to different values and fail the comparison of otherwise apparently equal values.

Bellow is a quick demonstration of that aspect:

package main

import (
	"fmt"
	"github.com/google/go-cmp/cmp"
	"math/big"
	"reflect"
	"strconv"
)

func main() {
	f, _ := strconv.ParseFloat("12.4", 64)
	a := new(big.Float).SetFloat64(f)

	b := big.NewFloat(12.4)

	c, _ := new(big.Float).SetString("12.4")

	x := cmp.Equal(a, b, cmp.Exporter(func(t reflect.Type) bool { return true }))
	y := cmp.Equal(c, b, cmp.Exporter(func(t reflect.Type) bool { return true }))
	z := cmp.Equal(a, c, cmp.Exporter(func(t reflect.Type) bool { return true }))

	fmt.Printf("a == b ? %v\n", x)
	fmt.Printf("c == b ? %v\n", y)
	fmt.Printf("a == c ? %v\n", z)
}

go-cmp allows for compared types to implement their own comparison mechanism, as described here:
https://pkg.go.dev/github.com/google/go-cmp/cmp#Equal

It would be a nice UX improvement for tftypes.Values to implement the Equal(..) method so they would just work as expected out of the box with go-cmp.

Handle type aliases in As

"Aliases" (not real aliases, not type Type2 = Type1, but type Type2 Type1) are handled transparently in cty using the reflect package. See zclconf/go-cty#18 for more details. We could, in theory, support this in As. Should we? Is it worth the added complexity? We've managed to avoid reflect for the most part so far, and I kind of like that simplicity. Things start getting hard to reason about as soon as reflect is introduced. But I could be convinced that

• This is an important feature to include
• This must be included at this level and cannot be done with a helper or at a higher level of abstraction

Unable to Unmarshal RawState to a type with OptionalAttributes

terraform-plugin-go version

v0.3.0

Relevant provider source code

package main

import (
	"github.com/hashicorp/terraform-plugin-go/tfprotov5"
	"github.com/hashicorp/terraform-plugin-go/tftypes"
)

var (
	roleType = tftypes.Object{
		AttributeTypes: map[string]tftypes.Type{
			"id":   tftypes.Number,
			"name": tftypes.String,
		},
		OptionalAttributes: map[string]struct{}{
			"name": struct{}{},
		},
	}

	objType = tftypes.Object{
		AttributeTypes: map[string]tftypes.Type{
			"id":   tftypes.Number,
			"name": tftypes.String,
			"role": tftypes.Map{AttributeType: roleType},
		},
	}
)

func main() {
	value := tftypes.NewValue(objType, map[string]tftypes.Value{
		"id":   tftypes.NewValue(tftypes.Number, 1),
		"name": tftypes.NewValue(tftypes.String, "foo"),
		"role": tftypes.NewValue(tftypes.Map{AttributeType: roleType}, map[string]tftypes.Value{
			"bar": tftypes.NewValue(roleType, map[string]tftypes.Value{
				"id":   tftypes.NewValue(tftypes.Number, 1),
				"name": tftypes.NewValue(tftypes.String, "baz"),
			}),
		}),
	})

	_, err := tfprotov5.NewDynamicValue(value.Type(), value)
	if err != nil {
		panic(err) // This should not error
	}

	rawState := &tfprotov5.RawState{JSON: []byte(`{"id":1,"name":"foo","role":{"bar":{"id":1,"name":"baz"}}}`)}
	_, _ = rawState.Unmarshal(objType)
}

Terraform Configuration Files

Unrelated

Expected Behavior

The above script should not panic. Basically, I feel like should be able to use the same tftypes.Type to generate a tftypes.Value that I pass into RawState.Unmarshal when I expect the state to match.

Actual Behavior

panic: ElementKeyString("bar"): can't use 
  tftypes.Object["id":tftypes.Number, "name":tftypes.String]
as 
  tftypes.Object["id":tftypes.Number, "name":tftypes.String?]

Steps to Reproduce

Run the supplied script

References

• https://discuss.hashicorp.com/t/new-sdk-unmarshall-rawstate-with-optionalattributes-fails/23856

Add support for OptionalAttrs inside tftypes.Objects

SDK version

build from main

Use cases

Terraform (via go-cty) includes support for optional attributes inside object-type attributes. Extending terraform-plugin-go allows provider developers to write object-type Attributes (in place of blocks) that have optional attributes.

Attempted solutions

The current method for handling situations like this is to create a Block with optional attributes, but there are times where an Attribute may be preferable to a Block (for example if explicitly setting a single object to null could be meaningful).

Proposal

Something along the lines of this patch, but with better documentation and (I'd guess) also updating the Is function to compare OptionalAttrs:

diff --git a/tfprotov5/tftypes/object.go b/tfprotov5/tftypes/object.go
index 7b18c9e..fa8816b 100644
--- a/tfprotov5/tftypes/object.go
+++ b/tfprotov5/tftypes/object.go
@@ -9,6 +9,7 @@ import "encoding/json"
 // attribute names or types are considered to be distinct types.
 type Object struct {
        AttributeTypes map[string]Type
+       OptionalAttrs  []string
 }

 // Is returns whether `t` is an Object type or not. If `t` is an instance of
@@ -52,5 +53,15 @@ func (o Object) MarshalJSON() ([]byte, error) {
        if err != nil {
                return nil, err
        }
+
+       optionalAttrs, err := json.Marshal(o.OptionalAttrs)
+       if err != nil {
+               return nil, err
+       }
+
+       if len(optionalAttrs) > 0 {
+               return []byte(`["object",` + string(attrs) + "," + string(optionalAttrs) + `]`), nil
+       }
+
        return []byte(`["object",` + string(attrs) + `]`), nil
 }

I can open a PR if this is a direction you're interested in.

References

Implement Terraform plugin client

terraform-plugin-go version

{
	"Path": "github.com/hashicorp/terraform-plugin-go",
	"Version": "v0.2.1",
	"Replace": {
		"Path": "github.com/pazderak/terraform-plugin-go",
		"Version": "v0.2.1",
		"Time": "2021-01-07T17:53:07Z",
		"Dir": "/home/kpazdera/go/pkg/mod/github.com/pazderak/[email protected]",
		"GoMod": "/home/kpazdera/go/pkg/mod/cache/download/github.com/pazderak/terraform-plugin-go/@v/v0.2.1.mod"
	},
	"Dir": "/home/kpazdera/go/pkg/mod/github.com/pazderak/[email protected]",
	"GoMod": "/home/kpazdera/go/pkg/mod/cache/download/github.com/pazderak/terraform-plugin-go/@v/v0.2.1.mod"
}

Use cases

I want to create a separate binary which will be able to use Terraform plugins. This binary will do the import of the existing resource to Terraform state. Finding of resources using Terraform plugin system will provide an unified way how to check a resource existence.

Attempted solutions

Proposal

Implementing a client counterpart to the existing GRPC server will allow to re-use the existing code. Client itself can be used in the main terraform binary instead of existing GRPC client or in any third-party binary.

References

Fix AttributePath situation

We currently have tfprotov5.AttributePath, which is what Diagnostics use to return information about which field specifically caused an error. And we defined a tfprotov5.AttributePathStepper interface and tfprotov5.WalkAttributePath function that will figure out what an attribute path is pointing to.

But our tftypes package (mostly) uses tftypes.Path to talk about errors.

We should probably figure out our UX here. I think right now my leaning is that tftypes.Path should be used by everything, and it'll get converted to a tfprotov5.AttributePathStepper at the last minute. to be included in the response.

Consider making DiagnosticSeverity an int

Right now DiagnosticSeverity is a uint32, but there's not really any specific reason for that. Maybe it should be an int, or a byte?

Write protoc script that uses docker

Potentially namely/protoc

This will simplify the ability for people to generate the code or for us to do comparisons in CI without having to install all the tooling manually.