zclconf / go-cty

A type system for dynamic values in Go applications

cty

cty (pronounced "see-tie", emoji: 👀 👔, IPA: /si'tʰaɪ/) is a dynamic type system for applications written in Go that need to represent user-supplied values without losing type information. The primary intended use is for implementing configuration languages, but other uses may be possible too.

One could think of cty as being the reflection API for a language that doesn't exist, or that doesn't exist yet. It provides a set of value types and an API for working with values of that type.

Fundamentally what cty provides is equivalent to an interface{} with some dynamic type information attached, but cty encapsulates this to ensure that invariants are preserved and to provide a more convenient API.

As well as primitive types, basic collection types (lists, maps and sets) and structural types (object, tuple), the cty type and value system has some additional, optional features that may be useful to certain applications:

• Representation of "unknown" values, which serve as a typed placeholder for a value that has yet to be determined. This can be a useful building-block for a type checker. Unknown values support all of the same operations as known values of their type, but the result will often itself be unknown.

• Representation of values whose types aren't even known yet. This can represent, for example, the result of a JSON-decoding function before the JSON data is known.

Along with the type system itself, a number of utility packages are provided that build on the basics to help integrate cty into calling applications. For example, cty values can be automatically converted to other types, converted to and from native Go data structures, or serialized as JSON.

For more details, see the following documentation:

• Concepts
• Full Description of the cty Types
• API Reference (godoc)
• Conversion between cty types
• Conversion to and from native Go values
• JSON serialization
• cty Functions system
• Compatibility Policy for future Minor Releases: please review this before using cty in your application to avoid depending on implementation details that may change.

---

License

Copyright 2017 Martin Atkins

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

go-cty's Issues

Nested go struct to cty value fails

I am trying to convert go struct (which is composed of other structs) into a city value
and It fails

json data i am sending

{
      "hostname": "x",
      "target_node": "x",
      "is_dc": true,
      "specs": {
        "cpu": 4,
        "memory": 9216
      },
      "source": "x",
      "disk": [
        {
          "boot_disk_pool": "x",
          "boot_disk_size": "x",
          "mount": "x"
        }
      ]
}

type Specs struct {
	Cpu    int `json:"cpu" cty:"cpu"`
	Memory int `json:"memory" cty:"memory"`
}

type VM struct {
	Hostname   string `json:"hostname" cty:"hostname"`
	TargetNode string `json:"target_node" cty:"target_node"`
	IsDc       bool   `json:"is_dc" cty:"is_dc"`
	Specs      Specs  `json:"specs" cty:"specs"`
}

func GetType() cty.Type {
	return cty.List(cty.Object(
		map[string]cty.Type{
			"hostname":    cty.String,
			"target_node": cty.String,
			"is_dc":       cty.Bool,
			"specs": cty.Map(cty.Object(
				map[string]cty.Type{
					"cpu":    cty.Number,
					"memory": cty.Number,
				},
			)),
		},
	))
}

var x []VM
_ = json.Unmarshal(JSON, &x)

result, err := gocty.ToCtyValue(x, Type)
if err != nil {
	panic(err)
}

error i am getting

can't convert Go struct to cty.Map(cty.Object(map[string]cty.Type{"cpu":cty.Number, "memory":cty.Number}))

can you please provide an example, plus the list of maps too if it is possible
PS: new to golang

FormatList stdlib function doesn't handle unknown sequence arguments

The following case is used when testing each of the arguments to see if it is a sequence that we'll iterate over, and to check its length:

go-cty/cty/function/stdlib/format.go

Lines 86 to 87 in d006e45

	case (argTy.IsListType() || argTy.IsSetType() || argTy.IsTupleType()) && !arg.IsNull():
	thisLen := arg.LengthInt()

This doesn't account for the situation where the value is of a sequence type but it is unknown, and so the LengthInt call here will panic in that case.

This must be handled slightly differently for lists and sets vs. tuples:

• For lists and sets, an unknown value has an unknown length, and so the overall result of this function can't be anything other than cty.UnknownVal(cty.List(cty.String)) because the length of the result can't be predicted.
• For tuples, the length of the sequence is part of the tuple type and so we can predict the length even if the exact value isn't known. However, we still can't iterate over an unknown tuple, and so the result in this case would be a known list of strings whose elements are themselves all unknown string values.

Given that the two above cases can mix, probably the best solution here is to just have a flag that we set for each case during iteration and then handle them both just before the iterLen == 0 check that follows, giving preference to the list/set outcome if both lists and tuples are present.

Coalesce function that now returns empty strings breaks lookup pattern

With TF the following pattern can be used:

${ coalesce(lookup(cloud.tags, "OrgXID", ""), lookup(cloud.tags, "org-xid", "")) }

Coalesce returning the empty string breaks the pattern.
Unfortunately lookup, does not accept null for the default argument which is required.

Is there another way to look for a value that may not exist, in a cty map? It seems like if the change to coalesce is desired then lookup should allow a default that is null.

third param for `lookup` function is not optional

The documentation for the lookup() stdlib function states:

There are two required arguments, map and key, plus an optional default, which is a value to return if no key is found in map.

However, if I call lookup({a = 1, b = 2}, "a") without a third argument, I get the error

Not enough function arguments; Function "lookup" expects 3 argument(s). Missing value for "default".

The function specification lists exactly three Params and no VarParam.

1.11.0 causes errors with //go:generate packer-sdc mapstructure-to-hcl2

Details: hashicorp/packer-plugin-sdk#128

Extend FormatDateFunc to support seconds since Unix Epoch

It would be handy to be able to convert a provided date to a Unix timestamp. I'm using packer which wraps this function via its timestamp() function.

How to modify a cty.ObjectVal?

I am new to Go and to go-cty, and I am wondering how to add a new element into a["type"] based on the following sample:

a := map[string]cty.Value{
	"type": cty.ObjectVal(map[string]cty.Value{
		"name": cty.ObjectVal(map[string]cty.Value{
			"id": cty.StringVal("value"),
		}),
	}),
}

I tried something like a["type"]["test"] = cty.StringVal("value") and a["type"].AsValueMap()["test"] = cty.StringVal("test2") but it didn't work.

How can I add a new element into a["type"]? Thanks

Nested conversion using gocty.FromCtyValue fails

Given a spec:

var mapDetailsSpec = &hcldec.ObjectSpec{
	"width": &hcldec.AttrSpec{
		Name:     "width",
		Required: false,
		Type:     cty.Number,
	},
	"font_size": &hcldec.AttrSpec{
		Name:     "font_size",
		Required: false,
		Type:     cty.Number,
	},
}

I can parse basic HCL:

width = 100
font_size = 9

	parser := hclparse.NewParser()
	f, diags := parser.ParseHCL([]byte(data), filename)
	val, moreDiags := hcldec.Decode(f.Body, mapDetailsSpec, nil)

Then converting to a golang struct works:

type MapDetails struct {
	Width    int `cty:"width"`
	FontSize int `cty:"font_size"`
}

	var mapDetails MapDetails
	err := gocty.FromCtyValue(val, &mapDetails)

However, when I try to do a nested structure:

size {
	width = 100
	font_size = 9
}

var mapDetailsSpec = &hcldec.ObjectSpec{
	"size": &hcldec.BlockListSpec{
		TypeName: "size",
		Nested: &hcldec.ObjectSpec{
			"width": &hcldec.AttrSpec{
				Name:     "width",
				Required: false,
				Type:     cty.Number,
			},
			"font_size": &hcldec.AttrSpec{
				Name:     "font_size",
				Required: false,
				Type:     cty.Number,
			},
		},
	},
}

I haven't been able to convert to a native golang struct:

type MapDetails struct {
	Size Size `cty:"size"`
}

type Size struct {
	Width    *int `cty:"width"`
	Height   *int `cty:"height"`
	Margin   *int `cty:"margin"`
	FontSize *int `cty:"font_size"`
}

and gocty.FromCtyValue always fails with "object or tuple value is required".

Add a deepmerge function

Since Terraform mainly uses functions provided by go-cty, the need to introduce a deepmerge as a builtin function (hashicorp/terraform#31815) could be satisfied by go-cty and help standardize the implementation of deepmerge instead of relying on providers to implement it in an expected way.

function/stdlib: SetSymmetricDifferenceFunc is actually SetSubtractFunc

Due to what I assume was a copy-paste error, the SetSymmetricDifferenceFunc seems to actually be implementing SetSubtractFunc due to calling the wrong method on the underlying set implementation.

go-cty/cty/function/stdlib/set.go

Lines 88 to 105 in f21e35a

	var SetSymmetricDifferenceFunc = function.New(&function.Spec{
	Params: []function.Parameter{
	{
	Name: "first_set",
	Type: cty.Set(cty.DynamicPseudoType),
	AllowDynamicType: true,
	},
	},
	VarParam: &function.Parameter{
	Name: "other_sets",
	Type: cty.Set(cty.DynamicPseudoType),
	AllowDynamicType: true,
	},
	Type: setOperationReturnType,
	Impl: setOperationImpl(func(s1, s2 cty.ValueSet) cty.ValueSet {
	return s1.Subtract(s2)
	}),
	})

There are also no unit tests for either SetSubtractFunc or SetSymmetricDifferenceFunc.

Configurable struct tags

For Nomad we'd love to be able to configure the struct field tag cty uses like the BasicHandle.TypeInfos field in codec.

With codec we can configure our codec with:

	h := &codec.MsgpackHandle{}
	h.TypeInfos = codec.NewTypeInfos([]string{"cty", "codec"})

...to enable struct field tag compatibility.

Ideally we could set the struct field tag on both to hcl to avoid confusion for users implementing plugins.

Example

With zero tag overriding our struct fields would need both tags:

	RunFor time.Duration `cty:"run_for" codec:"run_for"`

With codec's we can at least reduce that number to 1 tag:

	RunFor time.Duration `cty:"run_for"`

However "cty" means nothing to Nomad users, so replacing it with "hcl" seems optimal:

	RunFor time.Duration `hcl:"run_for"`

Incorrect $ parsing as string element

How repoduce:
Go to https://play.golang.org/
Put that code there:

package main
import (
	"fmt"
	"github.com/hashicorp/hcl2/hclwrite"
	"github.com/zclconf/go-cty/cty"
)
func main() {
	f := hclwrite.NewEmptyFile()
	rootBody := f.Body()
	rootBody.SetAttributeValue("hello", cty.StringVal("${world}"))
	fmt.Printf("%s", f.Bytes())
}

Receive:

hello = "$${world}"

Expected:
hello = "${world}"

I was trying a couple ways to escape that, but it doesn't work.

cty/json: configure (non-)HTML-escaping serialization

marshal() uses json.Marshal() which encodes using HTMLEscape:

So that the JSON will be safe to embed inside HTML <script> tags, the string is encoded using HTMLEscape, which replaces "<", ">", "&", U+2028, and U+2029 are escaped to "\u003c","\u003e", "\u0026", "\u2028", and "\u2029". This replacement can be disabled when using an Encoder, by calling SetEscapeHTML(false).

This behaviour is good for HTML settings.

The documentation for func (*Encoder) SetEscapeHTML states

In non-HTML settings where the escaping interferes with the readability of the output, SetEscapeHTML(false) disables this behavior.

We use cty in a non-HTML setting. So it would be nice to be able to configure whether HTML escaping is applied (using json.Marshal()) or not (e.g. using a custom Encoder with SetEscapeHTML(false)).

function/stdlib: TestCSVDecode fails with Go tip

On Go tip, the future Go 1.17, we've clarified the ParseError.Column field to be a 1-based byte index, to match the new Reader.FieldPos method. Previously the Column field was a somewhat inconsistent rune index. This has caused a test failure:

--- FAIL: TestCSVDecode (0.00s)
    --- FAIL: TestCSVDecode/CSVDecode(cty.StringVal("invalid\"thing\"")) (0.00s)
        csv_test.go:93: wrong error
            got:  parse error on line 1, column 8: bare " in non-quoted-field
            want: parse error on line 1, column 7: bare " in non-quoted-field
FAIL
exit status 1
FAIL	github.com/zclconf/go-cty/cty/function/stdlib	0.014s

In order to keep working with the future Go 1.17, this test should accept column 8 as well as column 7, or just ignore the column field.

Let me know if I can help. Thanks.

Marshal() generate invalid JSON

Hello,
during my investigation of a bug in a different project, noticed that exist cases when Marshal() can generate invalid JSON

...
	valMap := map[string]cty.Value{}
	valMap["test"] = cty.NullVal(cty.DynamicPseudoType)
	valMap["test2"] = cty.UnknownVal(cty.DynamicPseudoType)

	valMapAsCty, err := gocty.ToCtyValue(valMap, generateTypeFromValuesMap(valMap))
	if err != nil {
		log.Fatal(err)
	}

	jsonBytes, err := ctyjson.Marshal(valMapAsCty, cty.DynamicPseudoType)
	if err != nil {
		log.Fatal(err)
	}
	log.Println(string(jsonBytes))
...

Output:

2023/09/11 11:51:03 {"value":{"test":null,"test2":,"type":["object",{"test":"dynamic","test2":"dynamic"}]}

I would expect to fail with an error or generate a valid JSON

End-to-end example: https://github.com/denis256/go-tests/blob/master/zclconf-go-cty/main.go

panic with `AsString()` on result of json encoding of to_number conversion of null value

See code in https://go.dev/play/p/Y3j39bAiRKq:

package main

import (
	"fmt"

	"github.com/zclconf/go-cty/cty"
	"github.com/zclconf/go-cty/cty/function/stdlib"
)

func main() {
	toNumberFn := stdlib.MakeToFunc(cty.Number)
	nullVal := cty.NullVal(cty.DynamicPseudoType)
	args := []cty.Value{nullVal}
	n, err := toNumberFn.Call(args)
	if err != nil {
		panic(err)
	}
	fmt.Printf("result of toNumber(): %#v\n", n)

	args = []cty.Value{n}
	// obj := cty.ObjectVal(map[string]cty.Value{"n": n})
	// args = []cty.Value{obj}
	j, err := stdlib.JSONEncodeFunc.Call(args)
	if err != nil {
		panic(err)
	}
	fmt.Printf("result of json encode: %#v\n", j)
	fmt.Printf("result of json encode as string: %#v\n", j.AsString())
}

If I pass a cty.NullVal(cty.DynamicPseudoType) as param to a stdlib.MakeToFunc(cty.Number) conversion function and then pass the result to stdlib.JSONEncodeFunc I get a result (cty.UnknownVal(cty.String).RefineNotNull()) which cannot be "stringified" with .AsString(). Instead AsString() panics with

panic: value is unknown

Is this the intended behaviour?

Add support for decoding into structs with a custom tag

I'd like a feature for a project that I'm working on to be able to do something like gocty.FromCtyValue, but for a struct with field tags of a kind other than cty:"". Is this within the scope of this project / compatible with design goals?

I've taken a stab at it here - that branch updates func structTagIndices to take a tag string instead of always using cty, and adds a new function func FromCtyValueTagged that takes a new parameter tag string

My project uses this function to decode values into structs that are provided by a plugin and everything seems to work ( +cty tests pass )

How can I get Value as interface

How can I get the value in Value struct as interface?

hashicorp/packer-plugin-sdk incompatible with zclconf/go-cty v1.11.0

Overview of the Issue

If the package github.com/zclconf/go-cty v1.11.0 is used with the Packer SDK, then an error is thrown. I am unsure if this is an error with Packer SDK or go-cty. If the error is with the other package, then please redirect me. I am opening this issue on both trackers.

Reproduction Steps

Do anything with a Packer plugin built with go-cty v1.11.0 as this occurs immediately after Packer loads the plugin.

Packer Plugin SDK/go-cty versions

packer-sdk 0.3.1 and 0.3.2 both throw this error.
go-cty v1.10.0 works perfectly. v1.11.0 throws this error.

Operating system and Environment details

go 1.18

Log Fragments and crash.log files

panic: ConfigSpec failed: gob: type cty.Type has no exported fields [recovered]
	panic: ConfigSpec failed: gob: type cty.Type has no exported fields

goroutine 1 [running]:
log.Panic({0xc000b3c2a8, 0xc000b3c2e8, 0xc000b3c318})
	/opt/hostedtoolcache/go/1.17.11/x64/src/log/log.go:354 +0x65
github.com/hashicorp/packer/packer.(*cmdProvisioner).checkExit(0x90, {0x43e1e60, 0xc00052daa0}, 0x0)
	/home/runner/work/packer/packer/packer/cmd_provisioner.go:47 +0x7f
github.com/hashicorp/packer/packer.(*cmdProvisioner).ConfigSpec.func1()
	/home/runner/work/packer/packer/packer/cmd_provisioner.go:19 +0x39
panic({0x43e1e60, 0xc00052daa0})
	/opt/hostedtoolcache/go/1.17.11/x64/src/runtime/panic.go:1038 +0x215
github.com/hashicorp/packer-plugin-sdk/rpc.(*commonClient).ConfigSpec(0xc0007905e0)
	/home/runner/go/pkg/mod/github.com/hashicorp/[email protected]/rpc/common.go:44 +0x297
github.com/hashicorp/packer/packer.(*cmdProvisioner).ConfigSpec(0xc000b3e0e0)
	/home/runner/work/packer/packer/packer/cmd_provisioner.go:22 +0x65
github.com/hashicorp/packer/hcl2template.decodeHCL2Spec({0x5ce8d10, 0xc000725e60}, 0xc00053e9a8, {0x7f7dd9ff6450, 0xc00053e9a8})
	/home/runner/work/packer/packer/hcl2template/decode.go:17 +0x39
github.com/hashicorp/packer/hcl2template.(*HCL2Provisioner).HCL2Prepare(0xc0005da5a0, 0xc0005da570)
	/home/runner/work/packer/packer/hcl2template/types.hcl_provisioner.go:51 +0x439
github.com/hashicorp/packer/hcl2template.(*PackerConfig).startProvisioner(0xc00056bb80, {{{0xc0005340f7, 0x6}, {0xc0005340fe, 0x6}}, {0x0, 0x0}, {0x5ce8d10, 0xc000724b10}}, 0xc0005526c0, ...)
	/home/runner/work/packer/packer/hcl2template/types.build.provisioners.go:199 +0x6a5
github.com/hashicorp/packer/hcl2template.(*PackerConfig).getCoreBuildProvisioner(0x0, {{{0xc0005340f7, 0x6}, {0xc0005340fe, 0x6}}, {0x0, 0x0}, {0x5ce8d10, 0xc000724b10}}, 0xc0005526c0, ...)
	/home/runner/work/packer/packer/hcl2template/types.packer_config.go:482 +0x105
github.com/hashicorp/packer/hcl2template.(*PackerConfig).getCoreBuildProvisioners(0xc0005340fe, {{{0xc0005340f7, 0x6}, {0xc0005340fe, 0x6}}, {0x0, 0x0}, {0x5ce8d10, 0xc000724b10}}, {0xc0005300e0, ...}, ...)
	/home/runner/work/packer/packer/hcl2template/types.packer_config.go:470 +0x1cb
github.com/hashicorp/packer/hcl2template.(*PackerConfig).GetBuilds(0xc00056bb80, {{0x0, 0x0, 0x0}, {0x0, 0x0, 0x0}, 0x0, 0x0, {0x0, ...}, ...})
	/home/runner/work/packer/packer/hcl2template/types.packer_config.go:670 +0x14fc
github.com/hashicorp/packer/command.(*BuildCommand).RunContext(0xc000c68ba0, {0x5ce83e0, 0xc000c6c980}, 0xc0007967e0)
	/home/runner/work/packer/packer/command/build.go:182 +0x21e
github.com/hashicorp/packer/command.(*BuildCommand).Run(0xc000c68ba0, {0xc00006e0a0, 0x2, 0x2})
	/home/runner/work/packer/packer/command/build.go:40 +0xc5
github.com/mitchellh/cli.(*CLI).Run(0xc0003bca00)
	/home/runner/go/pkg/mod/github.com/mitchellh/[email protected]/cli.go:262 +0x5f8
main.wrappedMain()
	/home/runner/work/packer/packer/main.go:262 +0xb28
main.realMain()
	/home/runner/work/packer/packer/main.go:49 +0xf3
main.main()
	/home/runner/work/packer/packer/main.go:35 +0x19

Additional Information

I suspect this is referencing the cty.Type from the <plugin>.hcl2spec.go. If I attempt to regenerate that file with go-cty v1.11.0, then the following error is thrown:

mapstructure-to-hcl2: main.[]: Unexpected package creation during export data loading

Possible bug: Vendor conflict

I am new to go, my understanding of this error message is that somewhere in my code it is registering from the /vendor and somewhere it is getting the absolute path.

panic: gob: registering duplicate types for "github.com/zclconf/go-cty/cty.primitiveType": cty.primitiveType != cty.primitiveType

goroutine 1 [running]:
encoding/gob.RegisterName(0xc0001b4450, 0x2b, 0x3ff4e40, 0xc000422b93)
        /usr/local/Cellar/go/1.12.9/libexec/src/encoding/gob/type.go:820 +0x55e
github.com/terraform-providers/terraform-provider-alicloud/vendor/github.com/zclconf/go-cty/cty.init.3()
        /Users/souradeepnanda/go/src/github.com/terraform-providers/terraform-provider-alicloud/vendor/github.com/zclconf/go-cty/cty/types_to_register.go:52 +0x56a

Is there a way to not make it panic?

Why is cty.primitiveType != cty.primitiveType?

stdlib: SetProductFunc doesn't seem to handle refinements quite right

The Terraform team at HashiCorp accidentally upgraded to cty v1.13 and got refinements in a patch release. They plan to revert back to v1.12 for now but in the process they learned about hashicorp/terraform#32853 which they suspect is a bug in SetProductFunc's handling of refinements.

I've not yet reproduced this to confirm, but if refinements are the cause then it seems like the function is trying to apply collection-type-specific refinements to a value that isn't of a collection type.

cty.StringVal always doubles $ in `${}` output

Hello! Apologies if I'm missing something obvious, but I keep running into a strange issue. I am trying to print some dynamic HCL so terraform can inject a var into a string.

Expected output - query = ${thing.thing.id} but
body.SetAttributeValue("query", cty.StringVal("${thing.thing.id}")) produces
query = "$${thing.thing.id}"

Can anyone help me understand why this is happening and how to avoid it?
Thank you!

cty/json: Allow unsupported fields in json

Currently when we unmarshal json using cty/json to an object, all fields in the type must match the incoming json:

https://play.golang.org/p/REla4iDJbaQ

While i suppose you could call this a feature (the data doesn't match the type), it makes it difficult to allow the type to be forwards compatible with external json. I would expect it to behave similar to the go stdlib json, where fields that are not set in the target struct are silently ignored.

I think changing the check to skip the field rather than return an error would do this (but that's probably a backwards incompatible change?):

go-cty/cty/json/unmarshal.go

Lines 313 to 316 in 15177b9

	aty, ok := atys[k]
	if !ok {
	return cty.NilVal, objPath.NewErrorf("unsupported attribute %q", k)
	}

I'm happy to put together a PR but would like to get your thoughts first and also what you think the best solution would be.

function/stdlib: The setproduct function panics when an argument is an empty list

Hi,

Digging into an issue in Terraform: hashicorp/terraform#28524 which was caused by them upgrading from go-cty v1.8.1 to v1.8.2, I found an issue here.

When one of the argument of the setproduct is an empty list, it results in this panic:

Running tool: /usr/local/bin/go test -timeout 30s -run ^TestSetproduct$ github.com/zclconf/go-cty/cty/function/stdlib

--- FAIL: TestSetproduct (0.00s)
    --- FAIL: TestSetproduct/Setproduct([]cty.Value{cty.ListValEmpty(cty.EmptyObject),_cty.ListVal([]cty.Value{cty.StringVal("quick"),_cty.StringVal("fox")})}) (0.00s)
        /Users/matthieu/Documents/projects/oss/go-cty/cty/function/stdlib/collection_test.go:2187: unexpected error: panic in function implementation: runtime error: hash of unhashable type cty.ValueMarks
            goroutine 8 [running]:
            runtime/debug.Stack(0xc0000746c8, 0x12b5120, 0xc00015d3c0)
            	/usr/local/Cellar/go/1.15.2/libexec/src/runtime/debug/stack.go:24 +0x9f
            github.com/zclconf/go-cty/cty/function.errorForPanic(...)
            	/Users/matthieu/Documents/projects/oss/go-cty/cty/function/error.go:44
            github.com/zclconf/go-cty/cty/function.Function.Call.func1(0xc000074e98, 0xc000074eb8)
            	/Users/matthieu/Documents/projects/oss/go-cty/cty/function/function.go:291 +0x95
            panic(0x12b5120, 0xc00015d3c0)
            	/usr/local/Cellar/go/1.15.2/libexec/src/runtime/panic.go:969 +0x175
            github.com/zclconf/go-cty/cty.Value.Mark(0x13a5ee0, 0xc00015d3b0, 0x12a4260, 0xc000164e60, 0x12c03e0, 0x0, 0x12e8ea0, 0xc000163980, 0x0, 0xc000074c38)
            	/Users/matthieu/Documents/projects/oss/go-cty/cty/marks.go:208 +0x1ce
            github.com/zclconf/go-cty/cty/function/stdlib.glob..func33(0xc00015f100, 0x2, 0x2, 0x13a5ee0, 0xc00015d3a0, 0xc00015d3a0, 0x0, 0x0, 0x868cf, 0x868cf, ...)
            	/Users/matthieu/Documents/projects/oss/go-cty/cty/function/stdlib/collection.go:956 +0xf05
            github.com/zclconf/go-cty/cty/function.Function.Call(0xc00006d2c0, 0xc00015f100, 0x2, 0x2, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
            	/Users/matthieu/Documents/projects/oss/go-cty/cty/function/function.go:295 +0x51a
            github.com/zclconf/go-cty/cty/function/stdlib.SetProduct(...)
            	/Users/matthieu/Documents/projects/oss/go-cty/cty/function/stdlib/collection.go:1411
            github.com/zclconf/go-cty/cty/function/stdlib.TestSetproduct.func1(0xc000001980)
            	/Users/matthieu/Documents/projects/oss/go-cty/cty/function/stdlib/collection_test.go:2177 +0x88
            testing.tRunner(0xc000001980, 0xc00015d350)
            	/usr/local/Cellar/go/1.15.2/libexec/src/testing/testing.go:1127 +0xef
            created by testing.(*T).Run
            	/usr/local/Cellar/go/1.15.2/libexec/src/testing/testing.go:1178 +0x386
FAIL
FAIL	github.com/zclconf/go-cty/cty/function/stdlib	0.423s
FAIL

Some valid `Path` values cannot be applied to their source `Value`

A Value containing a set cannot use the path argument within a Transform callback for all paths.

Given an object like so:

ObjectVal(map[string]Value{
  "set": SetVal([]Value{
    ObjectVal(map[string]Value{
      "attr": StringVal("val"),
    }),
  }),
})

Using Transform to obtain each step's value from another object will encounter a path which cannot be used.

// using the same value to ensure all paths are valid, and should result in the same value
Transform(val, func(p Path, v Value) (Value, error) {
    return p.Apply(val)
})

This results in at step 1: key value not number or string when the path is for

cty.Path{cty.GetAttrStep{Name:"set"}, cty.IndexStep{Key:cty.ObjectVal(map[string]cty.Value{"attr":cty.StringVal("val")})}, cty.GetAttrStep{Name:"attr"}}

I'm not sure how we would intend to handle this. Sets are not currently index-able, so simply skipping the key value not number or string check doesn't work. Should we make all Path values valid for Apply on a correctly shaped Value, have special handling to detect when a set is encountered in the path, or maybe just document this as a potential hazard within Transform and Walk?

missing file docs/diff.md

The README.md has a link to diff.md. That file doesn't exist in master.

https://github.com/zclconf/go-cty/blame/master/README.md#L43

Should KeysFunc only return set(string) when given a map?

Current KeysFunc returns either a list(string) or a tuple with the correct number of string values when given an object. Because of the key constraints of maps and objects, both of these can be converted to directly to set(string).

While in most cases cty consumers will automatically convert these as needed, conversion is still necessary. The question was brought about by using map keys as an argument to for_each through modules where authors would like to discourage the use of the object values. See hashicorp/terraform#24409

Converting from Map to Object

I am trying to diagnose hashicorp/terraform#24092 and the root of my issue seems to be that go-cty does not allow conversions from cty.Map to cty.Object.

I noticed that the convert docs suggest that map-to-object conversions are an “unsafe” operation, so should work for some subsets, but I could not find an example of this working.

Is it possible to allow map-to-object conversions like https://play.golang.org/p/mcXBOocphcu (disclaimer: I am not a go programmer)?

convert: Type information lost when converting unknown value to DynamicPseudoType

The intended meaning of a conversion to DynamicPseudoType is to take no conversion action whatsoever and just return the given value verbatim.

This works for known values, but due to a special case for unknown values does not behave as expected for those:

go-cty/cty/convert/conversion.go

Lines 20 to 22 in 88fbe72

	if !in.IsKnown() {
	return cty.UnknownVal(out), nil
	}

This is correct behavior when the output type is known, since the conversion type selection logic already proved that a conversion between the in and out types is valid, but if the output type is DynamicPseudoType this causes the result to be DynamicVal, losing the original type information in the unknown value.

Instead, we should treat DynamicPseudoType as a further special case and just return the given unknown value verbatim if that is selected as the output type.

(Converting to dynamic is a strange thing to do specifically, but this behavior is intended as a convenience for callers that are doing a conversion on behalf of a transitive caller that is specifying a target type, so that such intermediate callers don't need to themselves handle this as a special case.)

How to convert cty.Value of unknown type into `interface{}`

I'm probably just missing something obvious here, and if so apologies for the noise.

I'm attempting to take user input from HCL that is composed of data with types that are unknown until runtime. The example HCL could look something like this:

myblock {
  user_provided_attrs = {
    foo = "foostring"
    bar = true
    buzz = vars.some_complex_var
  }
}

My first attempt was to decode this into cty Values and then transform using gocty.FromCtyValue(val, &decodedVal) where decoded val is map[string]interface{}. Of course, that doesn't work as FromCtyValue uses reflection on the type and maps to specific concrete types. interface{} doesn't match, so I get error: "incorrect type" which is the default error case.

I can see I could put something together using recursion myself, and if all else fails that's what I'll do. The only thing that has stopped me is the assumption that I must be missing a built in way to handle this case. Especially since the package docs have quite a bit of material on converting between unknown types etc.

Converting cty.NilVal to cty.String panics

Hello there ! We had this issue in packer: hashicorp/packer#8730

Basically, the default value of a variable is not set but its type is and when a user tries to access that variable without setting it, a panic occurs.

This was actually tested in the Packer codebase and successfully errored with the following error:

Invalid template interpolation value; The expression result is null. Cannot include a null value in a string template.

But this panics 'in real life' because packer uses
"github.com/zclconf/go-cty/cty/json".Marshal(value, cty.DynamicPseudoType)

I will try to open an PR to fix this, but I might fix something in the wrong place.

Here's a stack trace:

github.com/hashicorp/packer/vendor/github.com/zclconf/go-cty/cty.Type.Equals(0x0, 0x0, 0x84b1000, 0xc0004cd8f8, 0x41ccfd0)
	/Users/azr/go/src/github.com/hashicorp/packer/vendor/github.com/zclconf/go-cty/cty/type.go:42 +0xf5
github.com/hashicorp/packer/vendor/github.com/zclconf/go-cty/cty/convert.Convert(0x0, 0x0, 0x0, 0x0, 0x84b1000, 0xc0004cd8f8, 0x0, 0x0, 0x0, 0xcc74800, ...)
	/Users/azr/go/src/github.com/hashicorp/packer/vendor/github.com/zclconf/go-cty/cty/convert/public.go:43 +0x53
github.com/hashicorp/packer/vendor/github.com/hashicorp/hcl/v2/hcldec.(*AttrSpec).decode(0xc0005c88a0, 0xc0003fcfc0, 0x0, 0x0, 0x0, 0xc00000f7a0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/azr/go/src/github.com/hashicorp/packer/vendor/github.com/hashicorp/hcl/v2/hcldec/spec.go:209 +0x508
github.com/hashicorp/packer/vendor/github.com/hashicorp/hcl/v2/hcldec.ObjectSpec.decode(0xc0001870b0, 0xc0003fcfc0, 0x0, 0x0, 0x0, 0xc00000f7a0, 0xc000618df0, 0x402fd21, 0x7eafba8, 0xc000618df0, ...)
	/Users/azr/go/src/github.com/hashicorp/packer/vendor/github.com/hashicorp/hcl/v2/hcldec/spec.go:80 +0x238
github.com/hashicorp/packer/vendor/github.com/hashicorp/hcl/v2/hcldec.decode(0x84b0240, 0xc0001b46e0, 0x0, 0x0, 0x0, 0xc00000f7a0, 0x84b0f00, 0xc0001870b0, 0x49d8e00, 0x8482000, ...)
	/Users/azr/go/src/github.com/hashicorp/packer/vendor/github.com/hashicorp/hcl/v2/hcldec/decode.go:21 +0x118
github.com/hashicorp/packer/vendor/github.com/hashicorp/hcl/v2/hcldec.Decode(...)
	/Users/azr/go/src/github.com/hashicorp/packer/vendor/github.com/hashicorp/hcl/v2/hcldec/public.go:15
github.com/hashicorp/packer/hcl2template.decodeHCL2Spec(0x84b0240, 0xc0001b46e0, 0xc00000f7a0, 0xce6f290, 0xc0002a6780, 0x0, 0x0, 0x0, 0xc000563800, 0xcc01b28, ...)
	/Users/azr/go/src/github.com/hashicorp/packer/hcl2template/decode.go:17 +0xa0
github.com/hashicorp/packer/hcl2template.(*Parser).startBuilder(0xc000619370, 0xc00020a9f0, 0xc00000f7a0, 0xc000228108, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/Users/azr/go/src/github.com/hashicorp/packer/hcl2template/types.source.go:54 +0x2e4
github.com/hashicorp/packer/hcl2template.(*Parser).getBuilds(0xc000619370, 0xc0000beb00, 0x0, 0x0, 0xc0000beb00, 0x0, 0x0, 0x0)
	/Users/azr/go/src/github.com/hashicorp/packer/hcl2template/types.packer_config.go:102 +0x1d5
github.com/hashicorp/packer/hcl2template.(*Parser).Parse(0xc000619370, 0x7ffeefbff6f1, 0x2e, 0x0, 0xc0002e62a8, 0xc0002e6310, 0x38, 0x40, 0xc0000bea00, 0xc00013bda0)
	/Users/azr/go/src/github.com/hashicorp/packer/hcl2template/types.packer_config.go:150 +0xd1
github.com/hashicorp/packer/command.(*BuildCommand).GetBuildsFromHCL(0xc000186e70, 0x7ffeefbff6f1, 0x2e, 0x0, 0x0, 0xc000563800, 0x1)
	/Users/azr/go/src/github.com/hashicorp/packer/command/build.go:107 +0x123
github.com/hashicorp/packer/command.(*BuildCommand).GetBuilds(0xc000186e70, 0x7ffeefbff6f1, 0x2e, 0x1, 0x1, 0x7fffffffffffffff, 0x0)
	/Users/azr/go/src/github.com/hashicorp/packer/command/build.go:136 +0x543

Edit: I tried introducing Marshaling there and there and it did not trigger the panic.

gocty: Too inflexible about map vs. object distinction

gocty was written quite early on when maps and objects were thought of as very distinct ideas with different operations.

In the meantime, we've blurred that a little by allowing some of the collection operations to also work for object types, since that tends to be more convenient for calling applications that can therefore share code between the two.

Currently the gocty logic is quite picky about the distinction between these, requiring that objects be decoded into structs and maps be decoded into maps. It would be helpful to callers, and more consistent with how the main cty has evolved, to allow decoding objects into Go maps (as long as the attributes all conform to the Go element type) and to allow decoding maps into Go structs, where compatible.

Value.HasElement does not fully handle unknowns

The HasElement method currently handles two situations improperly:

• If the given value is not wholly known then the result must always be cty.UnknownVal(cty.Bool).
• If the given value is wholly known and not in the set but there is at least one partially-unknown value in the set then the result must always be cty.UnknownVal(cty.Bool).

adding jsonencode() block into a generated terraform file using golang cty

Hi
Kind of new at this, so apology in advance if I'm asking a question that was answered already.
I'm trying to generate a tf file using the golang cty library.
It's pretty straightforward. However, I have this section:

  custom_fields = {
    alternate_contact = jsonencode({somecode})
}

How can I add the jsonencode() section into my generated tf file

Provide a way to pretty-print Value

While cty.Value's GoString() is useful for basic debugging when one is already focusing on a simple types/values. More often in Terraform though we need to deal with deeply nested structures, where such one-lined output is not very readable.

It would therefore be useful to have a separate function, which (similarly) prints out values in a human-readable format, but accounts for more complex structures by indenting the output.

Example

This is the current output from GoString():

cty.ObjectVal(map[string]cty.Value{"allow":cty.SetVal([]cty.Value{cty.ObjectVal(map[string]cty.Value{"ports":cty.UnknownVal(cty.List(cty.String)), "protocol":cty.UnknownVal(cty.String)})}), "creation_timestamp":cty.NullVal(cty.String), "deny":cty.SetVal([]cty.Value{cty.ObjectVal(map[string]cty.Value{"ports":cty.UnknownVal(cty.List(cty.String)), "protocol":cty.UnknownVal(cty.String)})}), "description":cty.NullVal(cty.String), "destination_ranges":cty.NullVal(cty.Set(cty.String)), "direction":cty.NullVal(cty.String), "disabled":cty.NullVal(cty.Bool), "enable_logging":cty.NullVal(cty.Bool), "id":cty.NullVal(cty.String), "name":cty.UnknownVal(cty.String), "network":cty.UnknownVal(cty.String), "priority":cty.NullVal(cty.Number), "project":cty.NullVal(cty.String), "self_link":cty.NullVal(cty.String), "source_ranges":cty.NullVal(cty.Set(cty.String)), "source_service_accounts":cty.NullVal(cty.Set(cty.String)), "source_tags":cty.NullVal(cty.Set(cty.String)), "target_service_accounts":cty.NullVal(cty.Set(cty.String)), "target_tags":cty.NullVal(cty.Set(cty.String)), "timeouts":cty.NullVal(cty.Object(map[string]cty.Type{"create":cty.String, "delete":cty.String, "update":cty.String}))})

which would be much more readable in format similar to this:

cty.ObjectVal(map[string]cty.Value{
	"allow": cty.SetVal([]cty.Value{
		cty.ObjectVal(map[string]cty.Value{
			"ports":    cty.UnknownVal(cty.List(cty.String)),
			"protocol": cty.UnknownVal(cty.String),
		}),
	}),
	"creation_timestamp": cty.NullVal(cty.String),
	"deny":cty.SetVal([]cty.Value{
		cty.ObjectVal(map[string]cty.Value{
			"ports":    cty.UnknownVal(cty.List(cty.String)),
			"protocol": cty.UnknownVal(cty.String),
		}),
	}),
	"description":			cty.NullVal(cty.String),
	"destination_ranges":		cty.NullVal(cty.Set(cty.String)),
	"direction":			cty.NullVal(cty.String),
	"disabled":			cty.NullVal(cty.Bool),
	"enable_logging":		cty.NullVal(cty.Bool),
	"id":				cty.NullVal(cty.String),
	"name":				cty.UnknownVal(cty.String),
	"network":			cty.UnknownVal(cty.String),
	"priority":			cty.NullVal(cty.Number),
	"project":			cty.NullVal(cty.String),
	"self_link":			cty.NullVal(cty.String),
	"source_ranges":		cty.NullVal(cty.Set(cty.String)),
	"source_service_accounts":	cty.NullVal(cty.Set(cty.String)),
	"source_tags":			cty.NullVal(cty.Set(cty.String)),
	"target_service_accounts":	cty.NullVal(cty.Set(cty.String)),
	"target_tags":			cty.NullVal(cty.Set(cty.String)),
	"timeouts":			cty.NullVal(cty.Object(map[string]cty.Type{
		"create": cty.String,
		"delete": cty.String,
		"update": cty.String,
	}),
)})

How to convert object to go's map

Hi, I'm trying to parse cty.Object to go's map

if variable.Default.Type().IsObjectType() {
        defaultValue, err = formatDefaultObjectType(variable.Default)
	if err != nil {
		log.Fatal("[terraform][formatDefaultObjectType] ", err)
	}
}

func formatDefaultObjectType(v cty.Value) (string, error) {
	var (
		result string
		err    error
	)

	raw := make(map[string]interface{})

	err = gocty.FromCtyValue(v, &raw)
	if err != nil {
		return "", err
	}

	result = fmt.Sprintf("%+v", raw)

	return result, nil
}

got this error

map or object value is required

I already pass the map to FromCtyValue, and also try to change map key type to interface{} and the error still same.

Consider using "hash/maphash" for set element hashing

The internal implementation of sets of values uses a CRC32 checksum for deciding on a storage key for each value, because at the time of original implementation that was a straightforward mechanism available in the standard library.

However, CRC32 isn't really intended for that purpose and the standard library now includes hash/maphash as a hashing implementation that was designed with this use-case in mind.

Two potential hurdles to consider before adopting it, though:

• The underlying set interface uses int for hash values and therefore varies in size depending on the target platform. In practice because we use CRC32 we only ever populate the lower 32 bits of an int today, regardless of platform. maphash.Bytes produces int64, and so we'll need to decide whether it's acceptable to discard half of the bits it generates when running on a 32-bit platform, and possibly even when we're on a 64-bit platform. Being inconsistent based on target would mean that sets would traverse in a different order based on target, which would likely not be acceptable to some callers such as HashiCorp Terraform.
• Using a new hashing function would change the arbitrary-but-consistent traversal order we use when iterating over elements in sets of non-primitive types. We've technically never guaranteed a particular order, but nonetheless some applications may be unknowingly relying on the order. HashiCorp Terraform in particular effectively exposes the set traversal order to its own users as part of its language, and so Terraform users themselves might be unknowingly relying on the current ordering.

Parse string into cty type

Is there any way to parse a string into a cty type?

Here's the string I have

list(object({
    domain_name         = string
    origin_id           = string
    origin_path         = optional(string)
    connection_attempts = optional(number)
    connection_timeout  = optional(number)
}))

Encoding values with custom function (terraform provider)

Hello,

I'm trying to encode a map into the following format:

resource "grafana_data_source" "prometheus" {

  json_data_encoded = jsonencode({
    httpMethod        = "POST"
    prometheusType    = "Mimir"
    prometheusVersion = "2.4.0"
  })

What I found from the documentation is that cty provides out of the box support to encode Strings, Boolean and other primitives like this:

flt, check := value.(float64)
if check {
   resourceBody.SetAttributeValue(key, cty.NumberFloatVal(flt))
}

There is a way to achieve the same with a map[string]interface{}?

I was looking through the documentation but I could not find anything that resembles what I need to do.

Thanks.

cty.StringVal always doubles $ in ${} output Pt2

Hello again. With all due respect, I would like to open this question one more time. I am doing so because I believe my reasoning wasn't clear last time. Regardless of what HCL expects, CTY will not let me just print a single $. Therefore, I believe I am facing an edge case in CTY itself, not HCL. I tried to search this repo for where that may be happening, but I am not proficient enough to solve it. I looked for $ and none of the references seemed to match what I was looking for.

Hey @apparentlymart,

Thank you for the response! Hmm, my goal was to actually not escape it, as I don't want the resulting string to be interpreted literally. I am using CTY to generate HCL for me so I don't have to write dozens of HCL files by hand. My goal is that the CTY will spit out some HCL that has template interpolation baked into it.

To give the exact example, I'm building a bunch of New Relic dashboards. I want CTY to be able to produce HCL interpolation. Here's what I keep getting stuck with tho:

nrql {
  query = "FROM Metric SELECT clamp_max(sum(newrelic.sli.good) / sum(newrelic.sli.valid) * 100, 100) as 'SLO compliance' WHERE entity.guid = $${newrelic_service_level.asdf---test---order-management_apm.id}"
}

And here is the go code

baseSLOQuery := fmt.Sprintf("FROM Metric SELECT clamp_max(sum(newrelic.sli.good) / sum(newrelic.sli.valid) * 100, 100) as 'SLO compliance' WHERE entity.guid = ${newrelic_service_level.%s.id}", neatSLOName)
nrqlBody.SetAttributeValue("query", cty.StringVal(baseSLOQuery))

Do you have any suggestions?
Thanks again for reading and commenting :)

Originally posted by @tcgbrett in #161 (comment)

TestFormatDate fails with upcoming Go 1.20 release

Hi, when I run go test in the cty/function/stdlib directory using the current sources at HEAD of Go, which will be the future Go 1.20 release, I get

--- FAIL: TestFormatDate (0.00s)
    --- FAIL: TestFormatDate/2-12-02T00:00:00Z_parse_error (0.00s)
        datetime_test.go:225: wrong error
            got:  not a valid RFC3339 timestamp: cannot use "2-12-02T00:00:00Z" as year
            want: not a valid RFC3339 timestamp: cannot use "-02T00:00:00Z" as year
FAIL
FAIL	github.com/zclconf/go-cty/cty/function/stdlib	0.018s
FAIL

The test is expecting an exact match for the error string, which is generally not a good idea. The Go standard library doesn't promise that error strings will match exactly from release to release.

In this case the go-cty source code even says

			// Go parser seems to be trying to parse "2-12" as a year here,
			// producing a confusing error message.

The error message has gotten better, and now the test fails.

function/stdlib: the Modulo function panics if the first argument is an infinity

Need to decide whether this ought to succeed with another infinity or return an error, but either way it shouldn't generate a null pointer panic:

panic: runtime error: invalid memory address or nil pointer dereference
            goroutine 57 [running]:
            runtime/debug.Stack(0xc0002170d0, 0x82fca0, 0xbd29b0)
                ~/.goenv/versions/1.16.0/src/runtime/debug/stack.go:24 +0x9f
            github.com/zclconf/go-cty/cty/function.errorForPanic(...)
                ~/go/1.16.0/pkg/mod/github.com/zclconf/[email protected]/cty/function/error.go:44
            github.com/zclconf/go-cty/cty/function.Function.Call.func1(0xc0002178d8, 0xc0002178f8)
                ~/go/1.16.0/pkg/mod/github.com/zclconf/[email protected]/cty/function/function.go:291 +0x93
            panic(0x82fca0, 0xbd29b0)
                ~/.goenv/versions/1.16.0/src/runtime/panic.go:965 +0x1b9
            github.com/zclconf/go-cty/cty/function/stdlib.glob..func58.1(0xc0002176d0, 0xc0002176f0)
                ~/go/1.16.0/pkg/mod/github.com/zclconf/[email protected]/cty/function/stdlib/number.go:187 +0x110
            panic(0x82fca0, 0xbd29b0)
                ~/.goenv/versions/1.16.0/src/runtime/panic.go:965 +0x1b9
            math/big.(*Int).BitLen(...)
                ~/.goenv/versions/1.16.0/src/math/big/int.go:473
            math/big.(*Float).SetInt(0xc0001ffd40, 0x0, 0xc0001ffd40)
                ~/.goenv/versions/1.16.0/src/math/big/float.go:596 +0x26
            github.com/zclconf/go-cty/cty.Value.Modulo(0x9218e8, 0xc0000161d9, 0x888560, 0xc0001ffc80, 0x9218e8, 0xc0000161d9, 0x888560, 0xc0001ff410, 0x88ac7a, 0x1, ...)
                ~/go/1.16.0/pkg/mod/github.com/zclconf/[email protected]/cty/value_ops.go:694 +0x625
            github.com/zclconf/go-cty/cty/function/stdlib.glob..func58(0xc0001cb240, 0x2, 0x2, 0x9218e8, 0xc0000161d9, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
                ~/go/1.16.0/pkg/mod/github.com/zclconf/[email protected]/cty/function/stdlib/number.go:192 +0x106
            github.com/zclconf/go-cty/cty/function.Function.Call(0xc0000738f0, 0xc0001cb240, 0x2, 0x2, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
            ...

Feature Request: Pretty-print option for JSONEncodeFunc

E.g.,

JSONEncode(someVal)

// Produces e.g.,
// {a:1,b:2,c:3}

indentSize := 4
JSONEncode(someVal, indentSize)

// Produces e.g.,
// {
//     a: 1,
//     b: 2,
//     c: 3
// }

Proposal: JSON serialization of `cty.Path`

Background

I would like to transfer the marked cty.Value over a wire protocol so that the plugin can handle marked as sensitive values in Terraform. Currently, marked values cannot be serialized by either JSON/msgpack, so it just returns an error.
https://github.com/terraform-linters/tflint/blob/v0.45.0/plugin/server.go#L128-L137

cty.Value is marshaled/unmarshaled here:
https://github.com/terraform-linters/tflint-plugin-sdk/blob/v0.15.0/plugin/plugin2host/server.go#L159
https://github.com/terraform-linters/tflint-plugin-sdk/blob/v0.15.0/plugin/plugin2host/client.go#L329

As per Marks Under Serialization, I'm thinking of the following code that serializes marks in a different way:

unmarked, pvm := value.UnmarkDeepWithPaths()

serializedVal, _ := msgpack.Marshal(unmarked, ty)
serializedMarks := make([]SerializedMark, len(pvm))
for idx, v := range pvm {
  serializedMarks[idx].Marks = marshalMarks(v.Marks)
  serializedMarks[idx].Path = marshalPath(v.Path)
}

Response{Value: serializedValue, Marks: serializedMarks}

// plugin side

val, _ := msgpack.Unmarshal(resp.Value, ty)
pvm := make([]cty.PathValueMarks, len(resp.Marks))
for idx, v := range resp.Marks {
  pvm[idx].Marks = unmarshalMarks(v.Marks)
  pvm[idx].Path = unmarshalPath(v.Path)
}

val.MarkWithPaths(pvm)

Looking at the above, marshalMarks and unmarshalMarks is not trivial, but marshalPath and unmarshalPath seem like functions provided by cty package.

Proposal

Add json.MarshalPath just like json.MarshalType. Imagine something like this:

json.MarshalPath(cty.IndexStringPath("boop"))
// => [{"type": "IndexStep", "key": "boop"}]

json.MarshalPAth(cty.GetAttrPath("foo"))
// => [{"type": "GetAttrStep", "name": "foo"}]

What do you think? If this proposal is accepted, I can open a pull request. Thanks!