The line mentioned in the error compute_environment_data_source_test.go:20,37-38 starts at the block update_policy {}

$ find internal/service/batch -type f -name '*_test.go' \
        | sort -u \
        | xargs -I {} terrafmt fmt  --fmtcompat {}
ERRO[2023-11-10 14:37:35] block 2 @ internal/service/batch/compute_environment_data_source_test.go:149 failed to process with: failed to parse hcl: internal/service/batch/compute_environment_data_source_test.go:20,37-38: Invalid expression; Expected the start of an expression, but found an invalid expression token.
resource "aws_batch_compute_environment" "test" {
  compute_environment_name = "@@_@@ TFMT:%[1]q:TFMT @@_@@"

  compute_resources {
    allocation_strategy = "BEST_FIT_PROGRESSIVE"
    instance_role       = aws_iam_instance_profile.ecs_instance.arn
    instance_type       = ["optimal"]
    max_vcpus           = 4
    min_vcpus           = 0
    security_group_ids  = [
      aws_security_group.test.id
    ]
    subnets = [
      aws_subnet.test.id
    ]
    type = "EC2"
  }
  update_policy {
    job_execution_timeout_minutes = "@@_@@ TFMT:%[2]d:TFMT @@_@@"
    terminate_jobs_on_update      = %[3]v
  }

  type = "MANAGED"
}

data "aws_batch_compute_environment" "by_name" {
  compute_environment_name = aws_batch_compute_environment.test.compute_environment_name
}

Test code:

func testAccComputeEnvironmentDataSourceConfig_updatePolicy(rName string, timeout int, terminate bool) string {
	return acctest.ConfigCompose(testAccComputeEnvironmentConfig_baseDefaultSLR(rName), fmt.Sprintf(`
resource "aws_batch_compute_environment" "test" {
  compute_environment_name = %[1]q

  compute_resources {
    allocation_strategy = "BEST_FIT_PROGRESSIVE"
    instance_role       = aws_iam_instance_profile.ecs_instance.arn
    instance_type       = ["optimal"]
    max_vcpus           = 4
    min_vcpus           = 0
    security_group_ids  = [
      aws_security_group.test.id
    ]
    subnets = [
      aws_subnet.test.id
    ]
    type = "EC2"
  }
  update_policy {
    job_execution_timeout_minutes = %[2]d
    terminate_jobs_on_update      = %[3]v
  }

  type = "MANAGED"
}

data "aws_batch_compute_environment" "by_name" {
  compute_environment_name = aws_batch_compute_environment.test.compute_environment_name
}
`, rName, timeout, terminate))
}

terrafmt does not extract terraform files and does not do inline code formatting from the README.md file if, we start the Three backquotes after 4 spaces in the README.md file. It only works, if we start the Three backquotes at column 0.

Ex:

Works:

terrafmt blocks README.md

####### B1 @ #587
resource "aws_lambda_function" "pass" {
function_name = "test-env"
role = ""
runtime = "python3.8"

environment {
variables = {
AWS_DEFAULT_REGION = "us-west-2"
}
}
}
Does not work

terrafmt blocks README.md

note: markdown format allows having an offset.

In Go source files, a heuristic is used to identify if a block looks like Terraform before running tooling against it. It currently requires at least one line matching a resource or data source block or variable or output block.

In most cases this works, but if a reusable Terraform segment allows all names to be replaced, it is not detected. For example, the following is not detected:

func testAccAvailableAZsNoOptInConfigWithProvider(name, provider string) string {
	return fmt.Sprintf(`
data "aws_availability_zones" "%[1]s" {
  provider = %[2]s

  state = "available"

  filter {
    name   = "opt-in-status"
    values = ["opt-in-not-required"]
  }
}
`, name, provider)
}

When format verbs are enabled, the pattern matching should also match resource names contains format verbs.

Terraform 0.13 has changed how it handles the final newline in 0.12upgrade, causing test failures.

To help using terrafmt blocks in toolchains, add the option to use a null separator. This would allow e.g. piping to xargs -0 ...

There doesn't seem to be consistency between existing tools:

• find uses -print0
• GNU grep uses -Z, --null
• sort uses -z, --zero-terminated

I propose using --zero-terminated for the long option and -z for the short option, or possibly -0.

My configuration (hashicorp/terraform-provider-aws#14013) is

cidr_block      = (%[2]q == "cidr_block") ? %[3]q : null
ipv6_cidr_block = (%[2]q == "ipv6_cidr_block") ? %[3]q : null

and terrafmt reports an error

failed to process with: failed to parse hcl: ./aws/resource_aws_route_table_test.go:69,48-49: Unbalanced parentheses; Expected a closing parenthesis to terminate the expression.
...
cidr_block      = (TFFMTKTBRACKETPERCENT[2]q == "cidr_block") ? %[3]q : null
ipv6_cidr_block = (TFFMTKTBRACKETPERCENT[2]q == "ipv6_cidr_block") ? %[3]q : null

This error seems to be in addition to the issue reported in #37.

When a configuration contains a resource where the name is a format verb, the format verb is not replaced, causing HCL parsing to fail.

For example, in https://github.com/hashicorp/terraform-provider-aws/blob/38319aa13aea65d3e946450c1f881d8fef12e734/aws/resource_aws_quicksight_user_test.go#L195-L203

data "aws_caller_identity" "current" {}

resource "aws_quicksight_user" %[1]q {
  aws_account_id = data.aws_caller_identity.current.account_id
  user_name      = %[1]q
  email          = %[2]q
  identity_type  = "QUICKSIGHT"
  user_role      = "READER"
}

Expected Result

The line resource "aws_quicksight_user" %[1]q { should be replaced with something parseable as HCL

Actual Result

The line resource "aws_quicksight_user" %[1]q { is not changed, causing HCL parsing to fail

fix:

(.gitignore is only a bandaid solution as the files are still there)

Hi,

it would be nice to also be able to format terraform configuration samples included in the provider markdown documentation enclosed in ```terraform / ``` blocks, e.g.:

terrafmt fmt website/docs/r/myresource.html.markdown

Terraform for expressions cause terrafmt fmt -f (and therefore probably terrafmt diff and terrafmt upgrade012) to fail.

For example, the following snippet

resource "aws_elasticache_replication_group" "test" {
  replication_group_id          = %[1]q
  replication_group_description = "test description"
  node_type                     = "cache.t2.micro"
  subnet_group_name             = aws_elasticache_subnet_group.test.name
  security_group_ids            = [aws_security_group.test.id]

  node_groups {
	node_group_id              = %[3]q
	primary_availability_zone  = aws_subnet.test[0].availability_zone
	replica_availability_zones = [for x in range(1, %[2]d+1) : element(aws_subnet.test[*].availability_zone, x)]
	replica_count              = %[2]d
  }
}

causes the following error

ERRO[2021-02-01 14:43:47] block 25 @ aws/resource_aws_elasticache_replication_group_test.go:2399 failed to process with: failed to parse hcl: aws/resource_aws_elasticache_replication_group_test.go:11,50-51: Invalid expression; Expected the start of an expression, but found an invalid expression token.
resource "aws_elasticache_replication_group" "test" {
  replication_group_id          = "@@_@@ TFMT:%[1]q:TFMT @@_@@"
  replication_group_description = "test description"
  node_type                     = "cache.t2.micro"
  subnet_group_name             = aws_elasticache_subnet_group.test.name
  security_group_ids            = [aws_security_group.test.id]

  node_groups {
	node_group_id              = "@@_@@ TFMT:%[3]q:TFMT @@_@@"
	primary_availability_zone  = aws_subnet.test[0].availability_zone
	replica_availability_zones = [for x in range(1, %[2]d+1) : element(aws_subnet.test[*].availability_zone, x)]
	replica_count              = "@@_@@ TFMT:%[2]d:TFMT @@_@@"
  }
} 

Description

When using terrafmt diff -f with the following configuration snippet:

	return fmt.Sprintf(`
data "aws_caller_identity" "current" {}

resource "aws_signer_signing_profile" "test" {
  platform_id = "AWSLambda-SHA384-ECDSA"
  name        = replace(%[1]q, "-", "_")
}
# ...
`, rName)
}

An error is returned because the string is not quoted properly during format verb handling (note: replace(TFFMTKTBRACKETPERCENT[1]q, \"-\", \"_\")):

time="2020-11-24 18:23:42" level=error msg="block 1 @ ./aws/data_source_aws_signer_signing_job_test.go:34 failed to process with: failed to parse hcl: ./aws/data_source_aws_signer_signing_job_test.go:5,49-50: Missing argument separator; A comma is required to separate each function argument from the next.\ndata \"aws_caller_identity\" \"current\" {}\n\nresource \"aws_signer_signing_profile\" \"test\" {\n  platform_id = \"AWSLambda-SHA384-ECDSA\"\n  name        = replace(TFFMTKTBRACKETPERCENT[1]q, \"-\", \"_\")\n}\n\nresource \"aws_s3_bucket\" \"source\" {\n  bucket = \"%[1]s-source\"\n\n  versioning {\n    enabled = true\n  }\n\n  force_destroy = true\n}\n\nresource \"aws_s3_bucket\" \"destination\" {\n  bucket        = \"%[1]s-destination\"\n  force_destroy = true\n}\n\nresource \"aws_s3_bucket_object\" \"source\" {\n  bucket = aws_s3_bucket.source.bucket\n  key    = \"lambdatest.zip\"\n  source = \"test-fixtures/lambdatest.zip\"\n}\n\nresource \"aws_signer_signing_job\" \"test\" {\n  profile_name = aws_signer_signing_profile.test.name\n\n  source {\n    s3 {\n      bucket  = aws_s3_bucket_object.source.bucket\n      key     = aws_s3_bucket_object.source.key\n      version = aws_s3_bucket_object.source.version_id\n    }\n  }\n\n  destination {\n    s3 {\n      bucket = aws_s3_bucket.destination.bucket\n    }\n  }\n}\n\ndata \"aws_signer_signing_job\" \"test\" {\n  job_id = aws_signer_signing_job.test.job_id\n}\n"

Terraform 1.8 will introduce support for provider defined functions. hashicorp/hcl/v2 has been updated to support this syntax, so an update to this dependency is required in order to support providers implementing functions.

Here's an example of the HCL parsing failure in the AWS provider:

ERRO[2024-03-06 10:49:01] block 1 @ website/docs/functions/arn_build.html.markdown:19 failed to process with: failed to parse hcl: website/docs/functions/arn_build.html.markdown:3,1
9-20: Missing newline after argument; An argument definition must end with a newline.
# result: arn:aws:iam::444455556666:role/example
output "example" {
  value = provider::aws::arn_build("aws", "iam", "", "444455556666", "role/example")
}

Terminals such as iTerm2 will automatically create clickable links for patterns matching file paths and web URLs (to open, hover over while holding command key and click). The current output from terrafmt uses # as the delimiter between file names and line numbers, e.g.

aws/resource_aws_workspaces_directory_test.go#324
aws/resource_aws_workspaces_ip_group_test.go#127
aws/resource_aws_workspaces_ip_group_test.go#149
ERRO[0000] block 2 @ aws/resource_aws_xray_sampling_rule_test.go#194 failed to process with: failed to parse hcl: aws/resource_aws_xray_sampling_rule_test.go:3,20-21: Invalid expression; Expected the start of an expression, but found an invalid expression token.

When clicking those links they open in a web browser since presumably iTerm2 is treating them as a URL due to the # URL separator. Updating output to instead use : as a delimiter should allow terminals to treat these as file paths instead, so they open in your configured editor.

When Go format verbs are present in Terraform blocks, the blocks command outputs them as-is. When the blocks are sent to other Terraform tools, this causes parse errors.

Add the option to mask any format verbs in the blocks.

Use the existing fmtcompat/f flags

Needed for hashicorp/terraform-provider-aws#14722

When handling Terraform blocks in Go files, when a line contains more than one Go format verb, the --fmtcompat flag does not replace any of the flags.

Example 1

provider "aws" {
  ignore_tags {
    keys = [%[1]q, %[2]q]
  }

  skip_credentials_validation = true
  skip_get_ec2_platforms      = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true
}

Expected

ignore_tags {
  keys = ["@@_@@ TFMT:%[1]q:TFMT @@_@@", "@@_@@ TFMT:%[2]q:TFMT @@_@@"]
}

Got

ignore_tags {
  keys = [%[1]q, %[2]q]
}

Example 2

resource "aws_accessanalyzer_analyzer" "test" {
  analyzer_name = %[1]q

  tags = {
    %[2]q = %[3]q
  }
}

Expected

tags = {
  "@@_@@ TFMT:%[2]q:TFMT @@_@@" = "@@_@@ TFMT:%[3]q:TFMT @@_@@"
}

Got

tags = {
  %[2]q = "@@_@@ TFMT:%[3]q:TFMT @@_@@"
}

Needed for hashicorp/terraform-provider-aws#14722

When a configuration with format verbs has two quoted values on the same line, only the first is replaced.

For example, in https://github.com/hashicorp/terraform-provider-aws/blob/38319aa13aea65d3e946450c1f881d8fef12e734/aws/resource_aws_ecs_capacity_provider_test.go#L438-L448

resource "aws_ecs_capacity_provider" "test" {
  name = %[1]q

  tags = {
    %[2]q = %[3]q,
  }

  auto_scaling_group_provider {
    auto_scaling_group_arn = aws_autoscaling_group.test.arn
  }
}

Expected Result

the line%[2]q = %[3]q, should be replaced with "@@_@@ TFMT:%[2]q:TFMT @@_@@" = "@@_@@ TFMT:%[3]q:TFMT @@_@@", or something similar

Actual Result

the line%[2]q = %[3]q, is replaced with "@@_@@ TFMT:%[2]q:TFMT @@_@@" = %[3]q,, and terrafmt fmt fails.

When using terrafmt in a tooling pipelines, it can be useful to have actionable error codes to distinguish between different types of errors.

Motivating example

When calling

terrafmt diff ./aws --check --pattern '*_test.go' --quiet --fmtcompat

There are some edge cases with --fmtcompat, i.e. #30 and other cases, where parsing the Terraform configuration will fail. It would be useful to distinguish the case of "this block is not formatted" vs "this block could not be evaluated" (and vs "some other error occurred).

Proposed error codes

Error cases that can occur together should allow bitwise operations, e.g. an enclosing file could have both unformatted blocks and blocks that cannot be evaluated.

Other errors should return the code 1.

Are there other error cases that should have a specific code?

When a configuration contains a format verb as a parameter name, it is replaced with a quoted string, which fails HCL parsing.

For example, in https://github.com/hashicorp/terraform-provider-aws/blob/38319aa13aea65d3e946450c1f881d8fef12e734/aws/resource_aws_efs_file_system_test.go#L832-L836

resource "aws_efs_file_system" "test" {
  lifecycle_policy {
    %s = %q
  }
}

Expected Result

The line %s = %q should replace the leading bare string with something compatible with HCL parsing

Actual Result

The line %s = %q is replaced with "@@_@@ TFMT:%s:TFMT @@_@@" = "@@_@@ TFMT:%q:TFMT @@_@@", and terrafmt fmt fails

It seems like terrafmt prefers long one-liners over readable lists. It's after my own heart but perhaps less readable.

terrafmt is not happy with this but maybe should be:

resource "aws_docdb_cluster" "default" {
  availability_zones = [
    data.aws_availability_zones.available.names[0],
    data.aws_availability_zones.available.names[1],
    data.aws_availability_zones.available.names[2]
  ]
}

terrafmt is happy with this but maybe should not be:

resource "aws_docdb_cluster" "default" {
  availability_zones = [data.aws_availability_zones.available.names[0], data.aws_availability_zones.available.names[1], data.aws_availability_zones.available.names[2]]
}

Following test case will fail in fmtverbs_test.go:

		{
			name: "assigned-positional-not-standalone",
			block: `
resource  "resource"    "test" {
kat = %[1]s.id
byte = %[1]d.id
} 
`,
			expected: `
resource "resource" "test" {
	kat = %[1]s.id
	byte = %[1]d.id
}
`,
		},

Error message:

--- FAIL: TestFmtVerbBlock (0.00s)
    --- FAIL: TestFmtVerbBlock/assigned-positional-not-standalone (0.00s)
        fmtverbs_test.go:202: Got an error when none was expected: failed to parse hcl: test:3,7-8: Invalid expression; Expected the start of an expression, but found an invalid expression token.
FAIL
exit status 1
FAIL    github.com/katbyte/terrafmt/lib/format  0.006s

Currently, terrafmt upgrade012 outputs errors if a block has already been updated to Terraform 0.12 syntax. This is because of limitations in terraform 0.12upgrade, as explained in hashicorp/terraform#23512.

Calling terraform validate -json and looking for a non-zero value in the warning_count field will indicate validation warnings, which can be assumed to be (mostly) pre-0.12 syntax.

Hi @katbyte,

I'm getting several failed to find end of block errors on https://github.com/claranet/terraform-provider-zabbix/ (master branch)

# for f in zabbix/*_test.go ; do ~/go/bin/terrafmt fmt $f ; done
ERRO[0000] block 1 @ zabbix/resource_zabbix_host_group_test.go#56 failed to find end of block
ERRO[0000] block 1 @ zabbix/resource_zabbix_host_test.go#63 failed to find end of block
ERRO[0000] block 1 @ zabbix/resource_zabbix_item_test.go#53 failed to find end of block
ERRO[0000] block 2 @ zabbix/resource_zabbix_item_test.go#53 failed to find end of block
ERRO[0000] block 1 @ zabbix/resource_zabbix_template_link_test.go#141 failed to find end of block
ERRO[0000] block 2 @ zabbix/resource_zabbix_template_link_test.go#141 failed to find end of block
ERRO[0000] block 3 @ zabbix/resource_zabbix_template_link_test.go#141 failed to find end of block
ERRO[0000] block 1 @ zabbix/resource_zabbix_template_test.go#151 failed to find end of block
ERRO[0000] block 2 @ zabbix/resource_zabbix_template_test.go#151 failed to find end of block
ERRO[0000] block 3 @ zabbix/resource_zabbix_template_test.go#151 failed to find end of block
ERRO[0000] block 4 @ zabbix/resource_zabbix_template_test.go#151 failed to find end of block
ERRO[0000] block 5 @ zabbix/resource_zabbix_template_test.go#151 failed to find end of block
ERRO[0000] block 6 @ zabbix/resource_zabbix_template_test.go#151 failed to find end of block
ERRO[0000] block 7 @ zabbix/resource_zabbix_template_test.go#151 failed to find end of block
ERRO[0000] block 8 @ zabbix/resource_zabbix_template_test.go#151 failed to find end of block

Example block triggering the error:

func testAccZabbixHostGroupConfig(groupName string) string {
	return fmt.Sprintf(`
		resource "zabbix_host_group" "zabbix" {
			name = "%s"
		}`, groupName,
	)
}

Note: this happens with the freshly released v0.1.0 compiled with go 1.13.8 and 1.14.