[ISSUE] Issue with `databricks_aws_crossaccount_policy` data source after 1.39 update about terraform-provider-databricks

[ruloweb]

The databricks_aws_crossaccount_policy is an account level data source, not a workspace one, that was the bug introduced in 1.39.0, the PR #3343 changed the data source from common.Resource to common.WorkspaceData, then the fix in this thread updated it to common.AccountData and added the corresponding tests.

The only way it will start working again is with a patch release 1.39.1.

Another way is replacing the databricks_aws_crossaccount_policy data source with the policy you need.

Instead of:

data "databricks_aws_crossaccount_policy" "this" {}

resource "aws_iam_role_policy" "this" {
  name   = "cross-account-policy"
  role   = aws_iam_role.cross_account_role.id
  policy = data.databricks_aws_crossaccount_policy.this.json
}

You can do:
IMPORTANT: please read the corresponding documentation so your policy is actually the one you need.

resource "aws_iam_role_policy" "this" {
  name   = "cross-account-policy"
  role   = aws_iam_role.cross_account_role.id
  // TODO: temporal fix until a new version is released, see https://github.com/databricks/terraform-provider-databricks/issues/3421
  policy = jsonencode({
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": [
          "ec2:AllocateAddress",
          "ec2:AssignPrivateIpAddresses",
          "ec2:AssociateDhcpOptions",
          "ec2:AssociateIamInstanceProfile",
          "ec2:AssociateRouteTable",
          "ec2:AttachInternetGateway",
          "ec2:AttachVolume",
          "ec2:AuthorizeSecurityGroupEgress",
          "ec2:AuthorizeSecurityGroupIngress",
          "ec2:CancelSpotInstanceRequests",
          "ec2:CreateDhcpOptions",
          "ec2:CreateFleet",
          "ec2:CreateInternetGateway",
          "ec2:CreateKeyPair",
          "ec2:CreateLaunchTemplate",
          "ec2:CreateLaunchTemplateVersion",
          "ec2:CreateNatGateway",
          "ec2:CreatePlacementGroup",
          "ec2:CreateRoute",
          "ec2:CreateRouteTable",
          "ec2:CreateSecurityGroup",
          "ec2:CreateSubnet",
          "ec2:CreateTags",
          "ec2:CreateVolume",
          "ec2:CreateVpc",
          "ec2:CreateVpcEndpoint",
          "ec2:DeleteDhcpOptions",
          "ec2:DeleteFleets",
          "ec2:DeleteInternetGateway",
          "ec2:DeleteKeyPair",
          "ec2:DeleteLaunchTemplate",
          "ec2:DeleteLaunchTemplateVersions",
          "ec2:DeleteNatGateway",
          "ec2:DeletePlacementGroup",
          "ec2:DeleteRoute",
          "ec2:DeleteRouteTable",
          "ec2:DeleteSecurityGroup",
          "ec2:DeleteSubnet",
          "ec2:DeleteTags",
          "ec2:DeleteVolume",
          "ec2:DeleteVpc",
          "ec2:DeleteVpcEndpoints",
          "ec2:DescribeAvailabilityZones",
          "ec2:DescribeFleetHistory",
          "ec2:DescribeFleetInstances",
          "ec2:DescribeFleets",
          "ec2:DescribeIamInstanceProfileAssociations",
          "ec2:DescribeInstanceStatus",
          "ec2:DescribeInstances",
          "ec2:DescribeInternetGateways",
          "ec2:DescribeLaunchTemplates",
          "ec2:DescribeLaunchTemplateVersions",
          "ec2:DescribeNatGateways",
          "ec2:DescribeNetworkAcls",
          "ec2:DescribePlacementGroups",
          "ec2:DescribePrefixLists",
          "ec2:DescribeReservedInstancesOfferings",
          "ec2:DescribeRouteTables",
          "ec2:DescribeSecurityGroups",
          "ec2:DescribeSpotInstanceRequests",
          "ec2:DescribeSpotPriceHistory",
          "ec2:DescribeSubnets",
          "ec2:DescribeVolumes",
          "ec2:DescribeVpcAttribute",
          "ec2:DescribeVpcs",
          "ec2:DetachInternetGateway",
          "ec2:DetachVolume",
          "ec2:DisassociateIamInstanceProfile",
          "ec2:DisassociateRouteTable",
          "ec2:GetLaunchTemplateData",
          "ec2:GetSpotPlacementScores",
          "ec2:ModifyFleet",
          "ec2:ModifyLaunchTemplate",
          "ec2:ModifyVpcAttribute",
          "ec2:ReleaseAddress",
          "ec2:ReplaceIamInstanceProfileAssociation",
          "ec2:RequestSpotInstances",
          "ec2:RevokeSecurityGroupEgress",
          "ec2:RevokeSecurityGroupIngress",
          "ec2:RunInstances",
          "ec2:TerminateInstances"
        ],
        "Resource": "*"
      },
      {
        "Effect": "Allow",
        "Action": [
          "iam:CreateServiceLinkedRole",
          "iam:PutRolePolicy"
        ],
        "Resource": "arn:aws:iam::*:role/aws-service-role/spot.amazonaws.com/AWSServiceRoleForEC2Spot",
        "Condition": {
          "StringLike": {
            "iam:AWSServiceName": "spot.amazonaws.com"
          }
        }
      }
    ]
  })
}

[alexott]

Hmmm, if you use a provider with an alias, you need to specify it in the data source as well... Or it's used in the module?

[alexott]

Please provide logs as described in the troubleshooting guide.

[alexott]

Reproduced it myself

[mikalai-t]

I'm facing the same. Enabled Terraform debug logs, but didn't find anything to help explain the error:

2024-04-02T13:14:58.333Z [ERROR] provider.terraform-provider-databricks_v1.39.0: Response contains error diagnostic: @caller=/home/runner/work/terraform-provider-databricks/terraform-provider-databricks/vendor/github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/diag/diagnostics.go:58 @module=sdk.proto diagnostic_detail= diagnostic_severity=ERROR tf_data_source_type=databricks_aws_crossaccount_policy diagnostic_summary="cannot read aws crossaccount policy: cannot get client aws crossaccount policy: invalid Databricks Workspace configuration" tf_proto_version=5.4 tf_provider_addr=registry.terraform.io/databricks/databricks tf_req_id=229f7398-6061-0183-7091-0d1409d2a25b tf_rpc=ReadDataSource timestamp=2024-04-02T13:14:58.236Z
2024-04-02T13:14:58.333Z [ERROR] vertex "module.databricks.data.databricks_aws_crossaccount_policy.this[0]" error: cannot read aws crossaccount policy: cannot get client aws crossaccount policy: invalid Databricks Workspace configuration
2024-04-02T13:14:58.334Z [ERROR] vertex "module.databricks.data.databricks_aws_crossaccount_policy.this (expand)" error: cannot read aws crossaccount policy: cannot get client aws crossaccount policy: invalid Databricks Workspace configuration

Interesting that until v1.39.0 it didn't support additional arguments, so it's expected that everyone has an empty data source definition. But I wonder if some of them are required now...

[plamb]

We're seeing the same issue. We've reverted to 1.38.0.

[orolega]

Hmmm, if you use a provider with an alias, you need to specify it in the data source as well... Or it's used in the module?

Sorry didnt provide context of the configuration. The data source is used within the module. I see you reproduced it and have a fix though. Nice work!

[jsancheztejeda]

Hi, so what do we need to do to fix this error? wait for the 1.40.0 release?

[mikalai-t]

You could change the provider configuration from account to workspace for that particular data-source, or you could update provider version constraint to stick with 1.38.0 for the time being

[JanuaryThomas]

I'm having the same issue, with 1.39.0, I had to revert back to older version