Topic deleted from state store when ACL on topic is deleted about terraform-provider-kafka

[mchai]

Oh. I think I see what could be going on. It might be related to the broker setting allow.everyone.if.no.acl.found

As soon as we place an ACL on a topic, only superusers and those users granted access via an ACL can see it. So we need to ensure that whichever user is running the terraform job can see the topic after an ACL is applied to it.

[larsbrekken]

Just wanted to note that I was having the same issue with allow.everyone.if.no.acl.found set to true. From what I can tell, Terraform basically "locks itself out" after attaching the ACL to the topic, and causing the provider to get into a weird state (for me this included deleting the topic, and then later not recreating it after incorrectly concluding that it already existed).

After starting by defining an ACL allowing the user I'm running the Terraform script with access to topics I seem to be getting around this problem. Making this user a superuser is likely also a solution, but this option is unfortunately not available to me as I'm using AWS MSK.

This is what worked for me (in this case with an anonymous user, but the same principle should apply with a specific one):

resource "kafka_acl" "terraform-access-topics" {
  resource_name       = "*"
  resource_type       = "Topic"
  acl_principal       = "User:ANONYMOUS"
  acl_operation       = "All"
  acl_permission_type = "Allow"
  acl_host = "*"
}

[larsbrekken]

@Constantin07 I created a terraform user that we use when running terraform, plus an admin user that we can use just in case. This has been working well for us.

resource "kafka_acl" "terraform-topic" {
  resource_name       = "*"
  resource_type       = "Topic"
  acl_principal       = "User:CN=terraform-user"
  acl_operation       = "All"
  acl_permission_type = "Allow"
  acl_host            = "*"
}

# Terraform can perform all group operations
resource "kafka_acl" "terraform-group" {
  resource_name       = "*"
  resource_type       = "Group"
  acl_principal       = "User:CN=terraform-user"
  acl_operation       = "All"
  acl_permission_type = "Allow"
  acl_host            = "*"
}

# The admin user can perform all topic operations
resource "kafka_acl" "admin-topic" {
  resource_name       = "*"
  resource_type       = "Topic"
  acl_principal       = "User:CN=admin-user"
  acl_operation       = "All"
  acl_permission_type = "Allow"
  acl_host            = "*"
}

# The admin user can perform all group operations
resource "kafka_acl" "admin-group" {
  resource_name       = "*"
  resource_type       = "Group"
  acl_principal       = "User:CN=admin-user"
  acl_operation       = "All"
  acl_permission_type = "Allow"
  acl_host            = "*"
}

# The admin user can perform all transactional operations
resource "kafka_acl" "admin-txid" {
  resource_name       = "*"
  resource_type       = "TransactionalID"
  acl_principal       = "User:CN=admin-user"
  acl_operation       = "All"
  acl_permission_type = "Allow"
  acl_host            = "*"
}

[Constantin07]

Thanks a lot @larsbrekken Much appreciated. Do you know if this is required as well ?

resource "kafka_acl" "admin_cluster" {
  resource_name       = "*"
  resource_type       = "Cluster"
  acl_principal       = "User:CN=admin-user"
  acl_operation       = "All"
  acl_permission_type = "Allow"
  acl_host            = "*"
}

When I try to add it to MSK cluster I get:

kafka_acl.admin_cluster: Creating...

Error: kafka server: This most likely occurs because of a request being malformed by the client library or the message was sent to an incompatible broker. See the broker logs for more details.

  on main.tf line 67, in resource "kafka_acl" "admin_cluster":
  67: resource "kafka_acl" "admin_cluster" {

[Mongey]

🤔 I'm having a pretty hard time tracking down why this happens. It seems like once we create the ACL for the topic, kafka no longer responds with the topic in it's list of topics, and as such, it appears to have been deleted.

[Constantin07]

@larsbrekken just wondering have you been able to create a kind of superadmin user in MSK which has access to all topics upfront ? I'm struggling to understand how would I create an admin user in MSK as it's not documented anywhere on AWS side.

[larsbrekken]

@Constantin07 Sorry, I'm not familiar with the Cluster resource type. I searched our scripts and we're not defining that anywhere.

In case you missed it, broker logs are available in MSK now (you can e.g. direct them to an S3 bucket). Perhaps reviewing those would give you enough information to resolve the issue.

[Constantin07]

Thanks @larsbrekken

[azhurbilo]

I'm not familiar with the Cluster resource type

If you don't add Cluster ACL, all other ACLs are useless as any principal could connect and change ACLs via Kafka admin cluster API (as AWS MSK Kafka allow.everyone.if.no.acl.found == true by default)

@Mongey am I right?

[thennati]

Thanks a lot @larsbrekken Much appreciated. Do you know if this is required as well ?

resource "kafka_acl" "admin_cluster" {
  resource_name       = "*"
  resource_type       = "Cluster"
  acl_principal       = "User:CN=admin-user"
  acl_operation       = "All"
  acl_permission_type = "Allow"
  acl_host            = "*"
}

When I try to add it to MSK cluster I get:

kafka_acl.admin_cluster: Creating...

Error: kafka server: This most likely occurs because of a request being malformed by the client library or the message was sent to an incompatible broker. See the broker logs for more details.

  on main.tf line 67, in resource "kafka_acl" "admin_cluster":
  67: resource "kafka_acl" "admin_cluster" {

try resource_name as this then your user will have whole cluster level access = ["kafka-cluster"]

[larsbrekken]

@thennati I'm afraid I won't be able to help you with that. It's been a couple of years since I worked on this specifically, and I don't work with Kafka or Terraform at the moment. Good luck, though!