Zookeeper Connection Plaintext (Public Amazon MSK) not available? about terraform-provider-kafka

[arinhouck]

From what I can gather, zookeeper is the only option with Amazon MSK for managing ACLs. So it doesn't seem this is supporting Amazon MSK at all if I am not mistaken. Has anyone got this working with MSK?

[hugolesta]

@arinhouck, I was able to create ACL over MKS using plaintext

According to your code I can suggest turning on skip_tls_verify attribute to true, and trying again.

I'd suggest setting up the provider in the following way.

provider "kafka" {
  bootstrap_servers = split(",", var.servers)
  tls_enabled       = false
  skip_tls_verify   = true
}

[qq304635576]

@hugolesta if create ACL over MSK using SSL, It's failed. so do you have solution for that? As we know, considering security requirement, in common, using SSL is required in Production Environment.thanks

[qq304635576]

@arinhouck have any update? I also encounter the same issue with you.

[arinhouck]

@arinhouck, I was able to create ACL over MKS using plaintext

According to your code I can suggest turning on skip_tls_verify attribute to true, and trying again.

I'd suggest setting up the provider in the following way.

provider "kafka" {
  bootstrap_servers = split(",", var.servers)
  tls_enabled       = false
  skip_tls_verify   = true
}

Plaintext on which server urls? I'd assume bootstrap ones. You sure you are using zookeeper? As from what I understand the library maps to --bootstrap-server ... it doesn't use --zookeeper.connect=.... Is your cluster public as well?

@qq304635576 I ditched SCRAM and zookeeper. I ended up using IAM Auth which allows you to bypass zookeeper. I used https://github.com/devshawn/kafka-gitops using the following script from this comment to setup IAM auth.

devshawn/kafka-gitops#82 (comment)

[qq304635576]

@arinhouck Actually, Looks like no need to care about zookeeper, I can modify ACL by a client app named "offset explorer 2" without configuring Zookeeper over MSK using SASL_SSL manually. IAM Auth is a new feature, which is owned by AWS MSK only. considering MSK as bus info channel, It should be the most widely compatible with apps for auth. that's why I chose SASL_SSL. moreover, will check your recommendation, maybe I will change to IAM auth in future. thanks.

[qq304635576]

@arinhouck
Good news! I tested again via SASL_SSL & Port:9096. It woks.
(1) Set up as below:

provider "msk" {
bootstrap_servers = var.msk_kafka_brokers
tls_enabled = true
skip_tls_verify = true
sasl_username = local.raw_data.username
sasl_password = local.raw_data.password
sasl_mechanism = "scram-sha512"
}

(2)Terraform output:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following
symbols:

• create

Terraform will perform the following actions:

##kafka_acl.brokertopic will be created

• resource "kafka_acl" "brokertopic" {
  • acl_host = "*"
  • acl_operation = "All"
  • acl_permission_type = "Allow"
  • acl_principal = "User:broker"
  • id = (known after apply)
  • resource_name = "TEST_"
  • resource_pattern_type_filter = "Prefixed"
  • resource_type = "Topic"
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.

Enter a value: yes

kafka_acl.brokertopic: Creating...
kafka_acl.brokertopic: Creation complete after 2s [id=User:broker|*|All|Allow|Topic|TEST_|Prefixed]
Releasing state lock. This may take a few moments...

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.