Comments (11)
Oh. I think I see what could be going on. It might be related to the broker setting allow.everyone.if.no.acl.found
As soon as we place an ACL on a topic, only superusers and those users granted access via an ACL can see it. So we need to ensure that whichever user is running the terraform job can see the topic after an ACL is applied to it.
from terraform-provider-kafka.
Just wanted to note that I was having the same issue with allow.everyone.if.no.acl.found
set to true
. From what I can tell, Terraform basically "locks itself out" after attaching the ACL to the topic, and causing the provider to get into a weird state (for me this included deleting the topic, and then later not recreating it after incorrectly concluding that it already existed).
After starting by defining an ACL allowing the user I'm running the Terraform script with access to topics I seem to be getting around this problem. Making this user a superuser is likely also a solution, but this option is unfortunately not available to me as I'm using AWS MSK.
This is what worked for me (in this case with an anonymous user, but the same principle should apply with a specific one):
resource "kafka_acl" "terraform-access-topics" {
resource_name = "*"
resource_type = "Topic"
acl_principal = "User:ANONYMOUS"
acl_operation = "All"
acl_permission_type = "Allow"
acl_host = "*"
}
from terraform-provider-kafka.
@Constantin07 I created a terraform user that we use when running terraform, plus an admin user that we can use just in case. This has been working well for us.
resource "kafka_acl" "terraform-topic" {
resource_name = "*"
resource_type = "Topic"
acl_principal = "User:CN=terraform-user"
acl_operation = "All"
acl_permission_type = "Allow"
acl_host = "*"
}
# Terraform can perform all group operations
resource "kafka_acl" "terraform-group" {
resource_name = "*"
resource_type = "Group"
acl_principal = "User:CN=terraform-user"
acl_operation = "All"
acl_permission_type = "Allow"
acl_host = "*"
}
# The admin user can perform all topic operations
resource "kafka_acl" "admin-topic" {
resource_name = "*"
resource_type = "Topic"
acl_principal = "User:CN=admin-user"
acl_operation = "All"
acl_permission_type = "Allow"
acl_host = "*"
}
# The admin user can perform all group operations
resource "kafka_acl" "admin-group" {
resource_name = "*"
resource_type = "Group"
acl_principal = "User:CN=admin-user"
acl_operation = "All"
acl_permission_type = "Allow"
acl_host = "*"
}
# The admin user can perform all transactional operations
resource "kafka_acl" "admin-txid" {
resource_name = "*"
resource_type = "TransactionalID"
acl_principal = "User:CN=admin-user"
acl_operation = "All"
acl_permission_type = "Allow"
acl_host = "*"
}
from terraform-provider-kafka.
Thanks a lot @larsbrekken Much appreciated. Do you know if this is required as well ?
resource "kafka_acl" "admin_cluster" {
resource_name = "*"
resource_type = "Cluster"
acl_principal = "User:CN=admin-user"
acl_operation = "All"
acl_permission_type = "Allow"
acl_host = "*"
}
When I try to add it to MSK cluster I get:
kafka_acl.admin_cluster: Creating...
Error: kafka server: This most likely occurs because of a request being malformed by the client library or the message was sent to an incompatible broker. See the broker logs for more details.
on main.tf line 67, in resource "kafka_acl" "admin_cluster":
67: resource "kafka_acl" "admin_cluster" {
from terraform-provider-kafka.
🤔 I'm having a pretty hard time tracking down why this happens. It seems like once we create the ACL for the topic, kafka no longer responds with the topic in it's list of topics, and as such, it appears to have been deleted.
from terraform-provider-kafka.
@larsbrekken just wondering have you been able to create a kind of superadmin
user in MSK which has access to all topics upfront ? I'm struggling to understand how would I create an admin user in MSK as it's not documented anywhere on AWS side.
from terraform-provider-kafka.
@Constantin07 Sorry, I'm not familiar with the Cluster
resource type. I searched our scripts and we're not defining that anywhere.
In case you missed it, broker logs are available in MSK now (you can e.g. direct them to an S3 bucket). Perhaps reviewing those would give you enough information to resolve the issue.
from terraform-provider-kafka.
Thanks @larsbrekken
from terraform-provider-kafka.
I'm not familiar with the Cluster resource type
If you don't add Cluster ACL, all other ACLs are useless as any principal could connect and change ACLs via Kafka admin cluster API (as AWS MSK Kafka allow.everyone.if.no.acl.found == true by default)
@Mongey am I right?
from terraform-provider-kafka.
Thanks a lot @larsbrekken Much appreciated. Do you know if this is required as well ?
resource "kafka_acl" "admin_cluster" { resource_name = "*" resource_type = "Cluster" acl_principal = "User:CN=admin-user" acl_operation = "All" acl_permission_type = "Allow" acl_host = "*" }When I try to add it to MSK cluster I get:
kafka_acl.admin_cluster: Creating... Error: kafka server: This most likely occurs because of a request being malformed by the client library or the message was sent to an incompatible broker. See the broker logs for more details. on main.tf line 67, in resource "kafka_acl" "admin_cluster": 67: resource "kafka_acl" "admin_cluster" {
try resource_name as this then your user will have whole cluster level access = ["kafka-cluster"]
from terraform-provider-kafka.
@thennati I'm afraid I won't be able to help you with that. It's been a couple of years since I worked on this specifically, and I don't work with Kafka or Terraform at the moment. Good luck, though!
from terraform-provider-kafka.
Related Issues (20)
- Enable the GSSAPI SASL mechanism for kerberos-based authentication
- Add support for oauthbearer as the sasl mechanism HOT 6
- Unable to use aws-iam with Apache Kafka version 2.5.1 HOT 1
- Provider not working since version 0.6 HOT 22
- Terraform Plan takes a long time waiting for topics (intermittent)
- Ho do I use a PKCS12 keystore for MSK authentication?
- Version 0.7.0 does error on aim auth in comparison to 0.6.0 HOT 6
- Version 0.7.0 and 0.6.0 not working for kafka TLS brokers to create topics
- Version 0.7.0 and 0.6.0 not working for kafka TLS brokers to create topics
- Using this provider with multiple AWS Kafka clusters in the same AWS account HOT 1
- Provider wants to replace existing kafka_acls when a new entry/ creating new resource is being added. HOT 7
- `terraform plan` returns unexpected diff if brokers have cluster-wide default config HOT 1
- Overrides for Topic Placement Policy HOT 1
- Kafka Compatibility Query HOT 2
- Kafka Broker Issue HOT 1
- Provider Cannot Assume Role From EKS HOT 2
- Empty Summary: This is always a bug in the provider and should be reported to the provider developers
- MSK Serverless - The client is not authorized to access this topic
- Empty Summary: This is always a bug in the provider and should be reported to the provider developers.
- resource_pattern_type_filter gets ignored
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from terraform-provider-kafka.