had error: api key -10495 is invalid about kafka-proxy

[everesio]

This is probably miss configuration.
How are your kafka brokers and kafka clients configured, which versions are used ?
How do you start kafka-proxy ?

[hekaldama]

I am getting this as well. My topology is VPC peering my AWS to confluent.cloud and trying to get my localhost (outside of AWS env) to be able to reach confluent.cloud.

[everesio]

Could you provide server, proxy and client configuration ?

[hekaldama]

This is the environment I have setup:

https://docs.confluent.io/current/cloud/vpc.html

I don't have many settings on the server side available to me but I do know that I can connect to it on server:9092 from an instance running on my AWS env.

The proxy settings I am currently trying are:

kafka-proxy server --bootstrap-server-mapping "server:9092,0.0.0.0:9092" \
  --log-level debug \
  --tls-enable \
  --tls-insecure-skip-verify

The client settings I am using are something to the affect of:

// kafka client version 2.4.0
import org.apache.kafka.clients.admin.{AdminClient, AdminClientConfig}
import java.util.Properties

val config = ConfigFor("environment")
val properties = new Properties()
properties.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, config.bootstrapServerUrl)
properties.put(AdminClientConfig.REQUEST_TIMEOUT_MS_CONFIG, "300000")
properties.put("ssl.endpoint.identification.algorithm", "https")
properties.put("sasl.mechanism", "PLAIN")
properties.put("sasl.jaas.config", s"""org.apache.kafka.common.security.plain.PlainLoginModule required username="${config.clusterAPIKey}" password="${config.clusterAPISecret}";""")
properties.put("security.protocol", "SASL_SSL")
AdminClient.create(properties)

I did try with jaas on the proxy side with the same credentials, but was receiving the same error.

My specific error looks like:

INFO[2020-02-04T12:10:10Z] Starting kafka-proxy version 0.1.6           
INFO[2020-02-04T12:10:10Z] Bootstrap server server:9092 advertised as 0.0.0.0:9092 
INFO[2020-02-04T12:10:10Z] Listening on 0.0.0.0:9092 ([::]:9092) for remote server:9092 
INFO[2020-02-04T12:10:10Z] Ready for new connections
INFO[2020-02-04T12:11:31Z] New connection for server:9092 
DEBU[2020-02-04T12:11:31Z] Sending SaslHandshakeRequest mechanism: PLAIN  version: 0 
DEBU[2020-02-04T12:11:31Z] Successful SASL handshake. Available mechanisms: [PLAIN OAUTHBEARER] 
DEBU[2020-02-04T12:11:31Z] Sending authentication opaque packets, mechanism PLAIN 
DEBU[2020-02-04T12:11:31Z] Kafka request key -18175, version 0, length 369296128 
INFO[2020-02-04T12:11:31Z] Reading data from local connection on 172.28.196.187:9092 from 172.20.80.195:60232 (server:9092) had error: api key -18175 is invalid 

Where 172.28.196.187 is the kafka-proxy IP and 172.20.80.195 is my localhost.

LMK if there is anything else I can provide and I am digging into our env more to see what I can find out. Thanks!

[everesio]

1. Is the broker server:9092 really a TLS port (check e.g. with openssl s_client -connect server:9092) ?
   in most cases the port 9092 is a plain listener https://docs.confluent.io/current/installation/configuration/broker-configs.html
   control.plane.listener.name INTERNAL://192.1.1.8:9092, listener.security.protocol.map = INTERNAL:PLAINTEXT

if it is not, use the correct TLS port or do not use "--tls-enable" in the proxy.
If you want terminate TLS on proxy please use "--proxy-listener-tls-enable"

2. What is the version of the broker ? please provide output from bin/kafka-broker-api-versions.sh
   if your kafka client version is 2.4.0 and the server is also 2.4.0, the proxy will not work as the were changes in the kafka protocol which have not been implemented yet (it will be soon).
   However if you downgrade the client to 2.3.x, older message version version will be negotiated and proxy will work.

[datafisk]

@hekaldama Did you get kafka-proxy working properly with socks or http proxying towards Confluent Cloud? Very interested in any working configs if that is the case. Cheers.

[hekaldama]

@datafisk I was not able to work on it anymore after my original comment. I will definitely post back here if I figure anything out. We are going to be setting up AWS transit gateway to at least allow VPN -> confluent.cloud access.

[sarwarbhuiyan]

@hekaldama Did you get kafka-proxy working properly with socks or http proxying towards Confluent Cloud? Very interested in any working configs if that is the case. Cheers.

I can forward-proxy to Confluent Cloud via squid but not via sasl_ssl. Only when I set sasl_plaintext on the client side.

[datafisk]

I think ii got this sorted @sarwarbhuiyan , i will DM you a link.

[afoley-st]

I am running in to similar issues when connecting to confluent cloud from the kafka-proxy and wanted to see if anyone was successful with that:

had error: api key XXXX is invalid 

My server command:

./kafka-proxy server --bootstrap-server-mapping "confluent-server:9092,0.0.0.0:9092" \
  --debug-enable \
  --log-level="debug" \
  --tls-enable \
  --tls-insecure-skip-verify

And the kafka properties on the server:

bootstrap.servers=confluent-server:9092
ssl.endpoint.identification.algorithm=https
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="{{ API_KEY }}" password="{{ API_SECRET }}";

Does anything jump out from my configuration or kafka-proxy setup?

[p53]

@afoley-st don't you miss --sasl-enable option?

[afoley-st]

@afoley-st don't you miss --sasl-enable option?

The same issue arises:

./kafka-proxy server --bootstrap-server-mapping "confluent-server:9092,0.0.0.0:9092" \
  --debug-enable \
  --log-level="debug" \
  --sasl-enable \
  --sasl-username="${SASL_USERNAME}" \
  --sasl-password="${SASL_PASSWORD}" \
  --tls-enable --tls-insecure-skip-verify
INFO[2021-06-02T13:22:40Z] Starting kafka-proxy version 0.2.8           
INFO[2021-06-02T13:22:40Z] Bootstrap server confluent-server:9092 advertised as 0.0.0.0:9092 
INFO[2021-06-02T13:22:40Z] Listening on 0.0.0.0:9092 ([::]:9092) for remote confluent-server:9092 
INFO[2021-06-02T13:22:40Z] Ready for new connections                    
INFO[2021-06-02T13:22:50Z] New connection for confluent-server:9092 
DEBU[2021-06-02T13:22:50Z] Sending SaslHandshakeRequest mechanism: PLAIN  version: 0 
DEBU[2021-06-02T13:22:50Z] Successful SASL handshake. Available mechanisms: [PLAIN OAUTHBEARER] 
DEBU[2021-06-02T13:22:50Z] Sending authentication opaque packets, mechanism PLAIN 
DEBU[2021-06-02T13:22:50Z] Kafka request key -3327, version 1, length 369296129 
INFO[2021-06-02T13:22:50Z] Reading data from local connection on XXXXX:9092 from XXXXX:61791 (confluent-server:9092) had error: api key -3327 is invalid

Is the --proxy-listener-tls-enable setting a requirement?

[p53]

@afoley-st no --proxy-listener-tls-enable should not be requirement. What version of client/server you have? What is your listener.security.protocol.map

[afoley-st]

@afoley-st no --proxy-listener-tls-enable should not be requirement. What version of client/server you have? What is your listener.security.protocol.map

It's running Apache Kafka Version 2.7.0. I can't get the entire listener.security.protocol.map (don't have access because its a managed service via confluent), but the bootstrap endpoint is SASL_SSL://confluent-server:9092

[p53]

and client kafka version?

[afoley-st]

and client kafka version?

I've tried with both 2.7.0 and 2.8.0

[p53]

@afoley-st did you try also with 2.3 see #28 (comment)

[datafisk]

@afoley-st skip the properties for your kafka client, the kafka-proxy has the required settings as per this config and your client will just initiate the connection to the proxy with just the bootstrap as required config, the proxy takes care of the rest.

@afoley-st don't you miss --sasl-enable option?

The same issue arises:

./kafka-proxy server --bootstrap-server-mapping "confluent-server:9092,0.0.0.0:9092" \
  --debug-enable \
  --log-level="debug" \
  --sasl-enable \
  --sasl-username="${SASL_USERNAME}" \
  --sasl-password="${SASL_PASSWORD}" \
  --tls-enable --tls-insecure-skip-verify
INFO[2021-06-02T13:22:40Z] Starting kafka-proxy version 0.2.8           
INFO[2021-06-02T13:22:40Z] Bootstrap server confluent-server:9092 advertised as 0.0.0.0:9092 
INFO[2021-06-02T13:22:40Z] Listening on 0.0.0.0:9092 ([::]:9092) for remote confluent-server:9092 
INFO[2021-06-02T13:22:40Z] Ready for new connections                    
INFO[2021-06-02T13:22:50Z] New connection for confluent-server:9092 
DEBU[2021-06-02T13:22:50Z] Sending SaslHandshakeRequest mechanism: PLAIN  version: 0 
DEBU[2021-06-02T13:22:50Z] Successful SASL handshake. Available mechanisms: [PLAIN OAUTHBEARER] 
DEBU[2021-06-02T13:22:50Z] Sending authentication opaque packets, mechanism PLAIN 
DEBU[2021-06-02T13:22:50Z] Kafka request key -3327, version 1, length 369296129 
INFO[2021-06-02T13:22:50Z] Reading data from local connection on XXXXX:9092 from XXXXX:61791 (confluent-server:9092) had error: api key -3327 is invalid

Is the --proxy-listener-tls-enable setting a requirement?

[p53]

@afoley-st you should connect to your kafka-proxy server without SASL credentials/SSL (if are doing it right now, kafka-proxy already authenticates/encrypts connection with confluent kafka) as @datafisk mentioned

[vietanhduong]

If I enable SASL, it's working perfectly but, If I remove --sasl-enable flag and try to connect from client, this error will appear.

I was following this guide to connect to cluster.
https://docs.conduktor.io/kafka-cluster-connection/setting-up-a-connection-to-kafka/connecting-to-a-secure-kafka

[vietanhduong]

If I enable SASL, it's working perfectly but, If I remove --sasl-enable flag and try to connect from client, this error will appear.

I was following this guide to connect to cluster. https://docs.conduktor.io/kafka-cluster-connection/setting-up-a-connection-to-kafka/connecting-to-a-secure-kafka

After few hours debug, I realized the problem is protocol I’m using. Just changed it to SASL_PLAINTEXT and then everything works perfectly.

[datafisk]

Updating this ticket for the SASL passthrough issue against Confluent Cloud, that generates the API key error. Since Confluent Cloud uses. SASL_PLAN over TLS the proxy needs to be setup with TLS listeners and it will translate/proxy/passthrough the connection details to Confluent Cloud.
Below is an examle:

kafka-proxy server \
--bootstrap-server-mapping \
"pkc-lgwgm.eastus2.azure.confluent.cloud:9092,localhost:9092" \
--log-level DEBUG \
--proxy-listener-cert-file ./proxy.cert \
--proxy-listener-tls-enable \
--proxy-listener-key-file ./proxy.key \
--tls-enable

Then you just setup the kafka client to talk to the proxy, localhost in my example, so from kafkacat:

export USERNAME=<API_KEY/SASL_USERNAME>
export PASSWORD=<API_SECRET_KEY/SASL_PASSWORD>
export BROKERS=localhost:9092

kafkacat -b $BROKERS -L -X security.protocol=SASL_SSL -X sasl.mechanisms=PLAIN -X sasl.username=$USERNAME -X sasl.password=$PASSWORD 

If you use self-signed certs for some testing you want to add -X enable.ssl.certificate.verification=false to kafkacat to make testing the passthrough simpler.

[pleung826]

can anyone post a fully working config + start scripts for:
kafka-proxy -> squid -> confluent kafka ?

I'm not able to get this to work even without squid. I've tried a number of different CLI options to kafka-proxy and with confluent-7.3.0/bin/kafka-console-producer, none of it works for me.