Comments (5)
Now that we've got an ssl-test
topic on Heroku I'm digging into how to setup brod to talk to SSL Kafka. (And also looking around the landscape of Elixir Kafka clients.)
(time passes)
Ok, brod and the leading Elixir Kafka client (kafka_ex) have the same "problem". They expect the SSL certs to be actual files:
{:cacertfile, "cacert.pem"},
{:certfile, "cert.pem"},
{:keyfile, "cert_key.pem"},
I'm not sure how that will match up with having the certs as ENV values. I hope there's a way to hand over the cert values directly instead of handing over filenames with the cert values.
For comparison here's Ruby-Kafka: https://github.com/spreedly/ruby-kafka/blob/master/heroku-kafka-consumer.rb#L48-L61
brokers = ENV.fetch("KAFKA_URL").split(",")
ssl_ca_cert = ENV.fetch("KAFKA_TRUSTED_CERT")
ssl_client_cert = ENV.fetch("KAFKA_CLIENT_CERT")
ssl_client_cert_key = ENV.fetch("KAFKA_CLIENT_CERT_KEY")
kafka = Kafka.new(
seed_brokers: brokers,
client_id: client_id,
socket_timeout: 120,
logger: logger,
ssl_ca_cert: ssl_ca_cert,
ssl_client_cert: ssl_client_cert,
ssl_client_cert_key: ssl_client_cert_key,
)
from kaffe.
And here's how to connect to Heroku Kafka from kafkacat
: for posterity
$ kafkacat \
-t ssl-test \
-o beginning \
-e \
-b $(echo $KAFKA_URL | sed -e 's/kafka+ssl:\/\///g') \
-X security.protocol=ssl \
-X ssl.key.location=<(echo $KAFKA_CLIENT_CERT_KEY) \
-X ssl.certificate.location=<(echo $KAFKA_CLIENT_CERT) \
-X ssl.ca.location=<(echo $KAFKA_TRUSTED_CERT)
from kaffe.
Yeah it's pretty clear that brod and kafka_ex are both using the Erlang SSL module:
http://erlang.org/doc/man/ssl.html
Happily it looks like that module can accept values and not just files. I'll try setting them for brod. First with the certs as files like it wants and next with the underlying Erlang SSL module options for values.
from kaffe.
Woo! After much experimentation I finally found the magical combination of extracted pieces and SSL incantations to allow us to configure the Erlang SSL module used by brod to talk to the Heroku Kafka!
The magics:
def consumer_config do
[
auto_start_producers: false,
allow_topic_auto_creation: false,
ssl: [
cert: client_cert,
key: client_cert_key,
]
]
end
defp client_cert do
{_type, der, _} = "KAFKA_CLIENT_CERT"
|> System.get_env
|> :public_key.pem_decode
|> List.first
der
end
defp client_cert_key do
{type, der, _} = "KAFKA_CLIENT_CERT_KEY"
|> System.get_env
|> :public_key.pem_decode
|> List.first
{type, der}
end
This is all hacked together but working!
Next is to nicely expose the SSL configuration pieces. I have no problem with Kaffe assuming Heroku Kafka SSL setup since it is our client after all.
Perhaps a config like…
config :kaffe,
consumer: [
endpoints: [kafka: 9092],
ssl: [
client_cert: System.get_env("KAFKA_CLIENT_CERT"),
client_cert_key: System.get_env("KAFKA_CLIENT_CERT_KEY")
],
topics: ["whitelist"],
offset_commit_interval_seconds: 5, # default
begin_offset: :earliest # default :latest
],
producer: [
endpoints: [kafka: 9092],
ssl: [
client_cert: System.get_env("KAFKA_CLIENT_CERT"),
client_cert_key: System.get_env("KAFKA_CLIENT_CERT_KEY")
],
topics: ["decorated-whitelist"],
partition_strategy: :round_robin # default
]
Using default values…
config :kaffe,
consumer: [
endpoints: [kafka: 9092], # yeah, gotta pull these from KAFKA_URL
ssl: [
client_cert: System.get_env("KAFKA_CLIENT_CERT"),
client_cert_key: System.get_env("KAFKA_CLIENT_CERT_KEY")
],
topics: ["whitelist"]
],
producer: [
endpoints: [kafka: 9092], # yeah, gotta pull these from KAFKA_URL
ssl: [
client_cert: System.get_env("KAFKA_CLIENT_CERT"),
client_cert_key: System.get_env("KAFKA_CLIENT_CERT_KEY")
],
topics: ["decorated-whitelist"]
]
from kaffe.
What we get from KAFKA_URL:
iex(1)> "KAFKA_URL" |> System.get_env
"kafka+ssl://34.195.140.72:9096,kafka+ssl://34.195.135.147:9096,kafka+ssl://34.195.140.195:9096"
What brod wants for its endpoints configuration:
[{:"34.195.140.72", 9096}, {:"34.195.135.147", 9096}, {:"34.195.140.195", 9096}]
A straight up data transformation problem?! Yesssss
from kaffe.
Related Issues (20)
- Defining multiple handlers HOT 1
- worker_per_topic_partition with multiple topics HOT 1
- Examples not compatible with Elixir 1.10 or 1.11 HOT 2
- extract_der is giving error with SSL HOT 2
- Undefined function exponential_backoff HOT 10
- Offset doesn't get updated between runs and runs crash with OOM errors HOT 4
- async ack - lots of duplicate messages until I restart the application HOT 2
- Kaffe.Producer.produce_sync raises on timeout
- How to set kafka headers when publishing message HOT 1
- Invalid call to raise/reraise on brod/kpro error
- Add support for SCRAM mechanism in SASL authentication. HOT 1
- Module to help write ExUnit tests
- It's impossible to create 2 separate consumers for different topics
- Running mix with kaffe deps fails to download pc package from hex
- Wrong place for configuration
- Repeated rebalance cycle with kafka broker 2.3.0 HOT 16
- kaffe cannot recover from unreachable Kafka HOT 18
- Missing documentation HOT 8
- Connecting to a TLS-based Kafka instance under AWS MSK? HOT 18
- Receives notification when rebalance in progress/assignments revoked HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kaffe.