Discussion: Specifying Kafka configurations about streamsx.kafka

[ddebrunner]

Maybe rather than trying to have all these options on a single operator, maybe Kafka Producer/Consumer operator that handles the low-level properties and MessageHub Producer/Consumer wrappers that are only intended to support MessageHub and have their configuration focussed on that.

So one set focused on connecting directly to Kafka servers and one on connecting to MessageHub.

[Alex-Cook4]

+1 to the idea of MessageHub-only wrappers. This would help make a virtually fool-proof operator for that case.

[cancilla]

That wasn't what I meant, but instead the app should be able to have the last word in setting properties, so if it sets some_kafka_property='a' then that wins.

Let me see if I understand your point. You are suggesting that app config should primarily be used for setting things that will be the same across applications, such as broker urls or credentials. For other properties that affect how the application works (i.e. acks=all or buffer.size=whatever), the application should be allowed to set (or override) these properties as needed.

This boils down to a couple of things then:

1. The property precedence should be changed such that properties set at the application level take precedence over the properties set at the app config level
2. Documentation on the best practices when using these operators

[ddebrunner]

What would the format of the submission time value be?

[Alex-Cook4]

What would be the method for providing the username/password required in the jaas.conf file for the Message Hub login? One solution that I haven't tried out might be to leverage the new sasl.jaas.config Kafka Property. Discussion about this is here.

[cancilla]

@ddebrunner:

• For the submission-time value, I was planning to keep it as a list of key=value pairs (same as the existing Kafka operators)
• As for the MessageHub wrappers, could you provide a little more info on how you think this should work? Would the MessageHub ops be Java primitive operators that extend the "low-level" Kafka operators, or would they be SPL composites?

[cancilla]

@Alex-Cook4 Sorry, I forgot to mention the JAAS config in the above proposal. My plan is to eliminate the jaasFile and jaasFilePropName parameters and use the sasl.jaas.config property that you mentioned. I already prototyped this in my workspace and I am able to connect to MessageHub same as before. More interestingly, I am able to construct the value for this property using only the information provided in the MessageHub creds. This means users will no longer have to deal with JAAS files when connecting to MessageHub.

Users connecting to a secured on-prem Kafka cluster will still need to provide JAAS information, but they can pass in that info using the sasl.jaas.config property rather than as a separate file.

Edit: I added a section to the above proposal regarding SASL/JAAS.

[Alex-Cook4]

Excellent! Yes, I didn't realize that everything is in the credentials file now. That will make this so much easier.

I think @ddebrunner has done this before with the MQTT operators, but this would allow us to upload .sab files that just take a submission-time parameter or AppConfig to connect to MessageHub, then just have a publish/subscribe way for other Streams apps to get and write to MessageHub. Maybe that's obvious, but I find it exciting!

[ddebrunner]

For the submission-time value, I was planning to keep it as a list of key=value pairs (same as the existing Kafka operators)

Just seems like that's going to be pretty ugly to enter as a submission time value.

I can see why a submission value might be an idea, but at least for distributed/bluemix using an application configuration is a better approach. Then one has a submission parameter that provides the app config name rather than the actual configuration.

[ddebrunner]

I think we need to be careful about trying to cover every possible situation and making the configuration too complex, thus making the operator harder to use. Having five levels of finding config information with some having multiple entries might be over thinking the problem.

It's also important to note that there are (I think) two types of kafka configuration items:

1. Information to identify the Kafka broker
2. Configuration for the application, e.g. the setting for ensuring the message makes it to all brokers.

The second set should not be overwritten by configuration since it changes the application behaviour, e.g. breaking consistent region guarantees.

As @Alex-Cook4 pointed out, for MessageHub I want a simple foolproof way to connect to it, ideally a single parameter, which app config to use, maybe even defaulting to one named "message-hub".

[ddebrunner]

Java primitive or composite is really an implementation detail, though if the operator has knowledge of submission time parameters it has to be a composite.

I would be somewhat wary of adding support for submission time value support in an operator, it means that the operator always exposes ways to change its configuration, thus no longer left up to the application to decide. A microservice that exposes submission time value is fine, an operator I'm not so sure about, submission time values are really the application's responsibility.

[cancilla]

Just seems like that's going to be pretty ugly to enter as a submission time value.

I can see why a submission value might be an idea, but at least for distributed/bluemix using an application configuration is a better approach. Then one has a submission parameter that provides the app config name rather than the actual configuration.

@ddebrunner I agree that app configs make more sense. The use cases I have in mind for this are:

1. Running an app in standalone
2. Using the operators in a Java topology that is submitted using an Embedded context

In both cases, the submission-time parameter provides a straight-forward way to dynamically/programmatically set things like brokers or to test different property values.

[cancilla]

I think we need to be careful about trying to cover every possible situation and making the configuration too complex, thus making the operator harder to use. Having five levels of finding config information with some having multiple entries might be over thinking the problem.

That's a good point. I think we'll start without the kafkaProperty parameter and if someone presents a valid need for it, we can add it back in. It always easier to add something to the operators than to remove it.

[cancilla]

It's also important to note that there are (I think) two types of kafka configuration items:

1. Information to identify the Kafka broker
2. Configuration for the application, e.g. the setting for ensuring the message makes it to all brokers.

The problem is that Kafka doesn't make this sort of distinction...everything is a property, whether it's setting the brokers, setting up creds or changing app behaviour. I worry about trying to invent categories (connectivity vs app config) for the different properties and then treating those properties in a different way.

[ddebrunner]

@cancilla Maybe the same case for multple configuration items., e.g. multiple application configurations. What would you expect to be in each app config?

[ddebrunner]

I worry about trying to invent categories (connectivity vs app config) for the different properties

That wasn't what I meant, but instead the app should be able to have the last word in setting properties, so if it sets some_kafka_property='a' then that wins.

[cancilla]

Thanks for the feedback! As per the discussion, the operators will:

• allow properties to be specified either via an appConfig or propertiesFile
• properties specified via a propertiesFile will override any properties specified in an appConfig
• A separate repo and toolkit have been created to for MessageHub operators