Is pre-authentication against a second url supported? about kafka-connect-http

[castorm]

I think this feature might be hiding a bit more complexity than anticipated.

Regarding the token you get back from your authentication API:

• For how long is it valid?
• Is it for a number of requests of a period of time?
• How is it renewed? just by using it, or do you need to refresh it over time?

Regarding the implementation details, as of now, there are 2 ways of sharing state with the HTTP request:

• Offset: you can initially setup some offset properties that are later updated with the information of every record processed, so the last record processed determines the offset used in the next query. This is managed by Kafka Connect and persisted in Kafka, so it allows the connector to continue where it left in case of a restart. The kind of properties we would use here are timestamps or sequence numbers.
• Partition: just today I merged a feature that allows for the connector to partition requests, this allows to hit the same API with entirely different configurations, and route their responses to different topics (this part is not implemented yet). The point is, every partition has its own static state, which would be the set of properties that makes them different from each other.

What I initially thought about the implementation of this use case was to define a global/connector state resolved on startup, and accessible from where offset and partition state are currently being accessed, which is mainly HttpRequestFactory, where the HTTP Request is formed, usually with templated url, body, headers, etc.

However, and thinking about the different authentication scenarios resulting from the different answers to the questions above, we might want to deal with authentication in a specific way, instead of trying to generalize it into the above.

[castorm]

Hi @aggarwalb ,

Unfortunately, that usage pattern is not currently supported as of today.

I think it should be simple enough to implement though. feel free to take a look at the code and see if you could implement it yourself, I'll be happy to help any way I can.

Otherwise, it might not be implemented soon, as I'm a bit busy these days. Although I understand the value of this feature, so I think it will eventually be implemented.

Thanks,
Best regards.

[aggarwalb]

Hi @castorm ,
I am ready to contribute to this code if you are ok with this. Just one point if you can guide me how you want proceed with this further. This is very important feature as most the REST API world needs security access token to access the API's. If we are able to make this connector support this. then I will use this connector in our production solution else I have to write separate code in streams to fulfill my requirement.

[OneCricketeer]

The alternative to two calls per Kafka record (if not caching the API key) would be wrap the Kafka Connect API calls and embed the API key in the connector config

1. Hit https:///security-api/api (This gives back access token)
2. POST to kafka-connect/connectors with {"rest.api.key": "---key---", "rest.url": "https://customers"}

[aggarwalb]

@OneCricketeer When we hit the first API we will get record back in the kafka topic. What you are saying is reading the value from that topic and embed that value in connector config and then hit the second API with connector. Right?
If this is the case I would rather right one stream app in which I can wrap the two calls and process the message from source processor to sink processor.

[OneCricketeer]

record back in the kafka topic

No. You'd get the key back in an HTTP client, which you then forward as part of the connector config

[aggarwalb]

This means extending the existing code ?

[OneCricketeer]

Assuming your url doesn't support full URI like https://user:[email protected]/, then yes

[aggarwalb]

That is what I have already communicated in the above comments to @castorm that I am ready to contribute to code base for this requirement (as he agreed that this is good to have feature). The only thing I want is how he want things to be

[aggarwalb]

@castorm the case I am talking about needs to refresh token after a specified time interval. However, I agree with you that we have to think of all the authentication scenario's available for REST API's. I had seen some implementation being done in kafka http sink connector which we can include in this code base also. If you agree then we can plan implement authentication pattern's in it. By default we can configure it to No Auth.

[castorm]

Hi all,

As of 0.8.0 an HttpAuthenticator has been included to facilitate this extension. For now the available authentication types are just None and Basic, but implementing new types should be fairly straightforward as all it takes is implementing the method preparing the Authorization header:

@FunctionalInterface
public interface HttpAuthenticator extends Configurable {

    Optional<String> getAuthorizationHeader();
}

Authorization header is requested before executing the request, and after receiving a 401 response (so tokens could be refreshed at that point).

Oath2 or any other contribution for authentication mechanisms are welcome. At some point I might implement it myself if no contribution is received.

Thanks,
Best regards.

[ShawnUCD]

Hello @castorm, Are there any good examples of a working preauth config? I think the documentation is good when paired with the examples, but none seem to exist for the http.auth stuff yet. I'm new at this and trying to configure preauth.

[castorm]

Hi @ShawnUCD,

I'm afraid the reason there are no examples is because it's not just a matter of configuration. You have to implement your own authentication mechanism based on the extension point provided here (the interface shared in my last comment HttpAuthenticator).

The only authentication mechanism provided out-of-the-box is Basic auth, which, as documented, only requires for you to provide the following configuration properties:

"http.auth.type": "Basic",
"http.auth.user": "your-username",
"http.auth.pass": "your-password",

I hope that clears it up.
Best regards.

[ShawnUCD]

Yes, thank you for getting back to me. I'm trying to do a two-step auth where a bearer token is obtained (and renewed if expired) after POSTing user/pass (pub/priv API keys in this case) to an endpoint. If this kind of auth is not supported without more code (something that would take me a very long time) then I think I have my answer. Thank you for being so accessible, it's really nice to get such immediate feedback.

Shawn

[martimors]

May extend on our fork to cover getting a token from an auth endpoint now, but not sure how generalizable it is. The flow would be

• POST request to configured endpoint, ie. /auth, with payload like {"uid": "foo", "pwd": "bar"}
• retrieve token from response at configured path, ie. access_token if the response was {"access_token": "abcd1234567"}

Willing to contribute a feature for the main repository too, but some sparing would be welcome to cover as many use-cases as possible.

[martimors]

I've given this a shot at #142

[lobbin]

Just to post an update on this as well. The PR #142 works well in it's current form and I've added a patch to cache the token for a fixed time interval.

[szalapski]

I might use this. Maybe we can push #142 over the line?

@dingobar or @castorm, can you give guidance on how I might help? In the PR say you want integration tests as well as adaptation to work with OkHttp's Authenticator.

I'm not sure I can get my environment set up to build or run integration tests. I don't think I can really push actual development forward but I'd be happy to test. Or is this connector too dead to have hope? I'm beginning to wonder if Kafka Connect is dead, as it seems silly that there is no popular HTTP source connector anywhere.

Also, for what it's worth, I think fetching a token perhaps should be done preemptively, not having to get a 401 before doing so. If Token_Endpoint is the mode but we have no token, my vote is we should go get one before attempting the API call.