Update `aiokafka` library to support higher client versions > 3 about aiokafka

[Ben621]

Hey @vmaurin
can you please check what is you api_version when you set it auto ?

In my case after before they upgrade from kafka  2.7 to kafka 3.5 i was able to connect with auto
but after i got this error

Unable to request metadata from "kafkaserver:9093":

suddenly that work once after waiting 2 minutes for connecting and the api_version was 2.6.0

So, when i set api_version= "2.6.0" that connect directly, otherwise got error, or sometime i'm able to connect without knowing why it's happen !!!

@vmaurin BTW, according to the code you can't get broker version higher than 2.6.0

[vmaurin]

Hi @Ben621

My company is using aiokafka against AWS MSK 3.6.0 without issue. I think the code you highlight is just happening when you force the api_version to something different to "auto"

[Ben621]

Hey @vmaurin
Can please try to print the api_version
In my case we are using 3.4.x,
But I'm unable to connect to get the api_version
But suddenly that connect, I don't know why !!
After. Investigation that work when I set api_version to 2.8.1
Maybe are you familiar with this issue ?

Also I have checked in the GitHub action they are testing client "0.9.0.1" To "2.8.1"

[vmaurin]

From the codebase, it sounds parse_kafka_version you pointed out is only use when initializing a producer or a consumer with an api_version. If you let it unset like we do (auto), this method is never called. Then when the client (aiokafka) is connecting to the broker (3.6.0), they will negotiate API version for each individual methods part of the kafka protocol.

I guess then, this parameter is to force a specific version of the protocol, but it should not be used, as Kafka protocol is build for backward and forward compatibility (even today version of the broker probably still support clients in 0.11). So my advice is to let this parameter unset

[ods]

Hi @Ben621, thank you for highlighting the problem. We need to add newer versions for the broker to the CI matrix. This will probably require reworking how we build the Docker images for them. It would be nice if somebody volunteers to do this part.

[ods]

There is an error reported in aiokafka/util.py that restricts compatibility.

Could you please provide a complete report with the traceback?

@vmaurin BTW, according to the code you can't get broker version higher than 2.6.0

The method you're referring to looks weird to me. In my opinion, we should rather drop support for brokers <0.10.0 and rely solely on what ApiVersions returns.

[wbarnha]

Hi @Ben621, thank you for highlighting the problem. We need to add newer versions for the broker to the CI matrix. This will probably require reworking how we build the Docker images for them. It would be nice if somebody volunteers to do this part.

I've been meaning to do this in https://github.com/wbarnha/kafka-python-ng for newer Kafka broker versions. If I get a working solution, I'll open a PR here as well.