Comments (7)
Hey @vmaurin
can you please check what is you api_version
when you set it auto
?
In my case after before they upgrade from kafka 2.7 to kafka 3.5 i was able to connect with auto
but after i got this error
Unable to request metadata from "kafkaserver:9093":
suddenly that work once after waiting 2 minutes for connecting and the api_version
was 2.6.0
So, when i set api_version
= "2.6.0" that connect directly, otherwise got error, or sometime i'm able to connect without knowing why it's happen !!!
@vmaurin BTW, according to the code you can't get broker version higher than 2.6.0
from aiokafka.
Hi @Ben621
My company is using aiokafka against AWS MSK 3.6.0 without issue. I think the code you highlight is just happening when you force the api_version
to something different to "auto"
from aiokafka.
Hey @vmaurin
Can please try to print the api_version
In my case we are using 3.4.x,
But I'm unable to connect to get the api_version
But suddenly that connect, I don't know why !!
After. Investigation that work when I set api_version to 2.8.1
Maybe are you familiar with this issue ?
Also I have checked in the GitHub action they are testing client "0.9.0.1" To "2.8.1"
from aiokafka.
From the codebase, it sounds parse_kafka_version
you pointed out is only use when initializing a producer or a consumer with an api_version
. If you let it unset like we do (auto
), this method is never called. Then when the client (aiokafka) is connecting to the broker (3.6.0), they will negotiate API version for each individual methods part of the kafka protocol.
I guess then, this parameter is to force a specific version of the protocol, but it should not be used, as Kafka protocol is build for backward and forward compatibility (even today version of the broker probably still support clients in 0.11). So my advice is to let this parameter unset
from aiokafka.
Hi @Ben621, thank you for highlighting the problem. We need to add newer versions for the broker to the CI matrix. This will probably require reworking how we build the Docker images for them. It would be nice if somebody volunteers to do this part.
from aiokafka.
There is an error reported in aiokafka/util.py that restricts compatibility.
Could you please provide a complete report with the traceback?
@vmaurin BTW, according to the code you can't get broker version higher than 2.6.0
The method you're referring to looks weird to me. In my opinion, we should rather drop support for brokers <0.10.0 and rely solely on what ApiVersions
returns.
from aiokafka.
Hi @Ben621, thank you for highlighting the problem. We need to add newer versions for the broker to the CI matrix. This will probably require reworking how we build the Docker images for them. It would be nice if somebody volunteers to do this part.
I've been meaning to do this in https://github.com/wbarnha/kafka-python-ng for newer Kafka broker versions. If I get a working solution, I'll open a PR here as well.
from aiokafka.
Related Issues (20)
- Unexpected error during batch delivery HOT 1
- IncompatibleBrokerVersion: Kafka broker does not support the 'CreateTopicsRequest_v0' Kafka protocol. HOT 4
- [QUESTION] Restarting `AIOKafkaConsumer` after `AIOKafkaConsumer.stop()` HOT 3
- [QUESTION] How to close connection if Kafka server isn't responding?
- [QUESTION] Is there support to refresh OAuth tokens automatically based on token expiration duration?
- Add client.rack configuration option to support consumer fetching from closest replica (KIP-392)
- [QUESTION] idempotent producer and OutOfOrderSequenceNumber problem
- Can't Recover From NotLeaderForPartitionError
- Unclosed AIOKafkaConnection that does not cause the consumer to crash HOT 1
- Connection errors are spamming HOT 1
- A large and controversial proposal: replace protocol implementation with kio HOT 3
- Severe performance degradation after migrating to cramjam LZ4
- Losing Messages Between Producer and Consumer HOT 4
- lz4 compression codec not found even after installing lz4 module HOT 5
- Long running consumer stop receiving messages and it's state is not closed HOT 3
- Closing of connection from server don't caught for uvloop
- Producer.send_and_wait() hangs up forever if `sender_task` is cancelled
- [QUESTION] Is there a way to monitor the heartbeat state?
- [QUESTION] Is there a way to monitor the heartbeat state? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aiokafka.