Comments (5)
@ods I gave it a shot, but I can't get it working, I am a bit lost when it comes to debug protocol issue (it seems it is what I am facing here). If you have few tips, I would be happy to rework my copy
from aiokafka.
@vmaurin I checked your PR works with minimal modifications for v0 and v1, but fails with v2. It looks like something is wrong with tagged fields support in aiokafka. As a first step we could add it without v2 for now and keep debugging problems with tagged fields.
from aiokafka.
For the record, the error in log when sending DeleteRecordsRequest_v2
is:
[2024-01-29 15:53:59,551] ERROR Exception while processing request from 172.17.0.2:57466-192.168.65.1:21560-1 (kafka.network.Processor)
org.apache.kafka.common.errors.InvalidRequestException: Error parsing request header. Our best guess of the apiKey is: 21
Caused by: java.nio.BufferUnderflowException
at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:155)
at java.nio.ByteBuffer.get(ByteBuffer.java:723)
at org.apache.kafka.common.protocol.ByteBufferAccessor.readArray(ByteBufferAccessor.java:58)
at org.apache.kafka.common.protocol.Readable.readUnknownTaggedField(Readable.java:52)
at org.apache.kafka.common.message.RequestHeaderData.read(RequestHeaderData.java:135)
at org.apache.kafka.common.message.RequestHeaderData.<init>(RequestHeaderData.java:84)
at org.apache.kafka.common.requests.RequestHeader.parse(RequestHeader.java:95)
at kafka.network.Processor.parseRequestHeader(SocketServer.scala:999)
at kafka.network.Processor.$anonfun$processCompletedReceives$1(SocketServer.scala:1012)
at java.util.LinkedHashMap$LinkedValues.forEach(LinkedHashMap.java:608)
at kafka.network.Processor.processCompletedReceives(SocketServer.scala:1008)
at kafka.network.Processor.run(SocketServer.scala:893)
at java.lang.Thread.run(Thread.java:750)
from aiokafka.
@ods Good catch. I copy pasted the TaggedFields from others definition, but it seems not so usable. In the Java version, they just mention "flexibleVersion" and then it seems to use different serializer
- https://github.com/apache/kafka/blob/trunk/clients/src/main/resources/common/message/DeleteRecordsRequest.json
- https://github.com/apache/kafka/blob/trunk/generator/src/main/java/org/apache/kafka/message/MessageDataGenerator.java#L424
It doesn't seem to work without declaring TaggedFields too
from aiokafka.
What I've found so far: although aiokafka declares v1 headers, it doesn't actually use it. After fixing request header broker doesn't complain anymore. But I'm still struggling to fix the parsing response part.
from aiokafka.
Related Issues (20)
- [QUESTION] I am using AIOkafka, but i have some question. HOT 1
- Inconsistent producer start/stop
- AIOKafkaAdminClient.create_topics fails randomly because it selects a node at random HOT 1
- performance degradation of producer when having many topics HOT 1
- Can I use Azure Event Hubs with `aiokafka`? HOT 1
- 0.9.0 zstd codec depends on cramjam but missing in documentation HOT 1
- Let's put `aiokafka` under the `aio-libs` org on PyPI HOT 2
- Add create_acls function for kafka admin client
- AIOKafkaProducer failed to produce message with headers HOT 3
- I keep getting MessageSizeTooLargeError, error message gives size much bigger than actual message were given to producer. HOT 9
- Can't connect to kafka docker HOT 1
- asyncio.exceptions.CancelledError
- Regarding Kafka Connection
- invalid Type AioKafkaAdminClient create_partitions
- Proposal to Add Type Hints HOT 12
- Consumer stopped consuming, task Fetcher._fetch_task has finished HOT 8
- High Incoming request sum on Azure Event Hub
- [QUESTION] Unable connect to node with id: X: [Errno 111]: Connection refused
- admin client - failure to create topics (error code 41) HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aiokafka.