Comments (5)
One thing I'm not sure about is how to calculate the offset lag per partition. Maybe I'm missing something, but I don't see a way to get this using ListOffset, etc.
from kafkajs.
You will have to use broker#offsetFetch
because you want the group offset and not the topic offset.
from kafkajs.
But that just gives you the group offset, but to calculate the lag you'd also need the highwater mark and the starting offset.
from kafkajs.
It's a combination of listOffset and offsetFetch
from kafkajs.
I have a PoC implementation now that seems to work. I'll create a PR with it as soon as I've dealt with some error cases, and get some feedback. The assignment algorithm itself seems to produce balanced assignments. It's mostly calculating the offset lag that's not the nicest at the moment.
I'm not sure it makes sense to bundle it with the main distribution of KafkaJS though, as for most people the round-robin assigner would be fine. Since the assigner is pluggable anyway, I think it might make more sense to publish it as a separate module.
from kafkajs.
Related Issues (20)
- Incorrect API Versions Used on Clusters with Mixed Kafka Versions
- "[BrokerPool] Failed to connect to seed broker, trying another broker from the list: Connection error: Client network socket disconnected before secure TLS connection was established"
- Question: How to add multiple subscribers
- KafkaJS Consumer Offset Reset Issue after Manual Commit HOT 3
- Forking the repo to my company for now ExtendOSS/kafkajs come contribute
- kafkajs is not compatible with Nodev18 HOT 5
- Can I assign a consumer to a specific partition ?
- `eachBatchAutoResolve: false` and using of transaction`tx` results in an infinite number of messages
- How to prioritise the topic in KafkaJS
- Missing customLogger when sending message
- There is any way to find the connection pool size of my application?
- Bumping "ip" dev dependency to latest (prevent false-positives from vulnerability scanners) HOT 1
- How to consume messages from multiple topics HOT 1
- Increate KafkaJS producer default retry HOT 1
- Question: Is there a way to get the status of all consumers inside a consumer group? HOT 1
- Kafka producer.send(timeout: 1) other than 1ms- no other number seems to be working
- [readme] Community Slack invitation link is broken HOT 1
- maxBytesPerPartition not working as stated in the documentation HOT 1
- Has this project been abandoned? HOT 1
- RoundRobin Assignor doesn't acknowledge Member subscribed topics in assignment
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kafkajs.