zerodha / kaf-relay Goto Github PK
View Code? Open in Web Editor NEWReplicate and sync kafka topics between clusters in realtime. Supports topic re-mapping, healthchecks, and hot failovers for high availability.
License: MIT License
Replicate and sync kafka topics between clusters in realtime. Supports topic re-mapping, healthchecks, and hot failovers for high availability.
License: MIT License
In a scenario where kaf-relay is in failover mode and there is a topic called "topicA" present on both serverA and serverB (individual kafka brokers), if the messaging flow into serverA becomes stuck due to some issue, we require the capability to switch over to serverB if the lag on serverA exceeds a certain threshold.
Interesting project but I was wondering, how is this different from MM2?
Filter plugin allows us to filter messages when we forward from one cluster to another. But, using this feature in tandem with failover mode is of little use and can lead to improper states when resetting offsets.
https://github.com/zerodha/kaf-relay/blob/main/producer.go#L48
^ We allow mapping from partition X from source to partition Y to destination. We should remap this when we are pulling destination offsets for resetting consumer offsets.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.