Comments (3)
Thanks for your suggestion @srolija! The Mirus Source Connector is primarily intended for high-reliability data replication use cases, so our defaults are selected to minimize the risk of data loss at all costs — even at expense of increased risk of duplicate data. By using the earliest
policy we guarantee that all available data is replicated, even in exceptional circumstances. This policy covers us when new topics are added to the topic regex and, importantly, also in rare instances where a Kafka bug causes the current offset to become invalid and the consumer offsets are reinitialized. This is something we have occasionally seen, particularly in earlier Kafka releases and using latest
in that situation would certainly lead to data loss.
from mirus.
Completely get it.
Would it then make sense just to note the changes from the default consumer group? Asking since we had an issue where it started replicating enormous topic; and based on the docs we didn't understand that it has non-default configuration.
from mirus.
Yes, that does make sense - we should update the the docs.
from mirus.
Related Issues (20)
- Mirror not correct HOT 1
- Getting similar function as "measuring lag" (mirrormaker) in mirus HOT 6
- Deprecate Prefix & Suffix settings in favor of RegexRouter HOT 1
- Enabling SSL connectivity HOT 9
- failed to run the quick start example HOT 4
- Mirus failing to start when brokers are deployed to new ip's HOT 7
- Can we manipulate the filter any of the config parameters(eg. password configuration) during the GET call HOT 5
- mirus.connector.auto.restart.enabled doesnt work HOT 1
- org.apache.kafka.common.errors.RecordTooLargeException produce duplicated messages
- API end point for health check HOT 2
- Mirus deployed in k8s pod is failing with "java.lang.OutOfMemoryError" HOT 2
- Mirus tasks failing with error "Could not initialize class org.xerial.snappy.Snappy"
- Can Mirus run as a Kafka Connect connector? HOT 2
- Configuring the Mirus quickstart to replicate from one system to another HOT 7
- Not able to convert header value of replicated topic from ByteArray to String HOT 1
- Produce msg to topic test does not reflect on test.mirror HOT 1
- ReplayPolicy.IGNORE does not process new records HOT 2
- SSL configuration
- Excessive logging (Exception thrown while calling task.commitRecord())
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mirus.