Comments (4)
Ok I see. Well the short answer is no, not out-of-the-box based on the current version. However, many functionalities of the sink connector implementation have been written with customizability in mind. That said, you could come up with your own individual write model behaviour by implementing the corresponding interfaces and / or extending existing classes with your custom ones where you override the default behaviour.
One more question, just to make sure I fully got you. Instead of sourcing your real-time processing application based on the change streams topic resulting after sinking the data in to MongoDB, why can't you just directly source this from the original kafka topic that results from the Postgres CDC?
from kafka-connect-mongodb.
Hi @HyunSangHan!
thanks for reaching out. So you are right that the MongoDB SinkConnector is implemented such that it applies a replace operation on the document that was matched. This is by design, and allows to employ an upsert semantic for both insert and update operations that it processes from the kafka topic. Also a CDC message from kafka always results in updating the full document on MongoDB side, so partial updates aren't supported because you don't really need them for a CDC pipeline use case.
So maybe you can elaborate a bit, why would you need to have a differentiation in that regard? The end result after processing the CDC records with the currently imposed write model semantics should give you the correct document state on MongoDB side. But it might be, that I misunderstood your question.
from kafka-connect-mongodb.
I'm so glad to get your quick reply!! Thank you :)
So maybe you can elaborate a bit, why would you need to have a differentiation in that regard?
I am planning to make more pipeline after consuming the message.
Let me explain as below.
I told you "Postgres
--> Kafka
--> MongoDB
", that exactly means
Postgres --> Kafka --> (MongoSinkConnector) --> MongoDB
but there was a part that has been omitted. My whole plan is:
Postgres --> Kafka --> (MongoSinkConnector) --> MongoDB --> (Mongo change streams) --> (Real-time processing application) --> (Kafka) --> (Many applications as consumers) --> ...
As a result, I will use changes of documents to produce Kafka messages again by Mongo change streams
.
When using mongo change streams
, I can get updateDescription.updatedFields
that shows what fields are updated if it has update
as the operationType. However, if operationType is replace
, there's no field like that. As you can see mongodb docs:
update
: https://docs.mongodb.com/manual/reference/change-events/#update-eventreplace
: https://docs.mongodb.com/manual/reference/change-events/#replace-event
Finally, I need to get updateDescription.updatedFields
from Mongo change streams
and that's why I want MongoSinkConnector
to update documents with update
operationType.
Is there any way to do it with update
operationType by MongoSinkConnector
?
from kafka-connect-mongodb.
@hpgrahsl
It is a good question! I think that my explanation was not enough.
There are a few reasons:
- First of all, I need to sink the data anyway from Postgres to MongoDB to reuse them next time from MongoDB. (So I cannot skip this sink process.)
- If I directly source the data from the original Kafka topic, I couldn't guarantee if those data also saved to MongoDB successfully, because it seems like splitting the pipeline in two!(
Kafka --> MongoDB
as well asKafka --> real-time processing application
) That's why I want to consume the data after being successfully saved.(Kafka --> MongoDB --> real-time processing application
) - It's not necessary to depend on something like
Schema Registry
when consuming the data in the application anymore.
from kafka-connect-mongodb.
Related Issues (20)
- Log is not getting appended in File HOT 3
- Are there any configuration setting to get fullDocument Json only? HOT 1
- Replace strategy setting BsonId HOT 3
- support new compound shard key handling of Debezium Source Connector
- Version 1.4.0 on Confluent Hub HOT 3
- Avro schema "TopicNameStrategy" error HOT 1
- Removed records are put back again immediately. HOT 4
- where is the jar? HOT 1
- Use two WriteModel Strategies in one config? HOT 5
- Cannot convert Record to BSON HOT 2
- Issue with MongoDB sink connector, Records inserted to MongoDB are in the form of CDC structure instead of a mongo document structure HOT 1
- Can I customize Kafka topic from MongoSourceConnector? HOT 1
- Error with shard mongoDB
- Error: `operationType` field is doc is missing
- Post Processor Chain not working HOT 3
- MongoDB Sink Connector Fails with NullPointerException HOT 1
- Question: has this been removed from Confluent Hub? HOT 2
- Generating unwanted key for Delete HOT 1
- Write model strategy - optional upsert HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kafka-connect-mongodb.