Coder Social home page Coder Social logo

Comments (2)

ricellis avatar ricellis commented on July 28, 2024

Rather than a sink connector configuration option I think providing (and documenting) a SMT that is able to insert the message key into the message value as an _id field is the way to go.

from cloudant-kafka-connector.

ricellis avatar ricellis commented on July 28, 2024

Looking at the complexity that is coming out in #82 from trying to handle all the variations of inserting an _id into different types of messages and associated schema updates I think we need to re-think how we handle this.
One option would be to:

  1. Modify the ConnectRecordMapper to check for the presence of a custom header on the record and use the value of that header as the ID.
  2. Use the org.apache.kafka.connect.transforms.HeaderFrom to convert a key to the header, avoiding the need for us to have any custom SMT.

For the header, we would ensure the StringConverter was used as we only want strings for IDs. We would need to document the name of the header and the expectation that the value is a string. (this would cause errors for things that cannot convert cleanly to strings, inline with general Kafka behaviour on converters).

We would overwrite existing _id in the JSON blob before writing to Cloudant (with a warning).

from cloudant-kafka-connector.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.