Coder Social home page Coder Social logo

Comments (7)

itinycheng avatar itinycheng commented on August 20, 2024

@denganming
您说的确实是个问题,但加sign并不是每个引擎都支持,所以得有个比较普遍适用的方案;
最近我自己加了个update_after的转update,insert 或直接ignore的配置;
delete还没想太多,如果合适方案欢迎PR;

from flink-connector-clickhouse.

walkhan avatar walkhan commented on August 20, 2024

@itinycheng update策略,在1.14代码中看到是有的,实测不支持
image

那如何做到实时更新(delete和update)

from flink-connector-clickhouse.

itinycheng avatar itinycheng commented on August 20, 2024

sink.update-strategy的代码我并没合并到release-1.14分支;
不建议在Flink端生成类似update的操作,当前connector对update/delete支持也并不完善;
最好结合ClickHouse端表引擎来解决实际问题,单靠connector不太行;

from flink-connector-clickhouse.

walkhan avatar walkhan commented on August 20, 2024

sink.update-strategy的代码我并没合并到release-1.14分支; 不建议在Flink端生成类似update的操作,当前connector对update/delete支持也并不完善; 最好结合ClickHouse端表引擎来解决实际问题,单靠connector不太行;

@itinycheng ClickHouse我用了ReplacingMergeTree引擎,如果上游数据更新或者删除了,偶尔还是会出现数据多的问题。那么在1.15中sink.update-strategy能否规避这个问题呢?

from flink-connector-clickhouse.

itinycheng avatar itinycheng commented on August 20, 2024

sink.update-strategy的代码我并没合并到release-1.14分支; 不建议在Flink端生成类似update的操作,当前connector对update/delete支持也并不完善; 最好结合ClickHouse端表引擎来解决实际问题,单靠connector不太行;

@itinycheng ClickHouse我用了ReplacingMergeTree引擎,如果上游数据更新或者删除了,偶尔还是会出现数据多的问题。那么在1.15中sink.update-strategy能否规避这个问题呢?

Connector本身没办法保证end-to-end exactly once,具体数据多的原因给不出太多建议,最好根据自己场景排查下,如果是因为没有及时optimize table,可以另一个任务定时对表执行optimize操作;
sink.update-strategy配置只是给了我们个转换update语句的策略而已;

from flink-connector-clickhouse.

leosanqing avatar leosanqing commented on August 20, 2024

我们实现了这个功能,可以选择逻辑删除还是物理删除,逻辑删除支持指定 字段。默认使用 insert 进行行更新

from flink-connector-clickhouse.

itinycheng avatar itinycheng commented on August 20, 2024

我们实现了这个功能,可以选择逻辑删除还是物理删除,逻辑删除支持指定 字段。默认使用 insert 进行行更新

这是个很好的功能,有时间欢迎提PR~

from flink-connector-clickhouse.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.