Comments (7)
@denganming
您说的确实是个问题,但加sign并不是每个引擎都支持,所以得有个比较普遍适用的方案;
最近我自己加了个update_after的转update,insert 或直接ignore的配置;
delete还没想太多,如果合适方案欢迎PR;
from flink-connector-clickhouse.
@itinycheng update策略,在1.14代码中看到是有的,实测不支持
那如何做到实时更新(delete和update)
from flink-connector-clickhouse.
sink.update-strategy
的代码我并没合并到release-1.14分支;
不建议在Flink端生成类似update的操作,当前connector对update/delete支持也并不完善;
最好结合ClickHouse端表引擎来解决实际问题,单靠connector不太行;
from flink-connector-clickhouse.
sink.update-strategy
的代码我并没合并到release-1.14分支; 不建议在Flink端生成类似update的操作,当前connector对update/delete支持也并不完善; 最好结合ClickHouse端表引擎来解决实际问题,单靠connector不太行;
@itinycheng ClickHouse我用了ReplacingMergeTree引擎,如果上游数据更新或者删除了,偶尔还是会出现数据多的问题。那么在1.15中sink.update-strategy能否规避这个问题呢?
from flink-connector-clickhouse.
sink.update-strategy
的代码我并没合并到release-1.14分支; 不建议在Flink端生成类似update的操作,当前connector对update/delete支持也并不完善; 最好结合ClickHouse端表引擎来解决实际问题,单靠connector不太行;@itinycheng ClickHouse我用了ReplacingMergeTree引擎,如果上游数据更新或者删除了,偶尔还是会出现数据多的问题。那么在1.15中sink.update-strategy能否规避这个问题呢?
Connector本身没办法保证end-to-end exactly once
,具体数据多的原因给不出太多建议,最好根据自己场景排查下,如果是因为没有及时optimize table,可以另一个任务定时对表执行optimize操作;
sink.update-strategy
配置只是给了我们个转换update语句的策略而已;
from flink-connector-clickhouse.
我们实现了这个功能,可以选择逻辑删除还是物理删除,逻辑删除支持指定 字段。默认使用 insert 进行行更新
from flink-connector-clickhouse.
我们实现了这个功能,可以选择逻辑删除还是物理删除,逻辑删除支持指定 字段。默认使用 insert 进行行更新
这是个很好的功能,有时间欢迎提PR~
from flink-connector-clickhouse.
Related Issues (20)
- flink-connector-clickhouse里面的sink怎么使用?有例子吗? HOT 2
- java.lang.ClassCastException: class java.lang.Boolean cannot be cast to class java.lang.Number HOT 7
- I cannot use OPTION 'properties.*'. HOT 4
- Execution default of goal io.github.zentol.japicmp:japicmp-maven-plugin:0.16.0_m325:cmp failed: version can neither be null, empty nor blank HOT 2
- Add Apache Listen to File hard
- sink exception e.flink.connector.clickhouse.internal.ClickHouseBatchOutputFormat HOT 3
- how to pack a fat jar HOT 1
- 有人编译成功么,可以分享下么 HOT 1
- 使用flink 版本是1.19 flink cdc 时报错了,应该是序列化的问题 Caused by: java.lang.ClassNotFoundException: org.apache.flink.connector.clickhouse.internal.ClickHouseBatchOutputFormat HOT 4
- Support URLs with query params HOT 3
- Add issues And pull template to project
- [Bug]: Unit test error cause CI building failure
- [Bug]: not set Clickhouse userName and password have npe
- [Bug]: UInt Type Not Supprot to Read Or Write HOT 2
- [Feature]: Add Flink Clickhouse QuickStart For Read Or Write Data
- [Feature]: Support Unit5 Test For Clickhouse Connector
- flink1.17.2 clickhouse catalog集成kafka 报错:创建kafka表时报错 HOT 2
- [Bug]: 经常出现刷新异常 HOT 4
- [Enhance]: Ensure Consistency of Database and Parameters in Newly Created ClickHouse JDBC Connections HOT 2
- [Bug]: 发现除了insert,update和delete操作都无法正常同步
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from flink-connector-clickhouse.