Comments (7)
Hi @bestmike007. According to our discussion today, I've created a hotfix for your case. The code is in branch wkx/hotfix-cdc-source-int256, which is based on the most recently released v1.8. You can get the image with docker pull ghcr.io/risingwavelabs/risingwave:git-cb2561e7d4ab1cffa8e8f06b510df4e404f4b603
Here are some notes worth mentioning:
- The implemented solution adapts the second approach discussed earlier. It requires a simple change when creating a table: modifying the field type from numeric to rw_int256. For example, if the table in pg is
create table local_numeric(id int PRIMARY KEY, num numeric)
, then you should create table in Risingwave withCREATE TABLE local_numeric (id int,num rw_int256,PRIMARY KEY (id)) WITH ( ...
. - Currently, this solution is specific to PostgreSQL CDC, resembling a workaround rather than a robust feature, which is why it's not yet PR towards main branch.
- The fix contains a convertion for
decimal
of rust-postgres to RW'srw_int256
. In order to achieve this, I utilized another repo to first convertpg-decimal
to str and then convert str torw_int256
. The repo has not been actively maintained for some time, so this fix may be buggy. I'll continue to look for better solution.
Please let us know, if there's any further problem meet.
The issue should be open until we find a good general solution and merged it into main branch.
from risingwave.
what would be the solution?
@lmatz I vote for 2, which requires less storage
from risingwave.
what would be the solution?
- let users use
varchar
to ingestnumeric
first and then cast toint256
- or allow users to map from
numeric
toint256
directly
from risingwave.
Thanks @stdrc
This issue was initially discussed on Slack: https://risingwave-community.slack.com/archives/C03BW71523T/p1711419543170449
It'll be even better if it can be configurable to cast numeric fields into int256/uin256(if you have it in the future)/double/varchar.
from risingwave.
int256/uin256(if you have it in the future)/double/varchar.
Agree. We may also support varchar
and double
for different use cases.
from risingwave.
And I haven't tested yet, can you sink rw_int256 fields into postgres numeric fields?
from risingwave.
Additional note: special values nan
/inf
/-inf
may not work properly: #16395
from risingwave.
Related Issues (20)
- EOWC: close `RANGE`/`SESSION` window with watermark
- `ALTER TABLE` will refresh previous snapshot values for absent cells
- bug: fail to load workload identity token on azure environment
- cherrypick feat(udf): add metric of UDF memory usage (#16922) to branch release-1.9 HOT 1
- when using external schema, `struct`'s fields are not shown in `describe <table>` HOT 2
- Division by zero exception HOT 12
- e2e test time increased from 15min to 20min in ci HOT 1
- Performance lost after using `BTreeMap` for WITH properties
- hint user when column not found HOT 1
- bug: CI failure: integration test (madsim): progress not within bounds 0.9
- bug(main-cron): pulsar source check: gRPC request to meta service failed: Unknown error HOT 3
- feat: expose dedicated source for table in system catalog rw_sources
- Reclaim space more aggresively for table with vnode table watermark specificed (table with range delete)
- system table for source health status HOT 2
- SSL configurations is not supported for schema registry HOT 3
- Don't let jni_core depends on (the whole) storage crate HOT 2
- dep: try to use either aws-lc-rs or ring
- Rework ci labels
- reorganize directories at the root of the repo HOT 1
- error in metabase: function has_any_column_privilege does not exist HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from risingwave.