Comments (5)
@pcorbel does target-redshift
/target-postgres
create TIMESTAMP WITHOUT TIME ZONE
columns somewhere?
Currently, the thinking is that the only supported tables are those that the target
created.
from target-redshift.
@AlexanderMann
does target-redshift/target-postgres create TIMESTAMP WITHOUT TIME ZONE columns somewhere? --> No
I agree that only supported tables are those that the datamill-co/target-redshift
created.
However, as of today, we need the totality of the tables in the schema to be created by datamill-co/target-redshift
.
We can't have:
- schema_a/table_stitch_with_timestamp_without_time_zone
- schema_a/table_manually_created_with_timestamp_without_time_zone
- schema_a/table_singer_datamill
But only:
- schema_a/table_stitch_with_timestamp_without_time_zone
- schema_a/table_manually_created_with_timestamp_without_time_zone
- schema_b/table_singer_datamill
from target-redshift.
@pcorbel do you know whether you're getting these errors on just running the target
, or is it when a stream you're processing tries to persist to a table created by some other means?
ie, is it:
- stream
FOO
tries to persist to remote tablefoo
a table created and maintained bytarget-redshift
- another table,
bar
, exists in the remote schema which has been created by another process
- another table,
- stream
FOO
tries to persist to remote tablefoo
which exists in the remote schema which has been created by another process
I ask, because I think the guidance has been that target
s should probably be run in their own schemas and if they happen to work alongside other processes, that's great, but not a given. For target-redshift
specifically, we've turned down trying to support tables not created and maintained by target-redshift
.
The reason for this is that the stores a tonne of metadata into the remote table's COMMENT
and has to use that for most operations.
from target-redshift.
@AlexanderMann It is the 1 case you mentioned.
My usecase is that I would like to migrate some tables from Stitch to in-house extraction.
To be able to do it in a transparent and smooth way, I would like my end users to use the same table name, in the the same schema name. It is just the extraction that would be different.
from target-redshift.
Fascinating. So @pcorbel to be clear, you are not looking to mix and match maintenance of tables between target-redshift
and something else, but rather are looking to simply use it inside a schema which has other things in it?
If yes, I think we should identify where the code is blowing up by pulling in other stuffs first, as I can easily see this becoming a game of whack-a-mole with types etc.
from target-redshift.
Related Issues (19)
- Automatic varchar widening HOT 1
- Support IAM Roles
- Minor quality of life improvement HOT 1
- Add support for Postgres array types HOT 5
- invalid_records_detect in config not being respected HOT 5
- Add support for patternProperties HOT 7
- Performance: Use `CREATE TABLE ... LIKE ...` HOT 1
- connection closing before write HOT 5
- Memory consumption increase HOT 2
- Support for Redshift types like NUMERIC(20, 0) HOT 4
- Integrate with RedShfit Spectrum HOT 6
- Table comments breaking target HOT 3
- singer target-redshift, not able to connect to Redshift due to issue with SSL HOT 1
- Disable COMPUPDATE and STATUPDATE during copy for performance?
- CRITICAL cursor already closed / connection already closed HOT 2
- Column Sizing
- Meltano Recharge Issue: Datamill
- ImportError: cannot import name 'Mapping' from 'collections'
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from target-redshift.