Comments (5)
@robmoore yep. Thats the intention. It won't however store it or automagically do anything with it, so managing the state and passing the previous state into the next invocation is still the user's responsibility, not target-postgres
'.
from target-postgres.
Is there any guidance on why this target doesn't support actually doing something with STATE messages? Is it that it's hard, or not needed by the original authors? I would have figured everyone needed to store STATEs somewhere, and I feel like the big database all the data is going into is as good a place as any. @AlexanderMann perhaps you could comment -- do you ever want to support storing state messages somehow in this target?
from target-postgres.
Hey @airhorns, sorry, was on vacation then got swamped with a client!
So some easy questions then the meaty one:
Do you ever want to support storing state messages somehow in this target?
Presently, no. I don't see the benefit to this as I think it muddies the presently single concern of a target of "I get information, I persist it" and instead would make it need to somehow understand state which I think is typically only useful for the tap right?
The meaty question:
any guidance on why this target doesn't support actually doing something with STATE messages? Is it that it's hard, or not needed by the original authors?
DataMill originally crafted a runner for Singer tap
s and target
s to handle state
and whatnot. This took state out of the input and did some voodoo on it to persist it remotely as you can imagine wanting to persist STATE
messages to something like s3, Postgres, etc. which is different than the target.
This is to say, we didn't need it and felt that ignoring it to start was simpler and got something "pretty-good" working much more quickly.
Conclusion
I think your PR of #130 handles this in a more robust way for all folks if I'm not mistaken? I'd propose that being the conclusion to this issue.
If folks have other suggestions, I'm definitely up for hearing it!
(Also thanks everyone in here for contributing, super helpful)
from target-postgres.
@airhorns Will the changes in #130 output state for use by the tap on future runs?
from target-postgres.
Addressed in v0.1.9
from target-postgres.
Related Issues (20)
- The target ignores the common Postgres env vars.
- Schema error, can you guide me? HOT 5
- Tag version doesn't match the current version HOT 5
- Error with the target when paired with zendesk tap HOT 7
- Comment on pre-existing table crashes target-postgres
- Add option to autocreate schema if not present? HOT 5
- Performance issues loading 5M rows from CSV file HOT 1
- Can't drop tables with dependent objects during table version swap HOT 2
- KeyError attempting tap-csv to target-postgresql HOT 1
- crashing when in use HOT 5
- Update dependencies
- Allow setting loglevel through config file HOT 2
- switch CSV writer to the unix dialect so that newline chars are always quoted
- document how to activate a new version of postgres-singer-io tap schema - `key_properties` change detected
- Update psycopg dependancie
- Website target pip install documentation issue HOT 1
- How do I add new columns to existing table automatically? HOT 5
- pip install singer-target-postgres error on psycopg2 package on Python 3.8 virtualenv
- invalid_records_* doesn't ignore psycopg2 encoding errors HOT 1
- Broken Pipe - Symbol not found: _PQbackendPID HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from target-postgres.