datamill-co / singer-runner Goto Github PK
View Code? Open in Web Editor NEWA CLI and library to run Singer Taps and Targets
License: Apache License 2.0
A CLI and library to run Singer Taps and Targets
License: Apache License 2.0
I am using the singer-runner to run a tap and target combination programmatically. However, by executing it that way, a type conversion happens somewhere along the way.
tap_command = './tap-quickbooks-reports/tap_quickbooks_reports/__init__.py'
p = StdInOutPipe()
run_tap(
logger = LOGGER,
tap_command = tap_command,
tap_config = tap_config,
pipe = p
)
I've tried to use the StdInOutPipe as well as the FilePipe.
When running my tap in the CLI it works as desired and writes messages in the desired type. When running it through the code above, a float value will be written (with both pipes) as a String value. When modifying my singer schema it works, but I need a float value in the destination. I was not able yet to figure out where/why the conversion happens. I tested my code again, and the problem seems to be introduced in the singer-runner code (I might use it wrong though).
The output of the resulting field looks like this:
"subt_nat_amount": "38.72"
Any idea why this happens?
I followed the instructions in the readme to install from this repo but had to install these modules as well:
After doing so I was able to run singer-runner
successfully.
The current behavior is that the exit code is always 0
. Scripts that depend on this code to determine whether to exit or continue fail to exit as a result.
This is appears to be specific to the CLI. The main functions do return success / failure. The desired behavior is either to exit 1
explicitly or raise an exceptions, which should cause an exit 1
.
In order to support singer-io/getting-started#50 a webserver must accept HTTP requests and pass them on as WEBHOOK messages.
Proposal for how that could work:
singer-runner webhook-server
command would start an HTTP server.POST /webhooks/:pipe-name/:token
pipe-name
maps to a specific pipe instance ex mailchimp
. When a request is successful, a WEBHOOK message is placed on that pipe.token
maps to a single token passed in the config or environment. Perhaps a function or class could be passed in to allow for more complex auth.run_tap
would need to be changed to allow for input pipes.Create Pypi project and upload to pypi so users can pip install
Support S3 / blobstore lock files with expirations
Running both a tap and target with a memory pipe via CLI
Some taps modify their own config, the most common reason is to update the refresh tokens for APIs that forceable rotate them.
For parts of the config / filepaths, allow timestamps and run IDs to be passed through.
Eg
- s3://my-data-bucket/singer/messages-{timestamp}.txt
- s3://my-data-bucket/singer/messages-{run_id}.txt
Run ID should be a CLI option that defaults to a UUID if not provided
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.