Coder Social home page Coder Social logo

tap-s3-json's People

Contributors

dcereijodo avatar

Stargazers

avatar avatar

Watchers

avatar

tap-s3-json's Issues

Discontinue JsonPaths

The jsonpaths funtionality to flatten extracted data can be better handled on a subsequent processing stage. For example, with jq utility one can parse and filter JSON in a numberless amount of ways.

Simple example: Extract address data from an Order in it's own records.

tap-s3-json --tap.prefix PMT_ORDER/ORDER_CREATED --date 2019-06-19 --tap.limit 100 | jq '{address: .payload.user.address, billing: .payload.user.billing_address}'

We should not have this functionality inside the tap.

Support an 'date' parameter

Usually a tap will be used to pull data on a daily basis. We need to have the notion of an execution date so we can tap only for data related to the current execution (for example the last day)

Support 'recursive' mode

Currently tapping on an S3 prefix does not recurse to nested objects. This should be possible by setting a parameter 'recurse' in the config.

Passing an empty perfix leaves the tap zombie

Whenever a empty prefix is passed to the tap, the tap will trigger an error as expected, but the it won't shut down with exit status 1.

target/universal/stage/bin/tap-s3-json --tap.bucket_name ryft-public-sample-data --tap.prefix "" --region us-east-1

Exception in thread "main" com.pagantis.singer.taps.exceptions.InvalidPrefixException
	at com.pagantis.singer.taps.ObjectKeyUtils.buildS3Prefix(ObjectKeyUtils.scala:10)
	at com.pagantis.singer.taps.ObjectKeyUtils.buildS3Prefix$(ObjectKeyUtils.scala:7)
	at com.pagantis.singer.taps.S3Source.buildS3Prefix(S3Source.scala:30)
	at com.pagantis.singer.taps.S3Source.object_keys(S3Source.scala:45)
	at com.pagantis.singer.taps.S3Source.object_contents(S3Source.scala:57)
	at com.pagantis.singer.taps.TapS3Json$.delayedEndpoint$com$pagantis$singer$taps$TapS3Json$1(TapS3Json.scala:28)
	at com.pagantis.singer.taps.TapS3Json$delayedInit$body.apply(TapS3Json.scala:12)
	at scala.Function0.apply$mcV$sp(Function0.scala:34)
	at scala.Function0.apply$mcV$sp$(Function0.scala:34)
	at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
	at scala.App.$anonfun$main$1$adapted(App.scala:76)
	at scala.collection.immutable.List.foreach(List.scala:388)
	at scala.App.main(App.scala:76)
	at scala.App.main$(App.scala:74)
	at com.pagantis.singer.taps.TapS3Json$.main(TapS3Json.scala:12)
	at com.pagantis.singer.taps.TapS3Json.main(TapS3Json.scala)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.