amygdala / gae-dataflow

An example that shows how to periodically launch a Dataflow analytics pipeline from GAE Flex, that reads from Datastore.

Running Dataflow jobs from Google App Engine

This directory contains two different examples that show how you can run Cloud Dataflow pipelines from App Engine apps, as a replacement for the older GAE Python MapReduce libraries, as well as do much more.

The examples show how to periodically launch a Python Dataflow pipeline from GAE, to analyze data stored in Cloud Datastore; in this case, tweets from Twitter.

The example in sdk_launch shows how to launch Dataflow jobs via the Dataflow SDK. This requires the use of an App Engine Flex service to launch the pipeline.

The example in job_template_launch shows how to launch Dataflow jobs via job Templates. This can be done using only App Engine Standard. Prior to deploying the app, you create a pipeline template (via your local command line, in this example) for the app to use.

Contributions

Contributions are not currently accepted. This is not an official Google product.

gae-dataflow's Issues

[contribution] move to KMS from hard-coded credentials

I know you state no contributions, hence me raising an issue before proposing a change. :)

Would you accept a contribution that updates credentials to be read from KMS instead of environment variables?

Cheers,
-d