aastom / dataproc-templates

Dataproc Serverless templates and pipelines for solving simple in-Cloud data tasks

Dataproc Templates

Dataproc templates are an effort to solve simple, but large, in-Cloud data tasks, including data import/export/backup/restore and bulk API operations. The technology under the hood which makes these operations possible is the serverless spark functionality based on Google Cloud's Dataproc.

Google is providing this collection of pre-implemented Dataproc templates as a reference and to provide easy customization for developers wanting to extend their functionality.

Dataproc Templates (Java - Spark)

Please refer to the Dataproc Templates (Java - Spark) README for more information

• CassandraToBigQuery
• CassandraToGCS
• DataplexGCStoBQ(blogpost link)
• GCSToBigQuery (blogpost link)
• GCSToBigTable (blogpost link)
• GCSToGCS (blogpost link)
• GCSToJDBC (blogpost link)
• GCSToSpanner (blogpost link)
• GeneralTemplate
• HBaseToGCS(blogpost link)
• HiveToBigQuery(blogpost link)
• HiveToGCS (blogpost link)
• JDBCToBigQuery (blogpost link)
• JDBCToGCS (blogpost link)
• JDBCToSpanner
• KafkaToBQ (blogpost link)
• KafkaToGCS (blogpost link)
• PubSubToBigQuery (blogpost link)
• PubSubToBigTable (blogpost link)
• PubSubToGCS (blogpost link)
• RedshiftToGCS
• S3ToBigQuery(blogpost link)
• SnowflakeToGCS (blogpost link)
• SpannerToGCS (blogpost link)
• WordCount

Dataproc Templates (Python - PySpark)

Please refer to the Dataproc Templates (Python - PySpark) README for more information

• BigQueryToGCS (blogpost link)
• CassandraToBigquery
• GCSToBigQuery (blogpost link)
• GCSToBigTable(blogpost link)
• GCSToGCS (blogpost link)
• GCSToJDBC (blogpost link)
• GCSToMongo (blogpost link)
• HbaseToGCS
• HiveToBigQuery (blogpost link)
• HiveToGCS (blogpost link)
• JDBCToBigQuery (blogpost link)
• JDBCToGCS (blogpost link)
• JDBCToJDBC (blogpost link)
• MongoToGCS (blogpost link)
• RedshiftToGCS (blogpost link)
• SnowflakeToGCS (blogpost link)
• TextToBigQuery (blogpost link)

Dataproc Templates (Notebooks)

Please refer to the Dataproc Templates (Notebooks) README for more information

• HiveToBigQuery (blogpost link)
• MsSqlToBigQuery
• MySQLToSpanner (blogpost link)
• SQLServerToPostgres
• OracleToBigQuery
• OracleToSpanner (blogpost Link)

Getting Started

1. Clone this repository

    git clone https://github.com/GoogleCloudPlatform/dataproc-templates.git
   

2. Obtain authentication credentials

   Create local credentials by running the following command and following the oauth2 flow (read more about the command here.

    gcloud auth application-default login
   

   Or manually set the GOOGLE_APPLICATION_CREDENTIALS environment variable to point to a service account key JSON file path.

   Learn more at Setting Up Authentication for Server to Server Production Applications.

Note: Application Default Credentials is able to implicitly find the credentials as long as the application is running on Compute Engine, Kubernetes Engine, App Engine, or Cloud Functions.

3. Executing a Template

   Follow the specific guide, depending on your use case:

   • Dataproc Templates (Java - Spark)
   • Dataproc Templates (Python - PySpark)

Flow diagram

Below flow diagram shows execution flow for Dataproc Templates:

Contributing

See the contributing instructions to get started contributing.

License

All solutions within this repository are provided under the Apache 2.0 license. Please see the LICENSE file for more detailed terms and conditions.

Disclaimer

This repository and its contents are not an official Google Product.

Contact

Share you feedback, ideas, thoughts feedback-form

Questions, issues, and comments should be directed to [email protected]