A Singer target that writes data to Google BigQuery.
target-bigquery works together with any other Singer Tap to move data from sources like Braintree, Freshdesk and Hubspot to Google BigQuery.
(originally found in the Google API docs)
- Use this wizard to create or select a project in the Google Developers Console and activate the BigQuery API. Click Continue, then Go to credentials.
- On the Add credentials to your project page, click the Cancel button.
- At the top of the page, select the OAuth consent screen tab. Select an Email address, enter a Product name if not already set, and click the Save button.
- Select the Credentials tab, click the Create credentials button and select OAuth client ID.
- Select the application type Other, enter the name "Singer BigQuery Target", and click the Create button.
- Click OK to dismiss the resulting dialog.
- Click the Download button to the right of the client ID.
- Move this file to your working directory and rename it client_secrets.json.
Create a file called config.json in your working directory, following config.sample.json. The required parameters are the project name project_id, the dataset name dataset_id, and table name table_id.
First, make sure Python 3 is installed on your system or follow these installation instructions for Mac or Ubuntu.
target-bigquery can be run with any Singer Tap, but we'll use tap-fixerio - which pulls currency exchange rate data from a public data set - as an example.
These commands will install tap-fixerio and target-bigquery with pip and then run them together, piping the output of tap-fixerio to target-bigquery:
› pip install target-bigquery tap-fixerio
› tap-fixerio | target-bigquery -c config.json
INFO Replicating the latest exchange rate data from fixer.io
INFO Tap exiting normallyIf you're using a different Tap, substitute tap-fixerio in the final command above to the command used to run your Tap.
It is recommended to use target-bigquery with a service account.
- Download the client_secrets.json file for your service account, and place it on the machine where
target-bigquerywill be executed. - Set a
GOOGLE_APPLICATION_CREDENTIALSenvironment variable on the machine, where the value is the fully qualified path to client_secrets.json
It should be possible to use the oAuth flow to authenticate to GCP as well:
target-bigquerywill attempt to open a new window or tab in your default browser. If this fails, copy the URL from the console and manually open it in your browser.- If you are not already logged into your Google account, you will be prompted to log in.
- If you are logged into multiple Google accounts, you will be asked to select one account to use for the authorization.
- Click the Accept button to allow
target-bigqueryto access your Google BigQuery table. - You can close the tab after the signup flow is complete.
The data will be written to the table specified in your config.json.
Copyright © 2018 RealSelf, Inc.
![mend-for-github-com[bot] avatar](https://avatars.githubusercontent.com/in/30796?v=4 "mend-for-github-com[bot]")
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent