:timeout - timeout in seconds, checks on bucket for new export from BQ (if new export does not exist after timeout exceeded, then continue)
Bigquery fields are converted to text:
if your :target-pg-table already exists, it will try to convert value to its column type
If that table doesn’t exist it will be created and every field will be imported as text
Export?
You can set :export? key to false if you don’t want bq2pg to export your data from Bigqeury to Google cloud storage.
This value can be configured in main config file under :export? key or as BQEXPORT env variable.
Import?
You can set :import? key to false if you don’t want bq2pg to import your data into Postgresql database.
Keep in mind: bq2pg creates record for every last integration in .last_import.edn so if you have already imported some existing export it won’t be imported twice.
it means that it won’t be imported until export to the target folder finishes.
This value can be configured in main config file under :import? key or as PGIMPORT env variable.
Stream GZIP files
Bigquery data are exported to a batch of GZIP files which are loaded one by one and save space on your disk.
Examples
Example 1 - easy and quick
Download bq2pg-1.jar from releases
create your integration.edn like this:
{:export?true:import?true:gcs-name"my-bucket":gcs-folder"bq2pg":db {:dbtype"postgres":dbname"postgres":user"postgres":host"somehost":port5432:password"my_password"}
:integrations [{:name"test":query"SELECT * FROM `project.test.pokemons` LIMIT 1000":location"EU":target-pg-table"public.pokemons":method"replace":timeout120}]}
run java -jar bq2pg.jar integration.edn
Example 2 - clojure developer, installation
run ./install.sh
bq2pg will be compiled and you can run it by ~/bin/bq2pg
or add $HOME/bin to your PATH.
Example 3 - clojure developer, interactive use
run repl and switch to user namespace
you can see example configuration + integration you must update before you run integrate-dev!.