Comments (1)
We tested the branch (pip install git+https://github.com/dbt-labs/dbt-bigquery.git@batch-metadata-benchmarking#egg=dbt-bigquery
) and can confirm a fantastic improvement!
1156 sources completed in 35 seconds. I ran it twice more and time was very similar. Current time spans between 4 minutes and 12 minutes due to query volatility - but new solution reduces to 1 query which not only is a massive speed up but a huge reduction in volatility. This is a big win.
The only thing I've not yet fully validated is the accuracy of the reported time from the INFORMATION_SCHEMA table. BQ docs state that "The data in the INFORMATION_SCHEMA.TABLE_STORAGE view is not kept in real time, and updates are typically delayed by a few seconds to a few minutes". We use the source_status:fresher+
selector in our build jobs so the impact of this is we might not catch things that have changed immediately but they'd be picked up by our subsequent run - so potentially a small latency impact but a reasonable price for the improvement.
from dbt-bigquery.
Related Issues (20)
- [Bug] many timeouts with `priority: interactive` config HOT 7
- [Bug] Incremental models don't update new elements in struct column
- [Regression] `docs generate` raises error when querying dataset with BQ ML models HOT 1
- dbt bigquery 1.3 Flake8 is failing a mypy check
- [Bug] Cannot parse partition config when specified in property YAML file HOT 2
- [Test] testing workflow
- [Tech Debt] Run all python model tests in sequence across runs
- [Feature] Retrieving and using table/column descriptions from BigQuery
- [Bug] Using `grants` on `materialized_view` raises an error
- [Bug] Build verification is failing HOT 1
- [Bug] Updating partitioning/clustering on a table self referencing itself will fail and delete the table
- [Bug] Dataproc (Python models) do not retry polling on 5xx errors HOT 1
- [Bug] Data mix-up in struct with enforced contracts
- [Bug] Nested record field is not quoted and causes build error when contract enforced enabled. HOT 1
- [Bug] `dbt show` integration test using JSON struct is failing for `1.8`
- Update the build check workflow
- [Bug] spark_catalog requires a single-part namespace in dbt python incremental model
- [Bug] Jobs timing out early regardless of job_execution_timeout_seconds
- [Bug] Bug in partitions_match method causing tables to be dropped for full refresh even though partitioning spec did not change
- [Bug] is_incremental always return False when use Project Number instead of Project ID HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dbt-bigquery.