Comments (7)
Auto dbt-retry with exponential backoff would be nice to have as a ways of combating this issue from the dbt side of things. Manually triggering the retry will often work, but in an instance this AM, had to dbt retry (from dbt cloud) the same job 4 times before it went through.
from dbt-bigquery.
Adding comment here: this is not something that we can consistently reproduce, however, we are noticing GCP queries that take very little slot time (i.e. 0.03s of slot time) will take 3+ mins to complete.
We are using interactive mode to submit these queries.
We also can look at query hashes for long running queries, and see that those some query hashes do not take nearly as much time to run in prior runs.
from dbt-bigquery.
Noticed the same issue starting with our May 19, 2024, 11:45 PM EDT "production" run. Same issue on our morning run.
The first two times it happened, we were able to recover automatically (we use airflow to get exponential backoff as @chase-jones mentioned above.)
If an automated retry function is added, it would be friendly to add an option to run the retry with --select result:error
from dbt-bigquery.
For diligence, another thing I noticed was that there was a new minor release of the BigQuery Python client four days ago: 3.23.0
.
However, I don't see anything there that could affect this, which further leads me to think the issue is with BigQuery itself
from dbt-bigquery.
Yes, this is happening to us as well. We are encountering a timeout issue that we have not encountered previously. We've tried to rerun the job and it's failing at a different stage each time. Our pipeline ran fine all weekend and unexpectedly started failing this morning with no rhyme or reason.
from dbt-bigquery.
Do you also see the same timeout error when you run the query interactively in BigQuery console? If so, I would recommend creating a ticket directly with Google Cloud Support team.
from dbt-bigquery.
We ran the same dbt pipeline using GCP Cloud Build and it executed successfully without any slowdowns
from dbt-bigquery.
Related Issues (20)
- BigQuery tags do not work HOT 1
- [Bug] State modified does not pick up changes to policy tags
- [Bug] dbt grant doesn't work for clone
- [Feature] Improvement in data processed/cost incurred in insert_overwrite method HOT 4
- [Release Improvements] Refresh workflows for the `pyproject.toml` migration
- [Bug] persist_docs not working for seeds HOT 2
- Import relevant pytest(s) for cross-database `cast` macro
- [Bug] `docs generate` appears to be returning no table metadata when run with the `--no-compile` option HOT 1
- [Tech Debt] `test_dbt_debug` is not dropping its test schema
- [Bug] `docs generate` does not find all schemas when there are more than 10K schemas
- Cross-database `date` macro
- [Bug] incremental run with __dbt_tmp table does not log the real bytes_billed in run_results.json HOT 1
- [Bug] Requested entity not found when writing a Python model to BigQuery
- [Feature] support copy multiple tables in parallel using copy_partitions
- [Feature] Implement batch metadata freshness using `INFORMATION_SCHEMA.TABLE_STORAGE` instead of `client.get_table` HOT 1
- BigQuery authorized dataset HOT 1
- [Bug] --empty flag not working on Pseudo-columns HOT 3
- [Feature] Support overriding `grant_access_to`
- [Bug] Incremental models don't update new elements in struct column
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dbt-bigquery.