Comments (7)
Actually, I think the current API does support this, even without #25. The read_gbq function accepts a configuration
keyword argument which is a job configuration resource, so to allow large results one would do either
Standard SQL:
pd.read_gbq(
query,
'my-project',
dialect='standard',
configuration={
'query': {
'destinationTable': {
'projectId': 'my-project',
'datasetId': 'mydataset',
'tableId': 'mytable'
}
}
})
Legacy SQL:
pd.read_gbq(
query,
'my-project',
dialect='standard',
configuration={
'query': {
'allowLargeResults': True,
'destinationTable': {
'projectId': 'my-project',
'datasetId': 'mydataset',
'tableId': 'mytable'
}
}
})
Admittedly this is a bit onerous to do. We may wish to provide a friendlier interface for options such as these.
from python-bigquery-pandas.
Can now do this in open PR by passing it via configuration setting: #25
read_gbq(sql, configuration={"allow_large_results":True})
This uses new google-cloud-python api.
from python-bigquery-pandas.
People have posted some pretty elaborate workarounds on StackOverflow. https://stackoverflow.com/questions/34201923/python-bigquery-allowlargeresults-with-pandas-io-gbq/34203369
from python-bigquery-pandas.
@jasonqng you also have to add destinationTable.
from python-bigquery-pandas.
The updated answer on stack overflow suggests just using dialect='standard' like tswast did but more simply as
pd.read_gbq(query, 'my-super-project', dialect='standard')
and notes AllowLargeResults: For standard SQL queries, this flag is ignored and large results are always allowed. This worked for me but maybe it is not generic.
from python-bigquery-pandas.
I'm glad that worked for you. I believe there may be some size threshold where a destination table is required, even with standard SQL, but perhaps the threshold is larger than it was for legacy SQL.
from python-bigquery-pandas.
Closing, as this can be passed in via the configuration
argument to read_gbq
.
from python-bigquery-pandas.
Related Issues (20)
- tests.system.test_to_gbq: test_dataframe_round_trip_with_table_schema[load_csv-input_df1-None-table_schema1-api_methods1] failed HOT 3
- tests.system.test_to_gbq: test_series_round_trip[load_csv-input_series1-api_methods1] failed HOT 2
- docs: readthedocs build fails due to not compliant to config v2
- Provided Schema does not match Table, Cannot add fields HOT 2
- add `filters` parameter on `read_gbq` which applies when using a table ID as input
- Support JSON data type HOT 1
- use native support for namespace packages
- conda test failing with python 3.12 HOT 3
- tests.system.test_gbq.TestToGBQIntegration: test_upload_data_if_table_exists_replace failed HOT 2
- use faster `query_and_wait` method when available to skip BQ Storage API if all query results are already available
- tests.system.test_to_gbq: test_series_round_trip[load_parquet-input_series2-api_methods2] failed HOT 2
- feat: make google-cloud-bigquery-storage an optional dependency
- nullable numeric column does not handled well HOT 2
- pytest: internal failed HOT 1
- to_dataframe() does not execute the query in BigQuery when installing the package pandas-gbq HOT 1
- tests.system.test_to_gbq: test_dataframe_round_trip_with_table_schema[load_csv-input_df2-expected_df2-table_schema2-api_methods2] failed HOT 2
- Warning: a recent release failed HOT 1
- tests.system.test_gbq.TestToGBQIntegration: test_upload_data_with_newlines failed HOT 2
- restore progress bar for long-running queries
- tests.system.test_gbq.TestReadGBQIntegration: test_timeout_configuration failed HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from python-bigquery-pandas.