Comments (1)
Hey @nitin-kakkar, thank you for open this issue.
I've added compression options for Pandas.to_parquet and prepare the structures to do the same with Pandas.to_csv in the future.
But for now the second is blocked by a Pandas limitation.
How we aim to write compressed files directly to S3, we must wait until Pandas unlock the option of write compressed files in memory (instead of disk).
from aws-sdk-pandas.
Related Issues (20)
- Breaking change in awswrangler.dynamodb.read_items with version 3.5.0 HOT 1
- Iceberg Overwrite partition mode HOT 1
- dynamodb.read_items key_condition_expression and filter_expression used together throws an error for version 3.5.0 HOT 2
- Postgres Reserved Keyword Error HOT 2
- Improve DynamoDB test coverage
- Pyarrow 15.0.0 breaking changes HOT 6
- Lamba layers do not contain optional modules HOT 2
- v3.5 doesn't run on Glue Ray HOT 1
- Parquet S3 Client Side Encryption HOT 3
- Improve chunk size parameter validation for PostgreSQL queries HOT 2
- iam policies required for to_iceberg api HOT 4
- Pyarrow 15 compatability HOT 2
- awswrangler.athena.to_iceberg not supporting to synchronous/parallel lambda instances. HOT 3
- Repeated s3.to_parquet writes with index=True raise InvalidSchemaConvergence on s3.read_parquet with validate_schema=True HOT 4
- Writing to Athena Iceberg Table: Extra System Generated Directory in s3 HOT 2
- aws-wrangler creating new versions for iceberg table and creating metadata for each new version HOT 4
- awswrangler.s3.read_parquet_metadata does not support large_string datatype HOT 1
- Limit MaxResults for athena.list_query_executions
- to_parquet: mismatch between path and projection_storage_location_template HOT 1
- Metadata missing timezone breaks `s3.read_parquet` in awswrangler 3 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aws-sdk-pandas.