Comments (7)
Hi @eprochasson,
This is a known behavior; see #29. We have (partially) addressed this by allowing the size of individual string columns via Spark SQL's column metadata feature: https://github.com/databricks/spark-redshift#configuring-the-maximum-size-of-string-columns. However, this approach's ease-of-use is limited by Spark SQL's limited support for column metadata configuration. I'd love to have your feedback on whether our existing column-metadata-based approach addresses your needs or whether you have any suggestions for how we can make this easier to configure (or better-documented).
from spark-redshift.
I'll look into it. Sorry I missed it!
from spark-redshift.
It works beautifully. Minor issue in the documentation is that df.withColumn
returns a new DataFrame and does not modify the existing one (so that snippet won't work).
I wrote a short recipe for it here https://gist.github.com/eprochasson/d1feb24eb36eb882d580
You're very welcome to integrate it to the README if you feel it's worth it.
Thanks a ton.
from spark-redshift.
If the current README instructions don't work as written then we should update them. Want to submit a PR and I'll review this week?
from spark-redshift.
Indeed it doesn't (it's actually not even syntactically correct, it's missing a parenthesis). I'll submit a PR as soon as I have a minute.
from spark-redshift.
Apologies for misunderstanding this issue but I'm going through several resources to try and understand how to maintain my schema created outside of spark and then just truncating my tables from spark followed by writing with a savemode of overwrite. My problem exactly this issue with respect to my db netezza failing when it sees spark trying to save a text data type so I then have to go specify in my new jdbc dialect to use varchar(n) which does work however that just replaces all of my varchar columns (different lengths for different columns) with whatever I specified in my dialect which is not what I want. How can I just have it save the TEXT as varchar without specifying a length in the custom dialect?
from spark-redshift.
I was having this same problem, this answer worked for me:
Setting this config change before making the write command solved it for us:
spark.conf.set("spark.sql.legacy.charVarcharAsString", True)
from spark-redshift.
Related Issues (20)
- spark-redshift generate corrupted Parquet files
- Caused by: org.jets3t.service.impl.rest.HttpException: 400 Bad Request--> py4j.protocol.Py4JJavaError: An error occurred while calling o42.save. : org.apache.spark.sql.catalyst.errors.package$TreeNodeException: execute, tree:
- Any plan to support Scala 2.12? HOT 2
- tempdir is not being cleaned up after successful load to redshift HOT 1
- Spark SQL module compatability
- loading credentials from AWS_WEB_IDENTITY_TOKEN_FILE
- How are temporary files created in S3 when calling the driver? HOT 1
- Support unload to tempdir with specified KMS key
- Error when saving a dataframe to Redshift (java.lang.ArrayStoreException: java.lang.invoke.SerializedLambda) HOT 1
- [Question] Don't know how to save StructField
- Getting SignatureDoesNotMatch Error
- error while writing to Redshift: Cannot merge com.databricks.spark.redshift.HashSetAccumulator
- Pushdown filter not working with Varchar datatype HOT 1
- Is it possible to add columns to redshift table automatically? #1256
- Redshift support binary type since december 2021 HOT 1
- Databricks read from redshift first selected column returns wrong value if quotes used
- Java.lang.IllegalArgumentException: Wrong FS s3://bucket/folder/manifest.json -expected s3a://bucket HOT 1
- Spark 3.3 support HOT 1
- py4j.protocol.Py4JJavaError: An error occurred while calling o96.save.
- java.sql.SQLException: [Amazon][JDBC](11380) Null pointer exception.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from spark-redshift.