StringType defaults to TEXT which is varchar(256) about spark-redshift

[JoshRosen]

Hi @eprochasson,

This is a known behavior; see #29. We have (partially) addressed this by allowing the size of individual string columns via Spark SQL's column metadata feature: https://github.com/databricks/spark-redshift#configuring-the-maximum-size-of-string-columns. However, this approach's ease-of-use is limited by Spark SQL's limited support for column metadata configuration. I'd love to have your feedback on whether our existing column-metadata-based approach addresses your needs or whether you have any suggestions for how we can make this easier to configure (or better-documented).

[eprochasson]

I'll look into it. Sorry I missed it!

[eprochasson]

It works beautifully. Minor issue in the documentation is that df.withColumn returns a new DataFrame and does not modify the existing one (so that snippet won't work).

I wrote a short recipe for it here https://gist.github.com/eprochasson/d1feb24eb36eb882d580

You're very welcome to integrate it to the README if you feel it's worth it.

Thanks a ton.

[JoshRosen]

If the current README instructions don't work as written then we should update them. Want to submit a PR and I'll review this week?

[eprochasson]

Indeed it doesn't (it's actually not even syntactically correct, it's missing a parenthesis). I'll submit a PR as soon as I have a minute.

[robbyki]

Apologies for misunderstanding this issue but I'm going through several resources to try and understand how to maintain my schema created outside of spark and then just truncating my tables from spark followed by writing with a savemode of overwrite. My problem exactly this issue with respect to my db netezza failing when it sees spark trying to save a text data type so I then have to go specify in my new jdbc dialect to use varchar(n) which does work however that just replaces all of my varchar columns (different lengths for different columns) with whatever I specified in my dialect which is not what I want. How can I just have it save the TEXT as varchar without specifying a length in the custom dialect?

[yobdoy]

I was having this same problem, this answer worked for me:

Setting this config change before making the write command solved it for us: spark.conf.set("spark.sql.legacy.charVarcharAsString", True)