Comments (10)
Hmmm, could it just be S3 eventual consistency? I do often see files missing briefly due to this.
from spark-redshift.
For your temp dir, can you try using s3n
as the scheme ("tempdir", "s3n://spark-temp-dir"
)?
from spark-redshift.
@marmbrus For reference S3 FAQ says
Amazon S3 buckets in all Regions provide read-after-write consistency for PUTS of new objects...
Which should be the case here.
@yhuai The s3n
did work for me, and just to add, only using s3
fails every time. Maybe the README should be updated to reflect that.
from spark-redshift.
@gregrahn I get that all the time. Are you running locally and just adding hadoop-aws
to your classpath as a dependency? You'll probably find that the same code (with s3://
) works fine on EMR. I think you have to actually register s3://
in hdfs-site.xml
or something like that for it to work - not sure exactly, but it's definitely a local setup issue. Not a spark-redshift
issue, just a general S3 configuration drama.
from spark-redshift.
I also ran into some issues with s3://
so I'm wondering whether it would make sense to log a warning if we see that scheme being used.
from spark-redshift.
Have a clarification question. Does the scheme s3
at here mean the original S3 block fileystem? If so, I think it cannot read normal S3 data, right?
from spark-redshift.
/cc @aarondav, do you happen to know the answer to this offhand?
from spark-redshift.
One actionable thing that we can do here is to update the examples to show s3n://
.
from spark-redshift.
@jaley, I think that s3://
worked for you in EMR because EMR aliases that to a s3n://
-like FileSystem rather than the classic S3 block filesystem. See https://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-plan-file-systems.html and http://notes.mindprince.in/2014/08/01/difference-between-s3-block-and-s3-native-filesystem-on-hadoop.html
In a nutshell, I don't think that spark-redshift
will work with the classic S3 block filesystem, but I don't see a super easy way to provide helpful programmatic warnings about this: we can't / shouldn't filter based on the URI since s3://
does work in some environments. I suppose we could check the class of the actual FileSystem that's returned for the URI. I'll spend a few minutes to see how hard this is to do. If it turns out to be really tricky, then I'll probably defer this task until someone else hits this issue.
from spark-redshift.
In #76, I updated spark-redshift
to detect when the unsupported block FileSystem is being used and to throw a helpful error message. Note that we still do allow the s3://
URI scheme; the exception is only triggered if that scheme's FileSystem implementation is the unsupported one.
Therefore, I'm going to resolve this issue as fixed.
from spark-redshift.
Related Issues (20)
- spark-redshift generate corrupted Parquet files
- Caused by: org.jets3t.service.impl.rest.HttpException: 400 Bad Request--> py4j.protocol.Py4JJavaError: An error occurred while calling o42.save. : org.apache.spark.sql.catalyst.errors.package$TreeNodeException: execute, tree:
- Any plan to support Scala 2.12? HOT 2
- tempdir is not being cleaned up after successful load to redshift HOT 1
- Spark SQL module compatability
- loading credentials from AWS_WEB_IDENTITY_TOKEN_FILE
- How are temporary files created in S3 when calling the driver? HOT 1
- Support unload to tempdir with specified KMS key
- Error when saving a dataframe to Redshift (java.lang.ArrayStoreException: java.lang.invoke.SerializedLambda) HOT 1
- [Question] Don't know how to save StructField
- Getting SignatureDoesNotMatch Error
- error while writing to Redshift: Cannot merge com.databricks.spark.redshift.HashSetAccumulator
- Pushdown filter not working with Varchar datatype HOT 1
- Is it possible to add columns to redshift table automatically? #1256
- Redshift support binary type since december 2021 HOT 1
- Databricks read from redshift first selected column returns wrong value if quotes used
- Java.lang.IllegalArgumentException: Wrong FS s3://bucket/folder/manifest.json -expected s3a://bucket HOT 1
- Spark 3.3 support HOT 1
- py4j.protocol.Py4JJavaError: An error occurred while calling o96.save.
- java.sql.SQLException: [Amazon][JDBC](11380) Null pointer exception.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from spark-redshift.