Redshift sqlContext.read fails with S3 input path does not exist - but it does about spark-redshift

[marmbrus]

Hmmm, could it just be S3 eventual consistency? I do often see files missing briefly due to this.

[yhuai]

For your temp dir, can you try using s3n as the scheme ("tempdir", "s3n://spark-temp-dir")?

[gregrahn]

@marmbrus For reference S3 FAQ says

Amazon S3 buckets in all Regions provide read-after-write consistency for PUTS of new objects...

Which should be the case here.

@yhuai The s3n did work for me, and just to add, only using s3 fails every time. Maybe the README should be updated to reflect that.

[jaley]

@gregrahn I get that all the time. Are you running locally and just adding hadoop-aws to your classpath as a dependency? You'll probably find that the same code (with s3://) works fine on EMR. I think you have to actually register s3:// in hdfs-site.xml or something like that for it to work - not sure exactly, but it's definitely a local setup issue. Not a spark-redshift issue, just a general S3 configuration drama.

[JoshRosen]

I also ran into some issues with s3:// so I'm wondering whether it would make sense to log a warning if we see that scheme being used.

[yhuai]

Have a clarification question. Does the scheme s3 at here mean the original S3 block fileystem? If so, I think it cannot read normal S3 data, right?

[JoshRosen]

/cc @aarondav, do you happen to know the answer to this offhand?

[JoshRosen]

One actionable thing that we can do here is to update the examples to show s3n://.

[JoshRosen]

@jaley, I think that s3:// worked for you in EMR because EMR aliases that to a s3n://-like FileSystem rather than the classic S3 block filesystem. See https://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-plan-file-systems.html and http://notes.mindprince.in/2014/08/01/difference-between-s3-block-and-s3-native-filesystem-on-hadoop.html

In a nutshell, I don't think that spark-redshift will work with the classic S3 block filesystem, but I don't see a super easy way to provide helpful programmatic warnings about this: we can't / shouldn't filter based on the URI since s3:// does work in some environments. I suppose we could check the class of the actual FileSystem that's returned for the URI. I'll spend a few minutes to see how hard this is to do. If it turns out to be really tricky, then I'll probably defer this task until someone else hits this issue.

[JoshRosen]

In #76, I updated spark-redshift to detect when the unsupported block FileSystem is being used and to throw a helpful error message. Note that we still do allow the s3:// URI scheme; the exception is only triggered if that scheme's FileSystem implementation is the unsupported one.

Therefore, I'm going to resolve this issue as fixed.