cluster setup error: unknown spark version about spark-ec2

[omdv]

@DSLituiev try --hadoop-major-version=2, it worked fine for me.

I don't know if file have been uploaded, but i noticed that the name changed to spark-1.6.2-bin-hadoop1-scala2.11.tgz on https://www.apache.org/dist/spark/spark-1.6.2/

[DSLituiev]

Seemingly there is no archive for spark-1.6.2-bin-hadoop1.tgz there.
Using --hadoop-major-version 2 leads to the following:

2016-07-28 04:53:42 (3.83 MB/s) - ‘spark-1.6.2-bin-hadoop2.4.tgz’ saved [273797124/273797124]

Unpacking Spark
[timing] spark init:  00h 01m 12s
Initializing ephemeral-hdfs
ERROR: Unknown Hadoop version
[timing] ephemeral-hdfs init:  00h 00m 00s
Initializing persistent-hdfs
ERROR: Unknown Hadoop version
[timing] persistent-hdfs init:  00h 00m 00s
Initializing mapreduce
ERROR: Unknown Hadoop version
mapreduce/init.sh: line 20: return: -1: invalid option
return: usage: return [n]
File or directory /root/mapreduce doesn't exist!
[timing] mapreduce init:  00h 00m 00s
Initializing spark-standalone

[shivaram]

Yeah it looks like that file is missing -- @JoshRosen can you help in uploading spark-1.6.2-bin-hadoop1.tgz to the S3 bucket spark-related-packages ?

[DSLituiev]

@omdv: As I mentioned above (assuming nothing changed in code or repositories):
Using --hadoop-major-version 2 leads to the following:

2016-07-28 04:53:42 (3.83 MB/s) - ‘spark-1.6.2-bin-hadoop2.4.tgz’ saved [273797124/273797124]

Unpacking Spark
[timing] spark init:  00h 01m 12s
Initializing ephemeral-hdfs
ERROR: Unknown Hadoop version
[timing] ephemeral-hdfs init:  00h 00m 00s
Initializing persistent-hdfs
ERROR: Unknown Hadoop version
[timing] persistent-hdfs init:  00h 00m 00s
Initializing mapreduce
ERROR: Unknown Hadoop version
mapreduce/init.sh: line 20: return: -1: invalid option
return: usage: return [n]
File or directory /root/mapreduce doesn't exist!
[timing] mapreduce init:  00h 00m 00s
Initializing spark-standalone

[omdv]

@DSLituiev could you try it with = sign?

[DSLituiev]

@omdv: it works. Thank you!