dataandcaffeine / spark-wordcount

spark-wordcount

Running a WordCount Program with PySpark

1.Prerequisites : Single-Node Hadoop Cluster with Pydoop

1.1. Ubuntu 12.10, 64-bit desktop version

1.2. Oracle Java 8
		Install from command line :
		sudo add-apt-repository ppa:webupd8team/java
		sudo apt-get update
		sudo apt-get install oracle-java8-installer  
	
1.3. Hadoop 2.7.4
		Download from : http://hadoop.apache.org/releases.html

1.4. Spark 2.2.0 for Hadoop 2.7.*
		Download from : https://spark.apache.org/downloads.html

2. Start Spark

2.1. Start hadoop from hadoop folder line : 
		./start-all.sh
2.2. Check processes by running jps
		DataNode
		NameNode
		SecondaryNameNode
		TaskTracker
		JobTracker
		JPS
2.3. Start PySpark from inside spark folder :
		./bin/pyspark

3. Running WordCount

3.1. Create input and output directories on hdfs
		hdfs dfs -mkdir ~/input/
		hdfs dfs -mkdir ~/output/
3.2. Save input.txt to ~/input/
		hdfs dfs -copyFromLocal input.txt /input
3.3. Run WordCount.txt in pyspark shell.
3.4. Output will be available at hdfs://localhost:9000/output/output.txt