xiaoningyb / hadoop_projects

Including hadoop MR, YARN, Spark, Storm........

#hadoop mr, yarn, spark, storm on yarn

##1.run mr hadoop jar target/my-hadoop-1.0-SNAPSHOT.jar org.apache.hadoop.mr.App /user/ning/wordcount/in /user/ning/wordcount/out4

##2.run yarn hadoop jar target/hadoop-yarn-1.0-SNAPSHOT.jar org.apache.hadoop.mr.App /user/ning/wordcount/in /user/ning/wordcount/out4

##3.run the spark yarn spark-submit --class com.apache.spark.WordCount --master yarn-cluster --num-executors 3 --driver-memory 4g --executor-memory 2g --executor-cores 1 ./target/my-spark-1.0-SNAPSHOT.jar /user/ning/wordcount/in

##4.run the strom local: storm jar ./target/my-storm-1.0-SNAPSHOT-jar-with-dependencies.jar storm.example.ning.WordCountTopology

remote: storm jar ./target/my-storm-1.0-SNAPSHOT-jar-with-dependencies.jar storm.example.ning.WordCountTopology my-storm

##4.run the strom with kafaka start kafka: kafka-server-start.sh config/server.properties

create input topic: kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

create output topic: kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic topic_out

producter: kafka-console-producer.sh --broker-list localhost:9092 --topic test

consumer: kafka-console-consumer.sh --zookeeper localhost:2181 --topic topic_out --from-beginning

local: storm jar ./target/my-storm-1.0-SNAPSHOT-jar-with-dependencies.jar storm.example.ning.KafkaTopology

remote: storm jar ./target/my-storm-1.0-SNAPSHOT-jar-with-dependencies.jar storm.example.ning.KafkaTopology kafka

##5.run the spark streaming with kafaka local spark-submit --class com.apache.spark.JavaKafkaWordCount --master local[*] --num-executors 3 --driver-memory 1g --executor-memory 1g --executor-cores 1 ./target/my-spark-1.0-SNAPSHOT-jar-with-dependencies.jar localhost:2181 group spark-stream 1

remote(driver on client) spark-submit --class com.apache.spark.JavaKafkaWordCount --master yarn-client --num-executors 3 --driver-memory 1g --executor-memory 1g --executor-cores 1 ./target/my-spark-1.0-SNAPSHOT-jar-with-dependencies.jar localhost:2181 group spark-stream 1

remote(driver on cluster) spark-submit --class com.apache.spark.JavaKafkaWordCount --master yarn-cluster --num-executors 3 --driver-memory 1g --executor-memory 1g --executor-cores 1 ./target/my-spark-1.0-SNAPSHOT-jar-with-dependencies.jar localhost:2181 group spark-stream 1