ranxerox / mapr-streams-sparkstreaming-hbase Goto Github PK

Commands to run :

Step 1: First compile the project on eclipse: Select project  -> Run As -> Maven Install

Step 2: use scp to copy the ms-sparkstreaming-1.0.jar to the mapr sandbox or cluster

also use scp to copy the data sensor.csv file from the data folder to the cluster
put this file in a folder called data. The producer reads from this file to send messages.

scp  ms-sparkstreaming-1.0.jar user01@ipaddress:/user/user01/.
if you are using virtualbox:
scp -P 2222 ms-sparkstreaming-1.0.jar [email protected]:/user/user01/.

Create the topic

maprcli stream create -path /user/user01/pump -produceperm p -consumeperm p -topicperm p
maprcli stream topic create -path /user/user01/pump -topic sensor -partitions 3

get info on the topic
maprcli stream info -path /user/user01/pump


To run the MapR Streams Java producer and consumer:

java -cp ms-sparkstreaming-1.0.jar:`mapr classpath` solution.MyProducer

java -cp ms-sparkstreaming-1.0.jar:`mapr classpath` solution.MyConsumer

Step 3:  To run the Spark  streaming consumer:

start the streaming app
 
spark-submit --class solution.SensorStreamConsumer --master local[2] ms-sparkstreaming-1.0.jar

ctl-c to stop 

step 4: Spark streaming app writing to hbase

first make sure that you have the correct version of HBase installed, it should be 1.1.1:

cat /opt/mapr/hbase/hbaseversion 
1.1.1

Next make sure the Spark HBase compatibility version is correctly configured here: 
cat  /opt/mapr/spark/spark-1.5.2/mapr-util/compatibility.version 
hbase_versions=1.1.1

If this is not 1.1.1 fix it.


Create an hbase table to write to:
launch the hbase shell
$hbase shell

create '/user/user01/sensor', {NAME=>'data'}, {NAME=>'alert'}, {NAME=>'stats'}

Step 4: start the streaming app writing to HBase
 
spark-submit --class solution.HBaseSensorStream --master local[2] ms-sparkstreaming-1.0.jar

ctl-c to stop

Scan HBase to see results:

$hbase shell

scan '/user/user01/sensor' , {'LIMIT' => 5}

scan '/user/user01/sensor' , {'COLUMNS'=>'alert',  'LIMIT' => 50}

Step 5:
exit, run spark (not streaming) app to read from hbase and write daily stats 

spark-submit --class solution.HBaseReadRowWriteStats --master local[2] ms-sparkstreaming-1.0.jar

$hbase shell

scan '/user/user01/sensor' , {'COLUMNS'=>'stats',  'LIMIT' => 50}


cleanup if you want to re-run:

maprcli stream topic delete -path /user/user01/pump -topic sensor 
maprcli stream topic create -path /user/user01/pump -topic sensor -partitions 3
hbase shell 
truncate '/user/user01/sensor'