suhothayan / hadoop-spark-pig-hive

Docker with hadoop spark pig hive

Apache Hadoop distribution on Ubuntu with Spark, Pig, and Hive

The docker image Apache hadoop 2.9.2 distribution on Ubuntu 18.04 with Spark 2.4.3, Pig 0.17.0, and Hive 2.3.5

Find this on Docker Hub https://hub.docker.com/r/suhothayan/hadoop-spark-pig-hive

Build the image

docker build  -t suhothayan/hadoop-spark-pig-hive:2.9.2 .

Pull the image

docker pull suhothayan/hadoop-spark-pig-hive:2.9.2

Start a container

In order to use the Docker image you have just build or pulled use:

docker run -it -p 50070:50070 -p 8088:8088 -p 8080:8080 suhothayan/hadoop-spark-pig-hive:2.9.2 bash

Testing

You can run one of the hadoop examples:

# run the mapreduce
yarn jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.9.2.jar grep input output 'dfs[a-z.]+'

# check the output
hdfs dfs -cat output/*

Run

Hive

hive

or

 beeline -u jdbc:hive2://

Pig

pig

Spark

Scala

spark-shell

Python

pyspark

hadoop-spark-pig-hive's Issues

Map Reduce Jobs not competed

I have submitted several map-reduce jobs (jar) files, but all are getting stuck in map-reduce. Job: Running job: job_id state

i tried it with below command:

yarn jar mapreduce-design-patterns/mapreduce-design-patterns/target/mapreduce-design-patterns-1.0-SNAPSHOT.jar week4.count.WordCount stackapps.com/stackapps.com/comments/ output/wordCount

RunJar jarFile [mainClass] args...
bash-4.1# yarn jar mapreduce-design-patterns/mapreduce-design-patterns/target/mapreduce-design-patterns-1.0-SNAPSHOT.jar week4.count.WordCount stackapps.com/stackapps.com/comments/ output/wordCount
23/11/03 10:10:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
23/11/03 10:10:35 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
23/11/03 10:10:35 INFO input.FileInputFormat: Total input paths to process : 1
23/11/03 10:10:35 INFO mapreduce.JobSubmitter: number of splits:1
23/11/03 10:10:35 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1699020468106_0001
23/11/03 10:10:36 INFO impl.YarnClientImpl: Submitted application application_1699020468106_0001
23/11/03 10:10:36 INFO mapreduce.Job: The url to track the job: http://8c0d59f0a1c5:8088/proxy/application_1699020468106_0001/
23/11/03 10:10:36 INFO mapreduce.Job: Running job: job_1699020468106_0001

my operating system is lsb_release -a

please advise on this matter

Failed to get reduce task report after execution of pig in mapreduce mode

I want to run pig in mapreduce mode.
But I am getting this error.

2020-12-26 00:18:55,435 [main] INFO  org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2020-12-26 00:18:56,440 [main] INFO  org.apache.hadoop.ipc.Client - Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2020-12-26 00:18:57,442 [main] INFO  org.apache.hadoop.ipc.Client - Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2020-12-26 00:19:05,560 [main] WARN  org.apache.pig.tools.pigstats.mapreduce.MRJobStats - Failed to get reduce task report

Is this is an configuration issue? Please support