ptzagk / awesome-apachespark-collections

A curated list of awesome Apache Spark packages and resources.

awesome-ApacheSpark-collections or Awesome Spark

Book keeping of Apache Spark web search!
Also a curated list of awesome Apache Spark packages and resources.

Other github awesome links

• Datasciene
• Machine Learning
• DataSets
• Big Data
• Pipelines
• Awesome Awesomeness

Online Free Clusters

• DataBricks

Notebooks and IDEs

• Apache Zeppelin - Web-based notebook that enables interactive data analytics with plugable backends, integrated plotting, and extensive Spark support out-of-the-box.
• Spark Notebook - Scalable and stable Scala and Spark focused notebook bridging the gap between JVM and Data Scientists (incl. extendable, typesafe and reactive charts).
• sparkmagic - Jupyter magics and kernels for working with remote Spark clusters, for interactively working with remote Spark clusters through Livy, in Jupyter notebooks.

Books on Apache Spark

• Spark Internals
• Mastering Apache Spark 2.0
• Jaceklaskowski Gitbooks
• RDD Collection API Examples
• Databricks Knowledge Base GitBook
• Spark in Action - https://github.com/spark-in-action
• Learning Spark: Lightning-Fast Big Data Analysis 2015 by Holden Karau, Andy Kowinski, Matei Zaharia, Patrick Wendell Git: https://github.com/databricks/learning-spark
• Fast Data Processing with Spark by Holden Karau
• Advanced Analytics with Spark Patterns for Learning from Data at Scale by Sandy Ryza, Uri Laserson, Sean Owen, Josh Wills https://github.com/sryza/aas
• Machine Learning with Spark by Nick Pentreath
• Apache Spark Graph Processing by Rindra Ramamonjison
• Big Data Analytics with Spark - A Practitioner’s Guide to Using Spark for Large Scale Data Analysis, by Mohammed Guller
• Mastering Apache Spark by Packt
• Apache Spark 2 for Beginners https://github.com/PacktPublishing/Apache-Spark-2-for-Beginners

Blogs

• Spark.tc
• Cloudera
• Cloudera Spark UseCases
• DataBricks
• Hortonworks
• AmpCamp
• VisualApi

Must Read list

##Introduction

• Exploring Stockoverflow Dataset
  • Why Apache Spark is a Crossover Hit for Data Scientists
  • Real-time Data Mining with Spark
  • Accumulators
    • http://imranrashid.com/posts/Spark-Accumulators/
    • http://stackoverflow.com/questions/29494452/when-are-accumulators-truly-reliable

Spark + Hadoop

• Spark and HDFS
• Spark + HBase + HDFS

Spark Internals

SparkSQL

• Deep Dive into Spark SQL’s Catalyst Optimizer

Streaming

• Integrating Kafka and Spark Streaming
• http://www.infoq.com/articles/apache-spark-streaming

Spark on GPU / DeepLearning

• https://databricks.com/blog/2016/10/27/gpu-acceleration-in-databricks.html
• https://databricks.com/blog/2016/12/21/deep-learning-on-databricks.html
• https://docs.databricks.com/applications/deep-learning/tensorflow.html

Tips & Tricks

-http://blog.smaato.com/tuning-spark-streaming-applications/

Spark Packages

• http://spark-packages.org/
• File I/O CSV
• Neo4j [Neo4j-Spark Connector] (https://github.com/neo4j-contrib/neo4j-spark-connector)

Videos on Apache Spark

Channels

• https://www.youtube.com/user/TheApacheSpark
• https://www.youtube.com/channel/UC3q8O3Bh2Le8Rj1-Q-_UUbA
• https://www.youtube.com/user/BerkeleyAMPLab
• https://www.youtube.com/user/typesafehub

Playlists

• Apache Spark and Scala
• Popular Apache Spark & Machine learning videos
• Popular Videos - Apache Spark & Tutorial
• Popular Videos - Apache Spark

Github Projects - Ever Growing List!

Setup

1. https://github.com/clearstorydata-cookbooks/apache_spark
2. https://github.com/gwik/spark-cookbook
3. https://github.com/azavea/ansible-spark
4. https://github.com/tzolov/apache-spark-build-pipeline
5. https://github.com/aur-atomica-net/apache-spark
6. https://github.com/GELOG/docker-ubuntu-spark
7. https://github.com/kbastani/spark-neo4j

Spark Internals

1. https://github.com/JerryLead/SparkInternals

Spark Learning/Workshop

0. https://github.com/Mageswaran1989/aja
1. https://github.com/deanwampler/spark-workshop
2. https://github.com/ceteri/spark-exercises
3. https://github.com/lenards/explore-spark
4. https://github.com/seglo/learning-spark
5. https://github.com/ceteri/intro_spark
6. https://github.com/HadoopTW/CS100.1x
7. https://github.com/EvanZ/myvagrant
8. https://github.com/zfz/spark-cs100.1x
9. https://github.com/StephenHarrington/spark
10. https://github.com/gudiseva/Spark
11. https://github.com/hoangtamvo/spark
12. https://github.com/okaram/spark
13. https://github.com/linshiu/spark
14. https://github.com/jingjinggu/Apache_Spark
15. https://github.com/aur-atomica-net/apache-spark
16. https://github.com/dhesse/SparkTalk
17. https://github.com/adamliesko/bigdata-spark
18. https://github.com/skrusche63/spark-connect
19. https://github.com/spirom/LearningSpark

Spark

1. https://github.com/hohonuuli/sparknotebook
2. https://github.com/googlegenomics/spark-examples
3. https://github.com/sujee81/SparkApps
4. https://github.com/praveensripati/spark-examples
5. https://github.com/jdutton/spark-playground
6. https://github.com/arjones/spark-news
7. https://github.com/felixcheung/spark-notebook-examples
8. https://github.com/manku-timma/spark
9. https://github.com/joseratts/Spark
10. https://github.com/giocode/SparkTutorial
11. https://github.com/eenov8/apacheSpark
12. https://github.com/yu-iskw/spark-dataframe-introduction
13. https://github.com/rajanpupa/ApacheSparkExample
14. https://github.com/XD-DENG/Spark-practice

Streaming

1. https://github.com/prabeesh/SparkTwitterAnalysis
2. https://github.com/cotdp/spark-example-clickstream-social
3. https://github.com/ippontech/metrics-spark-receiver
4. https://github.com/aleph-w/ApacheSparkLearning

Sql

1. https://github.com/rnamboodiri/spark-cassandra-integrations
2. https://github.com/choi258/Spark_apache

MLLib

1. https://github.com/OndraFiedler/spark-recommender
2. https://github.com/marklit/recommend
3. https://github.com/staple/spark-agd
4. https://github.com/tizfa/sparkboost
5. https://github.com/rahmanusta/Spark-Bayes
6. https://github.com/spacedotworks/decisiontree_ApacheSpark

Spark Machine Learning

1. https://github.com/PredictionIO/PredictionIO
2. https://github.com/BaiGang/spark_multiboost
3. https://github.com/alitouka/spark_dbscan
4. https://github.com/amplab/keystone
5. https://github.com/krasserm/akka-analytics

Spark Streaming

1. https://github.com/miguno/kafka-storm-starter
2. https://github.com/killrweather/killrweather
3. https://github.com/NFLabs/ambari
4. https://github.com/rustyrazorblade/killranalytics

Spark + Visulization

1. https://github.com/FRosner/spawncamping-dds

Spark + WebServer

1. https://github.com/calrissian/spark-jetty-server

Spark + REST

1. https://github.com/spark-jobserver/spark-jobserver

Spark + Cassendra

1. https://github.com/datastax/spark-cassandra-connector

Spark + NoSQL datastore

1. https://github.com/Stratio/deep-spark
2. https://github.com/RussellSpitzer/spark-cassandra-csv
3. https://github.com/haosdent/spark-hbase

Spark + Elastic search

1. https://github.com/skrusche63/spark-elastic
2. https://github.com/mhausenblas/elsa
3. https://github.com/SHSE/spark-es

Spark + Azure + PowerBI

1. https://github.com/granturing/spark-power-bi

Spark + Genomics

1. https://github.com/bigdatagenomics/adam

Spark + Ruby

1. https://github.com/ondra-m/ruby-spark

Usefull Addons

1. https://github.com/amplab/spark-indexedrdd
2. https://github.com/mrsqueeze/spark-hash
3. https://github.com/simplymeasured/phoenix-spark
4. https://github.com/calrissian/spark-jetty-server
5. https://github.com/cloudera/spark-timeseries
6. https://github.com/skrusche63/spark-weblog

Tools

1. https://github.com/andypetrella/spark-notebook
2. https://github.com/ibm-et/spark-kernel
3. https://github.com/mraad/SparkProject
4. https://github.com/saurfang/sbt-spark-submit